Multi-modal perception based robotic arm type intelligent furniture cleaning robot and control method

Abstract

This invention discloses a robotic arm-type intelligent furniture cleaning robot and its control method based on multimodal perception. It includes a multimodal radar fusion modeling method and a curvature semantic localization method adapted to the robotic arm, constructing a ternary semantic map considering the distinction between soft and hard obstacles and outputting key parameters of the surface to be cleaned during localization to allow the robot to adjust its posture. Furthermore, this invention proposes an AI visual recognition model based on an improved multi-task recognition network to achieve high-precision perception of the working environment. Based on this, the robotic arm can be controlled to adopt either a forearm power-off gravity-adhesion energy-saving cleaning control mode or a continuous cleaning mode triggered by travel limits, using arm-chassis linkage. This invention integrates high-precision perception, three-dimensional operation execution, and intelligent adaptive cleaning, promoting the development of intelligent cleaning robots towards three-dimensional operation and adaptive control.

Description

Technical Field

[0001] This invention belongs to the field of cleaning robot technology, and relates to a robotic arm-type intelligent furniture cleaning robot based on multimodal perception and its control method. Background Technology

[0002] With the fast pace of modern life and the rise of pet ownership, home cleaning faces a dual challenge: on the one hand, the spread of hair, dander, and stains has become a core pain point in home cleaning; on the other hand, traditional cleaning methods have significant limitations—manual cleaning is inefficient and difficult to reach blind spots such as sofa crevices and high places on the headboard, handheld vacuum cleaners are physically demanding and have limited coverage, and ordinary robot vacuums only focus on floor cleaning and lack the ability to precisely clean furniture surfaces, let alone meet the differentiated cleaning needs of furniture made of different materials.

[0003] The technological shortcomings of existing intelligent cleaning equipment are becoming increasingly apparent: First, in terms of navigation and positioning, single-sensor perception solutions are easily affected by furniture obstruction and fabric coverage, resulting in time-consuming map building and low positioning accuracy, leading to missed cleaning or repetitive work; Second, in terms of the actuator, there is a lack of dedicated robotic arm components, which can only achieve planar cleaning and cannot cover three-dimensional space. Moreover, the few existing devices with robotic arms have problems such as insufficient positioning accuracy and motion vibration, making it difficult to adapt to complex furniture environments; Third, in terms of cleaning strategies, there is a lack of intelligent recognition capabilities for furniture materials and pollutants, and the cleaning intensity and mode are fixed, which can easily cause damage to sensitive materials (such as silk and leather). At the same time, the targeted cleaning effect on specific pollutants such as hair and dander is not good.

[0004] To address the aforementioned technical shortcomings, there is an urgent need to develop a new type of equipment that integrates high-precision perception, three-dimensional operation execution, and intelligent adaptive cleaning. This invention solves the core problems of traditional cleaning equipment, such as inaccurate perception, incomplete coverage, and imprecise cleaning, through multimodal radar fusion modeling and curvature semantic localization, precise control of a three-degree-of-freedom robotic arm, and innovation or synergy of AI visual recognition technology. It achieves comprehensive, refined, and safe cleaning of the home environment, meeting the demands of modern people for a high-quality living environment and driving the development of intelligent cleaning robots towards three-dimensional operation and adaptive control. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a robotic arm-based intelligent furniture cleaning robot and its control method based on multimodal perception. This includes a multimodal radar fusion modeling and a curvature semantic localization method adapted to the robotic arm. It constructs a ternary semantic map that considers the distinction between soft and hard obstacles and outputs key parameters of the surface to be worked on during the localization process so that the robot can adjust its work posture. This will help promote the development of intelligent cleaning robots.

[0006] The technical solution adopted in this invention is as follows: A control method for a robotic arm-type intelligent furniture cleaning robot based on multimodal perception is disclosed. The method includes multimodal radar fusion modeling and curvature semantic localization method adapted to the robotic arm. The robotic arm-type intelligent furniture cleaning robot is a three-degree-of-freedom robotic arm with a rotary joint, an upper arm telescopic joint, and a lower arm telescopic joint. Its chassis is equipped with LiDAR and millimeter-wave radar. The LiDAR is used to build a basic geometric boundary framework of the environment, and the millimeter-wave radar is used to penetrate and detect the space occupancy behind the obstruction. The fusion modeling constructs a ternary semantic map that considers the distinction between soft and hard obstacles. During the robot localization process, curvature semantic classification is performed on the LiDAR point cloud to extract key parameters of the surface to be worked on, which are then used by the robot to adjust its work posture.

[0007] In the above technical solution, the fusion modeling construction of a ternary semantic map that considers the distinction between soft and hard obstacles includes establishing transparency attribute labeling rules: when the lidar feedback area is "occupied" but the millimeter-wave radar feedback area is "idle", the area is labeled with the semantic tag "soft occlusion"; when both feedback is "occupied", it is labeled as "rigid obstacle"; when both feedback is "idle", it is labeled as "accessible / workable space", forming a ternary semantic map of "rigid obstacle-soft occlusion-idle space" and matching differentiated cleaning strategies.

[0008] Furthermore, after the LiDAR distortion is corrected, feature values ​​are extracted based on the coordinate components of the laser point and its neighboring points to quantify the local curvature. Based on the magnitude of the feature values, geometric semantic classification is performed on the surface of the home environment to distinguish between planar, curved, and corner areas. Normal vectors of planar and curved surfaces are extracted. During the robot's coordinate positioning process, the geometric semantic label and normal vector of the current working surface are output simultaneously to guide the cleaning head at the end of the robot arm to adjust its working posture.

[0009] Furthermore, the method also includes AI visual recognition technology to assist robot cleaning, specifically including: acquiring an image of the surface to be cleaned, preprocessing the image and inputting it into a multi-task recognition network model, wherein the multi-task recognition network model uses depthwise separable convolution to extract features from the input image to obtain multi-scale feature maps; After at least one feature extraction stage, each feature channel is weighted using a channel attention mechanism; the multi-scale feature map is fused using a feature pyramid network and a path aggregation network to obtain a deep feature vector; the deep feature vector is then concatenated with a multi-dimensional physical feature vector directly extracted from the preprocessed image to obtain a fused feature; wherein the multi-dimensional physical feature vector has clear physical meaning and quantifiability, including: RGB channel mean, RGB channel variance, image entropy, contrast, correlation, aspect ratio of the bounding rectangle of the object, circularity of the object, actual area of ​​the object, pollutant distribution density, material gloss, and the second and third moments in the color moments; based on the fused feature, material classification results, object detection results, and surface tilt angle estimation results are output in parallel, enabling the recognition system to simultaneously possess the generalization ability of deep learning and the physical interpretability and quantification accuracy of handcrafted features, making it more suitable for robotic arm cleaning control.

[0010] Further, acquire the RGB image and depth image of the surface to be processed, and preprocess the images, including: The RGB image and the depth image are coordinate registered to generate a pseudo-color depth map. A depth threshold is used to distinguish between the surface-attached target and the background, and candidate target areas are initially screened. Bilateral filtering is used to reduce noise in the image, and morphological dilation is used to enhance the salience of small components; Perspective correction is performed on the image based on gyroscope tilt data, and effective regions of interest are extracted through contour detection and area thresholding.

[0011] Furthermore, the multi-task recognition network model is trained in the following manner: The training dataset includes sample images, each of which is labeled with a material category label, a foreign object label, and an angle label. The material category label is selected from different categories of furniture materials. The angle label includes various angles. The foreign object label includes hard, fragile, flexible, and contaminant categories. Among them, the hard, fragile, and flexible categories are all items that need to be avoided, and the contaminant category is contaminants that need to be removed. For the material classification branch of the multi-task recognition network model, the Focal loss function is used, the EIOU loss function is used for the item detection branch, and the MSE loss function is used for furniture surface tilt angle estimation.

[0012] Furthermore, the method also includes a forearm power-off gravity-adhesion energy-saving and cleaning control mode, specifically including: Step 1: Obtain the material type, curvature, contaminant distribution density, and obstacle information of the surface to be worked on. When the material type belongs to a preset low-damage-risk material set, the curvature is less than the curvature threshold, the contaminant distribution density is less than the density threshold, and the obstacle information meets the preset safety conditions, the energy-saving mode is triggered. Step 2: Plan the reference pose of the robotic arm end effector parallel to the working surface, obtain the reference stroke of the robotic arm extension joint through inverse kinematics calculation, and drive the arm to the reference stroke and lock it. Step 3: De-energize the drive motor of the robotic arm's forearm extension joint, release the axial degree of freedom of the forearm, and let the forearm fall naturally under the action of gravity until the end cleaner contacts the working surface, establishing a stable normal contact force; Step 4: After the contact force stabilizes, start the cleaning function, and complete the cleaning trajectory by coordinating the rotation of the robotic arm's rotary joint with the movement of the chassis, during which the forearm motor remains de-energized. Step 5: After cleaning is complete, power on the forearm motor to retract the forearm, raise the upper arm, and exit the energy-saving mode.

[0013] Furthermore, the method also includes a continuous cleaning mode with arm-chassis linkage triggered by travel limit, specifically including: Step 1): Calculate the stroke percentage of each joint of the robotic arm in real time. When the stroke percentage of any joint reaches the first warning threshold, trigger the warning and start pre-planning the path; when it reaches the second linkage threshold, trigger the linkage mode. Step 2): Based on the type of joint reaching its limit and the deviation between the target path point and the maximum reachable range of the robotic arm, determine the compensation direction and compensation distance of the chassis so that the stroke ratio of each joint of the robotic arm returns to the preset optimal range after compensation. Step 3): Establish a synchronous matching model with constant absolute end velocity. Solve the joint compensation speed by using the Jacobian matrix pseudo-inverse to control the coordination between the chassis movement speed and the robot arm joint speed, so that the end cleaner maintains a constant linear velocity during chassis movement. Step 4): After the chassis moves to the target position, the travel percentage is checked and returned to the optimal range. The linkage mode is exited and the robotic arm continues to execute the cleaning trajectory. If the travel limit is triggered again, the above steps are repeated to achieve uninterrupted continuous cleaning.

[0014] Furthermore, in step 2), determining the chassis's compensation direction based on the joint type that has reached its limit includes: establishing a chassis coordinate system with the origin at the chassis's geometric center, the X-axis representing the robot's forward / backward direction (vertical), and the Y-axis representing the robot's left / right direction (lateral); when the joint that has reached its limit is a rotary joint, determining that the chassis will perform lateral translation compensation along the Y-axis; when the joint that has reached its limit is a telescopic joint, determining that the chassis will perform forward / backward translation compensation along the X-axis.

[0015] Furthermore, step 3) specifically involves: based on the real-time speed of the chassis. Joint compensation speed is calculated by solving the pseudo-inverse of the Jacobian matrix. To ensure the end speed Constant:

[0016] In the formula As the pseudo-inverse of the Jacobian matrix, the S-shaped velocity curve is used to plan the chassis acceleration and deceleration.

[0017] The beneficial effects of this invention are: This invention proposes a multimodal radar fusion modeling and curvature semantic localization method adapted to a robotic arm. It constructs a ternary semantic map considering the distinction between soft and hard obstacles and outputs key parameters of the surface to be worked on during localization, allowing the robot to adjust its posture. Furthermore, it proposes an AI visual recognition model based on an improved multi-task recognition network to achieve high-precision perception of the working environment. Based on this, the robotic arm can be controlled to adopt either a forearm power-off gravity-fit energy-saving cleaning control mode or a stroke-limit-triggered arm-chassis linkage continuous cleaning mode based on the recognition results. This invention integrates high-precision perception, three-dimensional operation execution, and intelligent adaptive cleaning, promoting the development of intelligent cleaning robots towards three-dimensional operation and adaptive control. Attached Figure Description

[0018] Figure 1 This is a physical diagram of the robotic arm intelligent furniture cleaning robot in one embodiment of the present invention; Figure 2 This is a schematic diagram of a three-degree-of-freedom robotic arm structure in one embodiment of the present invention; Figure 3 This is a hardware block diagram of an intelligent furniture cleaning robot in one embodiment of the present invention; Figure 4 The following is a comparison of the point cloud pass-through filtering effect in one embodiment of the present invention: (a) original point cloud before filtering, (b) lower half of the point cloud after filtering, and (c) upper half of the point cloud after filtering. Figure 5 This is a comparison diagram before and after outlier filtering in one embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the effect of converting a 3D point cloud map to a 2D raster map in one embodiment of the present invention; Figure 7 This is a two-dimensional optimized grid for path planning in one embodiment of the present invention; Figure 8 This is a simulation experiment diagram of the A* algorithm in one embodiment of the present invention; Figure 9 Here is the core flowchart of the A* algorithm; Figure 10This is a simulation diagram of DWA local path planning in one embodiment of the present invention; Figure 11 This is a diagram illustrating the annotation process and annotation results in one embodiment of the present invention; Figure 12 This is a schematic diagram of the improved YOLOv5s three-task recognition model structure in one embodiment of the present invention; Figure 13 This is a model training loss curve in one embodiment of the present invention; Figure 14 This is an example of the desktop foreign object recognition effect in one embodiment of the present invention; Figure 15 This is a schematic diagram illustrating the material recognition effect in one embodiment of the present invention; Figure 16 This is a schematic diagram illustrating the pollutant identification effect in one embodiment of the present invention; Figure 17 This is a schematic diagram of a high-precision targeted cleaning control process for a robotic arm in one embodiment of the present invention. Detailed Implementation

[0019] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] The present invention provides a control method for a robotic arm-type intelligent furniture cleaning robot based on multimodal perception. The robotic arm of this robot is a three-degree-of-freedom robotic arm. According to a specific embodiment of the invention, its core hardware components may include: a mobile chassis, a three-degree-of-freedom "one-rotation, two-extension" robotic arm (rotation joint J1, upper arm extension joint J2, lower arm extension joint J3), an end effector, a six-dimensional force sensor, an AI vision module, a multimodal radar perception module, and a main controller. The overall physical form and hardware architecture are as follows: Figures 1-3 As shown.

[0021] The control method includes three aspects: multimodal radar fusion modeling and curvature semantic localization, AI visual material recognition technology, and precise control of a three-degree-of-freedom robotic arm, as detailed below: I. Rapid Modeling and Positioning System for Home Environment Based on Multimodal Radar Fusion This invention employs a dual-radar penetration method combining laser and millimeter-wave radar to model the home environment and differentiate between hard and soft obstacles. Traditional single-laser radar can only detect the surface outline of objects and cannot penetrate soft obstructions such as sofa covers and bed skirts, misjudging them as insurmountable obstacles. This creates blind spots for cleaning areas such as under sofas and along bed edges, and also fails to provide a basis for the robotic arm to reach in and remove obstructions during cleaning. This innovative method combines the high-precision surface boundary detection capability of laser radar with the ≥80% non-metallic penetration characteristic of millimeter-wave radar, breaking the limitations of the traditional binary "occupied / vacant" map and innovatively constructing a ternary semantic map of "rigid obstacles - soft obstructions - vacant space," achieving a dual and accurate representation of the surface morphology and internal structure of the home environment. First, a basic geometric boundary framework for the home environment is established using LiDAR. Then, millimeter-wave radar is used to penetrate soft obstructions and detect the space occupancy and solid obstacles behind them. A dynamic weight fusion algorithm is used to quantify the penetration attributes and semantically label each spatial grid. The final output ternary semantic map can directly match differentiated cleaning strategies, solving the problem of misjudgment of soft obstructions from the root and providing complete decision support for the robotic arm's precision cleaning.

[0022] According to one embodiment of the present invention, a lidar is horizontally mounted at the front center of the robot's bottom chassis, performing a 360° omnidirectional scan to acquire three-dimensional point cloud data of the ground, furniture bottoms, and surrounding environment, constructing the basic geometric boundaries of the environment. This mounting position is close to the working surface, accurately capturing the contour details of table legs, bed legs, and low obstacles. Simultaneously, the robot's structural design avoids obstruction by the movement of the overhead robotic arm, ensuring no blind spots in the 360° scan and providing a stable spatial reference for chassis navigation and near-field robotic arm operations. The lidar acquires surface point clouds at a frequency of 10Hz, ensuring geometric boundary accuracy. A millimeter-wave radar is mounted side-by-side with the lidar at the front of the robot's bottom chassis, tilted upwards at a 30° angle to the horizontal. Utilizing its ≥80% non-metallic penetration rate, it detects solid obstacles (such as scattered toys or hidden stains) beneath soft obstructions (such as under sofas or inside mattresses), supplementing the lidar's perception deficiencies in obstructed scenarios, forming a dual environmental perception of "surface + depth." Millimeter-wave radar captures signals in the penetrating area at a higher sampling frequency. It achieves frame alignment of the two types of radar data through a time synchronization mechanism (linear interpolation to correct sampling deviation), ensuring the timeliness of data fusion.

[0023] Establish transparency attribute labeling rules. When the lidar reports "area occupied" but the millimeter-wave radar reports "area vacant", the system labels the area as "soft occlusion". When both report "occupied", it is labeled as "rigid obstacle". When both report "vacant", it is labeled as "accessible / operable space", forming a 3D map containing semantic attributes.

[0024] In one specific embodiment of the invention, for areas with "soft obstructions" (such as drooping sofa covers), the chassis navigation system plans an approach path and simultaneously sends a "push-away action command" to the robotic arm, controlling the cleaning head at the end of the robotic arm to gently push away the obstruction before performing vacuuming. For "empty" areas between "rigid obstacles" (such as gaps in sofa armrests), the system directly guides the robotic arm to reach into the gap for precise cleaning, avoiding the abandonment of work due to misjudging obstacles, as is common in traditional robots, thus achieving effective cleaning of obstructed areas and hidden gaps. The close-to-the-ground installation layout of the LiDAR can accurately reproduce the boundary dimensions of low spaces such as under sofas and beds, providing millimeter-level spatial constraint data for the robotic arm's reach-in cleaning, and preventing collisions between the robotic arm and the bottom of furniture during operation.

[0025] According to a specific embodiment of the present invention, a curvature semantic positioning method adapted for robotic arm cleaning of curved surfaces is adopted. Traditional robot positioning technology can only solve the coordinate problem of "where the robot is," which is completely disconnected from the operational requirements of "how the robotic arm cleans." This easily leads to problems such as poor cleaning head fit, missed cleaning, or even scratching of furniture surfaces when the robotic arm is working on curved surfaces such as tilted tabletops and curved sofa armrests. This method breaks through the technical limitations of traditional pure coordinate positioning. Through curvature semantic classification of LiDAR point clouds, the positioning data is transformed into cleaning operation parameters that the robotic arm can directly execute, realizing the synchronous output of "robot coordinates and cleaning operation parameters." This allows the positioning technology to directly serve the precise cleaning operation of the robotic arm. By calculating the curvature of LiDAR point clouds, geometric semantic classification of the home environment surface can be performed to accurately distinguish between planar, curved, and corner areas. Key parameters such as the normal vector and tilt angle of the working surface are extracted simultaneously. While the positioning system outputs the robot coordinates, these operation parameters are simultaneously transmitted to the robotic arm, directly guiding the cleaning head to complete the precise adjustment of the working posture. The specific method is as follows: Existing pass-through filters (such as...) can be used. Figure 4 ) or outlier filtering ( Figure 5 Methods such as [list of methods] are used to correct distortion in lidar data. Then, feature values ​​are extracted from the point cloud data after distortion correction based on the coordinate components of the lidar point and its neighboring points. To quantify local curvature, according to a specific embodiment of the present invention, the feature value extraction method can be as follows: Select five neighboring points (x, y, z) before and after each laser point to form a local region, and quantify the local curvature using two feature value calculation methods: Method 1:

[0026] Method 2:

[0027] Eigenvalues The larger the curvature value, the greater the degree of surface curvature. Based on the curvature value, the point cloud is divided into three core semantic categories: sharp feature points (…). Corresponding to furniture edges and door frame corners, these serve as positioning "anchor points" to ensure coordinate accuracy; flat feature points ( The corresponding flat areas, such as desktops and walls, are labeled "flat work areas"; curved surface feature points ( The corresponding curved areas, such as sofa armrests and curved table edges, are marked as "curved surface work areas".

[0028] For the "planar work area" and the "curved surface work area", the covariance matrix of the local point cloud is calculated by principal component analysis (PCA), and the eigenvectors are extracted to obtain the surface normal vector. This vector directly reflects the tilt direction and angle of the working surface. A mechanism for binding positioning and working parameters is established, whereby the positioning system outputs the robot's pose. Simultaneously, output the semantic labels and normal vectors of the current working area. The curvature value is used to form an integrated "positioning-operation" data package, which is transmitted to the robotic arm control module in real time through ROS topics, realizing the direct conversion of positioning data into operation parameters.

[0029] For the "planar working area", the robotic arm uses the normal vector Adjust the cleaning head tilt approach angle To 0°, ensure the cleaning head is perpendicular to the plane to maximize the suction area; for "curved work areas", the system dynamically adjusts according to the curvature value. The angle (smoothly varying within the range of 0° to 5°) ensures that the cleaning head always maintains the optimal angle with the normal of the curved surface, avoiding scratching the material and ensuring a good fit; for "sharp feature points" (furniture edges and corners), the robotic arm automatically reduces the contact pressure (e.g., from 2.0N to 0.5N on wooden surfaces) and adjusts the cleaning trajectory to bypass the edges and corners, preventing damage to the furniture and achieving adaptive cleaning for different geometric feature areas.

[0030] Based on high-quality point clouds after distortion correction and curvature semantic classification, the system can accurately identify different geometric feature areas such as furniture corners, planar work areas, and curved work areas, and simultaneously output corresponding pose adjustment parameters to directly guide the robotic arm to complete adaptive cleaning operations. Experimental data shows that this method reduces the contact error between the robotic arm cleaning head and the work surface to ≤3mm, increases the cleaning coverage of curved areas by 70%, and reduces the material damage rate of furniture corners to 0. It completely solves the core problem of traditional positioning technology, which only outputs coordinates and is disconnected from the needs of robotic arm operations, and achieves precise coordination between robot positioning and robotic arm cleaning operations.

[0031] Furthermore, this invention proposes an optimal chassis docking navigation method based on the reachability constraints of the robotic arm. Traditional chassis navigation only focuses on obstacle avoidance and safe docking, completely neglecting the reachability of the robotic arm for cleaning operations. This easily leads to situations where the robot docks safely without collisions, but the robotic arm cannot reach the target cleaning area, or a poor docking position causes force imbalance on the robotic arm, ultimately resulting in cleaning omissions, poor cleaning effects, or even scratching furniture. This method breaks through the traditional logic of independent chassis navigation, taking the robotic arm's 3-DOF workspace and force characteristics as the core constraints of chassis navigation, and proposes an optimal docking method for the working posture, allowing chassis navigation to serve the efficient operation of the robotic arm, achieving globally optimal planning of "chassis docking position and robotic arm working posture". The specific implementation method is as follows: Based on the structural parameters (extension / extension range of the upper arm / lower arm, rotation angle range) of the "one-rotation-two-extension" three-degree-of-freedom robotic arm, a kinematic model is established using the DH parameter method. The forward / inverse kinematic equations of the robotic arm are solved, and the joint space is traversed to obtain all reachable positions of the robotic arm end in Cartesian space. A three-dimensional workspace map of the robotic arm is constructed to clarify the robotic arm's operational coverage area corresponding to different chassis docking positions.

[0032] By combining force sensor data at the end of the robotic arm, the optimal force range for cleaning different material surfaces (e.g., 2.0N for wood surfaces and 0.5N for leather surfaces) is determined through experiments. Based on the kinematic model of the robotic arm, the safe stopping distance between the chassis and the furniture corresponding to this force range is calculated in reverse. This distance is used as a navigation constraint to avoid insufficient force on the robotic arm and incomplete cleaning due to stopping too far away, or excessive squeezing and damage to the furniture due to stopping too close, thus ensuring the rationality of the force during robotic arm operation.

[0033] Based on this, a reachability map is constructed: the 3D workspace of the robotic arm is orthogonally projected onto the chassis navigation plane to obtain the maximum coverage area of ​​the robotic arm at each chassis position; combined with safe docking distance constraints, the projected area is clipped to remove positions where the force requirements are not met; the clipped area is discretized into a grid map, forming "reachable operation grids" and "unreachable grids," where reachable grids indicate that when the chassis is docked at that position, the robotic arm can cover the target cleaning area while meeting the force constraints. Figure 6 , Figure 7 As shown, by converting a 3D point cloud map to a 2D raster map and overlaying accessibility constraints, a 2D optimized raster map for path planning is obtained. In this map, reachable operation rasters are marked as passable areas, and unreachable rasters are marked as prohibited areas.

[0034] In the path planning phase, global path planning is first performed on the optimized grid map based on the A* algorithm, such as... Figure 8-9 As shown, paths passing through reachable work grids are prioritized to ensure that the chassis docking position meets the robot arm's work coverage requirements. To achieve dynamic obstacle avoidance and local path optimization, the DWA algorithm is specifically improved by adding "robot arm reachability" and "center of gravity steady state" scoring items to the local path evaluation function. The improved evaluation function is as follows:

[0035] The variables in the formula are defined as follows: heading: the directional deviation between the path endpoint and the target point, used to guide the robot towards the target point; dist: the distance between the path and the nearest obstacle, used to ensure the robot's obstacle avoidance safety; velocity: the robot's movement speed, used to optimize the smoothness and efficiency of the path; reachability: the reachability score of the robot arm in the grid where the path endpoint is located, provided by the reachability map, ensuring that the robot arm can cover the target area after docking; stability: the steady-state score of the robot chassis's center of gravity, evaluated by calculating the center of gravity offset of the supporting polygon, to prevent the robot from tipping over. For reachability weights, As a steady-state weight for the center of gravity, by adjusting the weight coefficient, priority is given to the path that is "accessible to the robotic arm and has a stable center of gravity", so as to avoid the robotic arm's operation being restricted or the robot tipping over after the chassis is parked.

[0036] To improve operational adaptability across different furniture scenarios, a work pose set planning system is implemented: the navigation target point is no longer a single coordinate, but rather a "work pose set" containing multiple candidate docking positions. Based on the size and shape of the target cleaning area, the system performs multi-objective optimization with the goals of "maximizing the robotic arm's operational coverage" and "balancing the force at the end effector," selecting the optimal docking point overall. For large furniture such as sofas and beds, the chassis automatically adjusts the stopping distance according to the working space of the robotic arm (e.g., stopping 30cm away from the sofa) to ensure that the robotic arm can cover the surface and edge areas of the sofa. Large-area cleaning can be completed without frequent movement of the chassis, improving work efficiency. For small furniture such as dining tables and bedside tables, the chassis ensures balanced force on the robotic arm when it is parked, so that the cleaning head maintains constant contact pressure during operation. This not only improves the cleaning effect but also effectively protects the furniture material, achieving a dual improvement in the efficiency and quality of the robotic arm's operation.

[0037] like Figure 10As shown, the improved DWA local path planning algorithm can generate smooth paths that balance accessibility and stability in real time in dynamic environments. Experimental data shows that this method increases the coverage of the robotic arm's work by 50%, reduces cleaning time by 30%, and reduces material damage rate to 0 due to force balance, achieving a deep integration of chassis navigation and robotic arm operation, and achieving a balance between efficient cleaning and safety protection.

[0038] II. AI-based visual recognition model-assisted cleaning Building upon the above, this invention also proposes a method for robot cleaning based on AI visual recognition technology. To balance the strong semantic perception capabilities of deep learning with the physical interpretability and quantitative accuracy of manually designed features, this invention adopts a recognition architecture that combines deep network features with multi-dimensional handcrafted features: First, an image of the surface to be cleaned is acquired and preprocessed. The image is then input into an improved multi-task recognition network model, where multi-scale feature extraction is performed through depthwise separable convolution. A channel attention mechanism is introduced during the feature extraction stage to strengthen the weights of key features. Finally, multi-scale deep feature fusion is achieved through a feature pyramid network and a path aggregation network to obtain a highly robust deep feature vector.

[0039] Meanwhile, for physical quantities that are strongly related to cleaning control, such as furniture material properties, surface tilt angle, contaminant density, and object geometry, this invention further extracts multi-dimensional manual feature vectors. These features have clear physical meaning and precise quantification capabilities, and can directly provide numerical basis for robotic arm posture adjustment and cleaning strategy selection.

[0040] The semantic features extracted by the deep network are combined with the multi-dimensional hand-designed physical features to obtain the final fused features. Based on these fused features, the network can output material classification results, object detection results, and surface tilt angle estimation results in parallel, achieving dual perception capabilities of "semantic perception" and "physical quantification". This ensures robustness of recognition in complex scenarios and outputs accurate physical parameters that can be directly used for closed-loop control of robotic arms.

[0041] This invention does not replace manual features with a single network, but rather integrates the two, enabling the system to possess both the generalization ability of deep learning and the physical interpretability and quantification accuracy of manual features, thus making it more suitable for robotic arm cleaning control.

[0042] Specifically, according to one embodiment of the present invention, its implementation includes the following: 1. Sensor module deployment and data acquisition 1.1 Selection and Installation of Core Sensing Module AI Vision Perception Module: Utilizing an enhanced OpenMV-H7 vision module with an integrated OV7725 image sensor, mounted 8cm above the vacuum cleaner actuator. This module supports dynamic switching between 640×480 (high-precision mode) and 320×240 (energy-saving mode), with a frame rate of 30fps and a distortion rate ≤1.5%. A built-in hardware acceleration unit enables real-time image preprocessing. The installation position covers the 20cm working radius of the vacuum cleaner head to ensure complete detection while avoiding dust contamination of the lens during vacuuming. Simultaneously, it achieves dynamic tracking in sync with the robotic arm's movement.

[0043] Six-dimensional force feedback sensor: Utilizing a Nano 17 high-precision force sensor, integrated between the robotic arm end effector and the vacuum cleaner actuator. Measuring range 0-30N, resolution 0.01N, sampling frequency 500Hz, it collects real-time contact pressure data between the vacuum cleaner head and furniture surfaces, providing a basis for flexible contact control and preventing damage from hard objects or insufficient adhesion to soft surfaces.

[0044] Tilt and Depth Auxiliary Sensors: An MPU6050 six-axis gyroscope (measurement range ±180°, accuracy ±0.1°) is configured to detect the tilt angle of furniture surfaces; supplemented by an Intel Realsense D435 depth camera to acquire 3D point cloud data to distinguish foreign objects from dust, while also assisting in calculating the spatial attitude of the tilted surface, providing data support for adjusting the dust collection angle.

[0045] Environmental Adaptive Sensor: Install a miniature ambient light sensor (sampling frequency 10Hz, range 0-1000 lux) to dynamically adjust the exposure parameters of the vision module (exposure time extended to 50ms in low light scenes, gain reduced to 1.2 times in strong light scenes) to ensure the accuracy of foreign object and material recognition under complex lighting conditions.

[0046] 1.2 Data Transmission and Synchronization Mechanism The vision module communicates with the Raspberry Pi 4B (running the ROS Noetic system) via a USB 3.0 interface, with an image data transmission bandwidth of up to 5Gbps and a latency of ≤5ms; the force sensor connects to the STM32F103 main control chip via an I2C interface, using a custom communication protocol (baud rate 115200bps) with a transmission latency of ≤10ms.

[0047] Establish a multi-sensor time synchronization mechanism: Based on the Raspberry Pi system timestamp, frame synchronization of visual images, force feedback data, ambient light data, and tilt angle data is achieved through hardware trigger signals (a synchronization pulse is triggered every 33ms). The synchronization error is ≤3ms, ensuring real-time linkage between sensing data and cleaning control commands.

[0048] 1.3 Dataset Construction and Preprocessing Dataset Creation: A 3D dataset of materials, foreign objects, and tilt angles was constructed, covering 12 common furniture materials (wood, glass, metal, plastic, marble, leather, fabric, silk, rattan, velvet, leather, and acrylic). Each material category includes four tilt angle scenarios: 0° (horizontal), 15°, 30°, and 45°. Foreign object types cover common household obstacles, categorized into three types based on physical characteristics: hard objects (such as small daily necessities like keys, remote controls, mobile phones, glasses, and mice), fragile objects (such as glass cups, ceramic cups, candle holders, and ornaments), and flexible objects (such as tissues, clothing, and towels). It also includes contaminants to be removed, such as dust (density 0.1-0.5 g / m²) and hair (length 2-8 cm). 80 samples were collected for each scene, totaling 12×4×8×80=30720 images, which were divided into training set (21504 images), validation set (6144 images) and test set (3072 images) in a 7:2:1 ratio.

[0049] Data annotation strategy: The LabelImg tool is used for triple labeling: ① Material category labels (12 categories); ② Foreign object labels (including 3 main categories of foreign objects + "no foreign objects", totaling 4 categories, with bounding box coordinates and dimensions); ③ Angle labels (0°, 15°, 30°, 45°). Pixel-level masks are used to annotate dust and hair, distinguishing between dust-collecting objects and obstacles to be avoided. Annotation results are stored in XML format (e.g., ...). Figure 11 This includes a tag index, normalized bounding box, tilt angle value, and target type (obstacle avoidance / vacuuming) identifier.

[0050] Data augmentation and preprocessing: Mosaic technology was used to randomly stitch together 4 samples with different materials, tilt angles and foreign objects to simulate a complex home scene; combined with operations such as random horizontal flipping (probability 0.5), rotation (±5°, simulating tilt angle error), brightness adjustment (±20%), and Gaussian noise addition (variance ≤0.01), the training set was expanded to 86016 images.

[0051] The preprocessing workflow includes: Gaussian filtering for noise reduction (the filtering formula is...). (σ=1.5), histogram equalization enhancement (calculation formula is...) ,in grayscale The number of pixels is reduced, and the ROI region is extracted based on Canny edge detection (threshold 50-150), background interference is removed, and the amount of invalid computation is reduced by 65%.

[0052] 2. Feature Extraction and Intelligent Recognition System 2.1 Image Preprocessing Workflow 1. Multi-source data fusion preprocessing: RGB images and depth data are fused, and a pseudo-color depth map is generated through coordinate registration. A depth threshold (≤5mm) is used to distinguish between surface-attached targets (dust, hair, foreign objects) and the background, and candidate target areas are initially screened.

[0053] 2. Adaptive noise reduction and enhancement: A bilateral filtering algorithm is used to replace the traditional Gaussian filtering algorithm. While preserving the details of the edges of foreign objects (such as the outline of a pen and the edges of a key), noise is reduced, and the signal-to-noise ratio of the filtered image is improved by 45%. For small components such as hair, morphological dilation operation (3×3 kernel) is used to enhance the saliency of the target.

[0054] 3. Tilt Correction and ROI Extraction: Perspective correction is performed on tilted surface images based on gyroscope tilt data, converting the tilted scene into a horizontal viewpoint for easier feature extraction; contour detection and area filtering are used (foreign object area ≥ 0.5cm²). 2 Dust area ≥0.1cm 2 Extract effective ROI with an accuracy rate of ≥97%.

[0055] 2.2 Multi-dimensional Feature Fusion Extraction To overcome the shortcomings of pure deep networks in terms of accurate quantization of physical quantities, interpretability, and direct mapping of control commands, this invention further constructs 15-dimensional multi-dimensional handcrafted feature vectors based on deep features. It covers five dimensions: color, texture, pollutants, home furnishings, and surface appearance. The meaning and calculation method of each parameter are shown in Table 1. Table 1: Definition of Parameters for Multi-Dimensional Fusion Feature Vector

[0056] 2.3 Improved YOLOv5s Three-Task Recognition Model To address the computing power limitations of the embedded platform (Raspberry Pi 4B) for cleaning robots and the needs of home scenarios, the YOLOv5s model was specifically optimized. A three-task network (named YOLOv5s-Vac) was constructed to perform "material recognition + foreign object detection + tilt estimation," adapting to the computing power of the embedded platform. The network structure is as follows: Figure 12 As shown: 2.3.1 Backbone Network Optimization Depth-wise separable convolution replacement: All standard convolutional layers in the original YOLOv5s backbone network (CSPDarknet-53) are replaced with depth-wise convolutions (Depth-wise Conv + Point-wise Conv). Through a two-step operation of channel-wise filtering and point-wise fusion, the computational cost is reduced by 75% while retaining feature extraction capabilities, and the number of parameters is reduced from 7.5M to 1.8M, making it suitable for the computing power requirements of a Raspberry Pi 4B.

[0057] ECA attention mechanism embedding: An ECA (Efficient Channel Attention) attention layer is added after the C3 module of the backbone network, and the weights of each feature channel are adaptively adjusted through 1D convolution (kernel_size=3). This mechanism can enhance the representation ability of key features such as foreign object edges (such as brush outlines) and contaminant morphology (such as hair edges and dandruff outlines) without adding too much additional computation (the number of parameters increases by ≤0.5%).

[0058] 2.3.2 Neck Network Optimization The system employs an FPN (Feature Pyramid Network) + PAN (Path Aggregation Network) structure, fusing multi-scale features: an 80×80 feature map (small scale, used to detect fine hairs and small dandruff), a 40×40 feature map (medium scale, brushes), and a 20×20 feature map (large scale, used to identify furniture materials and tilt angles), ensuring detection accuracy for targets of different sizes.

[0059] An adaptive threshold filter (confidence threshold of 0.3) is added during the feature fusion process to remove low-confidence feature maps, reduce redundant calculations, and improve inference speed (20% improvement in inference speed on the Raspberry Pi platform).

[0060] 2.3.3 Header Network Design Material recognition branch: Outputs the probability distribution of 12 furniture material categories, optimized using the Focal loss function, formula: (in , This loss function can balance the problem of imbalanced samples (such as fewer samples of niche materials like silk and rattan), improving the material recognition accuracy to 95.8%.

[0061] Item detection branch: Output the bounding box coordinates of 4 item types. Based on confidence level, the EIOU loss function is used to optimize positioning accuracy, as shown in the formula below. (b is the center of the prediction box) (where c is the center of the ground truth bounding box and c is the diagonal length of the minimum bounding rectangle). This loss function can solve the localization error in scenarios with overlapping bounding boxes, achieving a mAP@0.5 of 92.3% for household item detection.

[0062] Tilt angle estimation branch: Regression of 0°-90° numerical values, MSE loss (formula) ) Optimize error.

[0063] 2.4 Model Training and Performance Validation 2.4.1 Training Environment and Parameter Settings Hardware environment: Intel Core i7-10700 processor (8 cores and 16 threads), 32GB RAM, NVIDIA RTX 3060 GPU (6GB VRAM), 1TB SSD storage; embedded terminal uses Raspberry Pi 4B (4GB RAM, OpenVINO acceleration enabled).

[0064] Software environment: System Ubuntu 18.04, deep learning framework PyTorch 1.12, programming language Python 3.8, dependent libraries OpenCV 4.5 and NumPy 1.21.

[0065] Core training parameters: initial learning rate 0.001, using cosine annealing decay strategy (T_max=300, eta_min=0.00001); batch size 16 (GPU training) / 4 (embedded validation); number of iterations 300 (the first 5 epochs are warm-up training, with the learning rate linearly increasing to 0.001); the optimizer used is AdamW (weight_decay=0.0001) to suppress overfitting.

[0066] 2.4.2 Training Process and Performance Metrics During training, the model loss curve gradually converged: the loss of the material recognition branch decreased from the initial 2.8 to 0.35, the loss of the pollutant detection branch decreased from the initial 3.2 to 0.42, and the performance on the validation set stabilized after 200 epochs.

[0067] Final model performance: Raspberry Pi inference speed 28fps, material recognition accuracy 96.2%, foreign object detection mAP@0.5 93.5% (hard foreign objects 95.1%), tilt angle estimation error ≤0.5°, meeting real-time requirements.

[0068] In other embodiments of the present invention, other specific multi-task recognition models may be adopted based on the core concept of the present invention to achieve automatic recognition of different materials, different types of foreign objects, and different tilt angles, which will not be elaborated here.

[0069] III. Closed-loop control method for targeted cleaning of robotic arms based on gravity adaptive energy saving and arm-chassis linkage This invention, building upon multimodal environmental semantic mapping and AI visual recognition technologies, addresses the core pain points of existing cleaning robotic arms, such as high energy consumption due to long reach, discontinuous operation, and complete disconnect between the robotic arm and chassis navigation. It constructs an integrated control system encompassing perception and decision-making, pattern matching, energy-saving control, limit linkage, and closed-loop verification. The core innovations include two main technological branches: first, a pioneering gravity-based energy-saving cleaning control method for the forearm when power is off, fundamentally solving the high energy consumption problem caused by the high torque drive of long reach forearms; second, a continuous cleaning method based on arm-chassis linkage triggered by the robotic arm's travel limit, addressing the issues of needing to stop and replan after reaching the travel limit, low cleaning efficiency due to frequent start-stop cycles, and trajectory breakage. Ultimately, this achieves low-energy, high-efficiency, and damage-free continuous cleaning across all home scenarios.

[0070] 1. Modeling of the system coupled coordinate system and basic model This section provides the quantitative algorithm foundation for the two core control methods, realizing the kinematic decoupling of the robotic arm and chassis, the precise quantification of travel limits, and the coordinate unification of control commands, which is a prerequisite for the full-process algorithm implementation.

[0071] 1.1 Construction of the Arm-Chassis Coupled Coordinate System Establish a four-level unified coordinate system to achieve coordinate consistency across global positioning, chassis navigation, and robotic arm control, thus avoiding data deviations between multiple modules: World coordinate system {W}: Global map coordinate system, established by the first part of the multimodal SLAM system, providing a global reference for all motions; Chassis coordinate system {B}: The origin is located at the geometric center of the chassis, the X-axis points forward, the Y-axis points to the left, and the Z-axis points vertically upward. The chassis pose in the world coordinate system is... ,in For planar coordinates, For heading angle; The robot arm base coordinate system {0} is rigidly connected to the chassis coordinate system {B}, with its origin located at the center of the robot arm's rotary joint J1. The fixed transformation matrix between {0} and {B} is... , which are known constants that have been pre-calibrated; The end effector coordinate system {E} is located at the center of the cleaning head suction port and is rigidly connected to the end of the forearm. It is the direct control object for cleaning operations.

[0072] 1.2 Kinematic Modeling of a Three-DOF Robotic Arm The "one-rotation, two-extension" robotic arm used in this invention includes a rotary joint, an upper arm / lower arm telescopic joint, and an end effector for cleaning. Its structure provides a hardware foundation for kinematic modeling (e.g., Figures 1-3 (As shown). The DH parameter method is used to establish analytical models of the robot's forward and inverse kinematics, providing core mathematical support for subsequent pose planning, stroke limit determination, and velocity calculation. The DH parameters of the robot are shown in the table below:

[0073] (1) Forward kinematic model The pose mapping of the end effector relative to the base coordinate system is established using a homogeneous transformation matrix. The general form of the homogeneous transformation matrix for a single joint is:

[0074] Substituting the DH parameters, we obtain the total transformation matrix of the end coordinate system relative to the base coordinate system:

[0075] This matrix allows for the direct calculation of the three-dimensional position of the end-effector cleaning head in the base coordinate system under any joint variables. With posture.

[0076] (2) Inverse kinematics model Inverse kinematics calculation is performed using a closed-loop analytical method, given the desired end-effector pose. It can quickly solve the corresponding joint variables. Solution speed This meets the requirements for real-time control. Simultaneously, it utilizes the Jacobian matrix... Establish a mapping relationship between joint velocities and end-effector Cartesian space velocities:

[0077] This provides a foundation for subsequent speed synchronization control and force control algorithms.

[0078] 1.3 Quantitative Modeling of Robotic Arm Stroke Limit Constraints This invention employs two-level threshold quantization for the robotic arm's operating range, providing precise triggering criteria for coordinated follow-up control and avoiding ambiguous limit determinations. Define joint space limits: Determine the physical travel boundaries of each joint. , ; Two trigger thresholds are set, with the first-level warning threshold being the percentage of travel of any joint. ≥90% triggers an early warning, prompting the chassis to initiate pre-planning of the following path in advance; The threshold for triggering a level 2 linkage is: the percentage of travel of any joint. ≥98%, trigger arm-chassis linkage follow-up control, chassis moves in real time to compensate for position; Formula for calculating the percentage of trip:

[0079] in For joints Real-time location, For discrete time steps.

[0080] 2. Forearm power failure gravity-adhesion energy-saving and cleaning control mode This method combines the aforementioned AI visual recognition results to determine high-hardness, low-damage-risk flat work scenarios. By completely de-energizing the forearm drive motor and relying on its own gravity to achieve stable contact and cleaning, it eliminates the drive energy consumption of the forearm joint at the source, while ensuring cleaning effectiveness and work safety. This embodiment describes the entire process of this energy-saving mode control method, from triggering to execution, from safety control to exit, specifically including: 2.1 Multi-dimensional verification of energy-saving mode trigger conditions In this example, the material category is determined by the AI ​​visual material recognition module. Pollutant distribution density D, curvature value of the working surface output by curvature semantic localization. The local semantic map obstacle information determines whether to trigger this energy-saving mode. Specifically: Material grade verification: The 12 categories of home furnishing materials are divided into two compatibility levels. Only materials of grade I can trigger the energy-saving mode. Grade I (Compatible with): Glass, marble, metal, ceramic tile, acrylic; low risk of damage, high hardness; Level II (Prohibited): Leather, silk, wood, fabrics, rattan, velvet, leather; high risk of damage, requires delicate control. Surface morphology verification: only when the curvature value of the working surface is... The validation passes when the value is less than 0.2 (for flat surfaces); this mode is disabled for curved surface scenarios. Contaminant risk verification: Verification is only passed when the contaminant distribution density D < 0.8 (no large areas of stubborn stains), to avoid insufficient cleaning power in gravity adhesion mode; Obstacle safety check: There are no rigid obstacles or easily damaged foreign objects within 5cm of the work surface to avoid collision damage in the event of a power outage; If all four checks pass, the power-saving mode trigger flag will be output. =1, otherwise output =0, switch to variable parameter flexible admittance control mode.

[0081] This step outputs the energy-saving mode trigger flag. Pattern matching results.

[0082] 2.2 Inverse kinematics calculation of the reference pose for operation The input for this step is the trigger flag. Target cleaning area world coordinates Normal vector of the working surface .

[0083] The control logic is as follows: 1) Coordinate unification transformation: Convert the world coordinates of the target cleaning point to coordinates in the robot arm base coordinate system:

[0084] 2) Baseline attitude planning: Planning the normal vector between the end-effector's main axis and the working surface. Fully aligned, tilted to near angle This ensures that the suction port plane is parallel to the working surface, providing a uniform contact base for gravity bonding; 3) Calculation of boom reference height: The total weight of the forearm and end effector is obtained through calibration experiments. 1. Elastic coefficient of the forearm transmission mechanism Calculate the reference travel of the upper arm extension joint. To ensure stable contact during natural fall after power failure of the forearm, the calculation formula is as follows:

[0085] in For the work surface Shaft height, For robotic arm base Shaft height; 4) Inverse kinematics solution: Based on the reference pose, the reference joint variables are solved using an inverse kinematics model. ,in (The forearm is initially fully retracted).

[0086] The output of this step is the baseline joint variable. Arm locking command.

[0087] 2.3 Core Balance Control for Forearm Power-Off Gravity Fit The input for this step is the baseline joint variable. Six-dimensional force sensor for real-time contact force

[0088] The control logic is as follows: 1) Upper arm locking action: Drives the upper arm extension and retraction joint. Exercise to The motor brake is triggered to lock, fixing the boom position and torque, and completing the benchmark height positioning; 2) Forearm power-off release: Forearm extension and retraction joint The drive motor sends a power-off command to release the motor drive and the brake lock, completely releasing the axial extension and retraction freedom of the forearm. At this time, the forearm is only subject to its own weight, the end weight, and the support force of the working surface. 3) Gravity-based contact balance control: The forearm falls naturally under the influence of gravity until its end contacts the working surface, establishing a contact force balance equation:

[0089] in The sliding friction force of the forearm transmission mechanism is within the calibration range. The corresponding stable contact force range is It fully meets the dust collection and sealing requirements of hard flat surfaces; 4) Steady state verification: when Maintain within the stable range The above determines that gravity bonding is complete, and a bonding completion flag is output. Start the vacuum motor.

[0090] The outputs of this step are: forearm power off command, vacuuming start command, and bonding completion marker. .

[0091] 2.4 Safety Redundancy Control in Operation Processes The input for this step is real-time contact force. AI visual foreign object detection results and real-time feedback on joint positions; The control logic is as follows: 1) Collision emergency control: Set a threshold for sudden change in contact force. If detected If the collision is determined to be a foreign object, immediately execute the following: ① Power on and lock the forearm motor; ② Quickly raise the boom. ③ Stop the vacuum cleaner motor to avoid damaging furniture and equipment; 2) Adaptive fine-tuning of bonding pressure: If visual re-inspection reveals stain residue rate ,by Step length fine-tuning of boom reference height until contact force achieve No forearm motor drive is required throughout the entire process; 3) Overtravel protection: Real-time monitoring of forearm extension and retraction. ,like This immediately triggers the forearm to be electrically locked, preventing the mechanical structure from detaching.

[0092] The outputs of this stage are emergency control commands and boom fine-tuning commands.

[0093] 2.5 Joint-chassis coordinated execution of cleaning trajectory The input for this step is the bonding completion marker. Pollutant distribution density ; The control logic is as follows: 1) Clean trajectory generation: Generates an adapted trajectory based on contaminant distribution density. Using a spiral trajectory, A zigzag trajectory is used; 2) Trajectory execution decoupling: Completely avoids forearm motor drive, decomposing the cleaning trajectory into two dimensions of energy-free execution: Lateral sweeping: rotating joints via robotic arm The rotation of the cleaning head allows for horizontal coverage. Longitudinal coverage: The cleaning head achieves longitudinal coverage by moving the chassis back and forth; 3) Motion synchronization control: Establish Joint speed With chassis moving speed The synchronous matching model ensures that the linear velocity of the cleaning head trajectory remains constant. Matching formula:

[0094] in The real-time extension length of the forearm is fed back by the joint encoder to ensure a smooth and seamless trajectory.

[0095] The output of this step is Joint rotation speed command, chassis movement speed command.

[0096] 2.6 Cleaning effect closed-loop verification and mode exit The input for this step is the real-time image of the working surface from the end-view camera and the boundary information of the cleaning area; The core control logic is as follows: Cleaning effect re-inspection: The residual stain rate on the work surface is detected by AI visual model. If the residual rate is < 5%, the cleaning of the area is deemed complete. Safe Exit Mode: After cleaning, follow the exit procedure in sequence: ① Power on the forearm motor to drive the forearm back to the zero position. ① Lock the device; ② Raise the boom to a safe height; ③ Stop the vacuum motor; ④ Switch to chassis navigation mode. Output: Mode exit command, area cleaning complete indicator.

[0097] 3. Continuous cleaning mode with arm-chassis linkage triggered by the robotic arm's travel limit. This method addresses the pain points of downtime and replanning caused by the robotic arm's travel limits, as well as low cleaning efficiency. It achieves uninterrupted operation through limit prediction, chassis positioning, speed synchronization, and simultaneous moving and cleaning. It is fully compatible with energy-saving cleaning modes and flexible admittance control modes. The core process is as follows: 3.1 Real-time prediction and early warning of travel limits The input for this step is the real-time joint variables of the robotic arm. Next path point of the target cleaning trajectory Joint limit threshold; The core control logic is as follows: 1) Calculate the stroke percentage of each joint in real time according to the stroke percentage formula. ; 2) Set two-level threshold triggering: Trigger Level 1 warning ( ), initiate path pre-planning; Trigger secondary linkage ( ), start the chassis to provide real-time support; 3) Predict whether the next path point exceeds the reachable space of the robotic arm based on the inverse kinematics model. If it is unreachable, it will be triggered in advance. To avoid trajectory breakage.

[0098] The output of this step is a warning flag. Linkage trigger flag Limit joint numbering .

[0099] 3.2 Precise Calculation of Replacement Direction and Distance The input for this step is the linkage trigger flag. Limit joint numbering Target path points ; The core control logic is as follows: 1) Matching extreme types with fill strategies: Rotational limits correspond to chassis Lateral translation of the axis to fill the gap; Expansion / extension limits correspond to the chassis The axis is shifted forward and backward to compensate for the missing position. 2) Compensation distance calculation: Based on the deviation between the target path point and the maximum reachable range of the robotic arm, the compensation distance is calculated.

[0100] In the formula The target path point is defined by the chassis coordinate system coordinates. The coordinates of the maximum reachable point in the robotic arm's extreme position; 3) Calculate the optimal positioning of the chassis for backup. This ensures that the percentage of the maximum joint travel after the replacement falls back to the optimal range of 40-60%, thus avoiding frequent triggering of the limit.

[0101] The output of this step is the chassis clearance distance. , movement direction vector Target pose 3.3 Arm-Chassis Motion Synchronization Speed ​​Matching The inputs for this stage are the compensation direction and distance, and the desired linear velocity at the end point. Cleaning trajectory curvature; The core control logic is as follows: 1) Establish a velocity synchronization matching equation, constraining the terminal absolute velocity to remain constant. To ensure consistent cleaning results:

[0102] In the formula The absolute velocity in the final world coordinate system is constrained. ; 2) Based on the real-time chassis speed, the joint compensation speed is calculated using the pseudo-inverse of the Jacobian matrix to ensure a constant end-effector speed:

[0103] In the formula This is the pseudo-inverse of the Jacobian matrix; 3) Use an S-shaped speed curve to plan chassis acceleration and deceleration, and jerk. Maximum acceleration To avoid trajectory deviation.

[0104] The output of this stage is the chassis real-time speed command. Real-time speed commands for robotic arm joints 3.4 Continuous Trajectory Closed-Loop Control with Simultaneous Movement and Clearing The inputs for this stage are real-time speed commands, end-effector visual trajectory deviation, and contact force feedback from the six-dimensional force sensor. The core control logic is as follows: 1) Track Deviation PID Correction: By visually detecting the lateral deviation Δe between the cleaning head and the preset trajectory at the end, a PID closed-loop correction model is established to ensure that the trajectory tracking error is ≤2mm.

[0105] 2) Constant contact force control: During chassis movement, the energy-saving mode fine-tunes the boom height, while the flexible mode uses an admittance control law to ensure the end-effector normal contact force. Stablize; 3) Synchronous Adaptation of Cleaning Parameters: The vacuum motor power is adjusted in real time according to the chassis movement speed to avoid missed areas. The outputs of this step are trajectory correction commands, contact force control commands, and vacuum power adaptation commands.

[0106] 3.5 Complete the replacement verification and exit the linkage mechanism The input for this step is the real-time pose of the chassis. Real-time joint travel percentage 1. Cleaning area coverage; the core control logic is as follows: 1) After the chassis moves to the target pose, check whether the percentage of all joint travel has fallen back to the optimal range of 40%-60%. If it meets the requirement, the replacement is considered complete. 2) After filling in the gaps, place the replacement part in the middle. The chassis position is stopped and locked, and the robotic arm continues to execute the cleaning trajectory; 3) If the target area is not fully covered and the extreme warning is triggered again, repeat steps 3.1-3.4 to achieve uninterrupted continuous cleaning.

[0107] The output of this step is the linkage exit command, the chassis lock command, and the loop execution flag.

[0108] 3.6 Safety Boundary and Obstacle Avoidance Redundancy Control The inputs for this stage are a global semantic map, real-time radar obstacle detection data, and chassis pose. The core control logic for this stage is as follows: 1) Before chassis replacement, the path passability is verified by multi-mode radar. If rigid obstacles are found, replacement is stopped and the path is replanned. 2) Constrain the chassis's positioning range to not exceed the passable area of ​​the global map to avoid falls and collisions; 3) When dynamic obstacles such as people or pets are detected on the path, the chassis will immediately stop and the robotic arm will retract to a safe position. Operation will resume after the obstacle is removed.

[0109] The output of this stage is a path replanning command and an emergency stop protection command.

[0110] 4. Overall closed-loop control execution process According to a specific embodiment of the present invention, all the methods involved in the present invention can be integrated to form a complete closed loop integrating perception, decision-making, control, execution, and verification. The overall control process can be as follows: Figure 17 As shown, the overall execution process is divided into 8 stages: Global initialization: The robot starts up and constructs a global ternary semantic map through multimodal radar, completing the loading of the AI ​​vision model and the zero-point calibration of the robotic arm; Target area navigation: Using the chassis optimal docking navigation method, the robot is navigated to the initial optimal docking position of the target furniture and the initial position of the chassis is locked. Local perception and recognition: The robotic arm unfolds and collects data on the working surface through end sensors to identify material type, surface curvature, contaminant distribution and obstacle information; Control mode matching: Perform multi-dimensional verification of energy-saving mode. If the conditions are met, switch to the forearm power-off gravity bonding mode; otherwise, switch to the flexible admittance control mode. Cleaning trajectory execution: A cleaning trajectory is generated, the robotic arm performs the cleaning operation, and the travel limit is predicted in real time; Linkage and follow control: If the travel limit linkage flag is triggered, the arm-chassis linkage and follow algorithm is executed, and the chassis fills in the gap in real time, realizing cleaning while moving; Cleaning effect verification: The cleaning effect is re-inspected in real time by the end vision. If the residue rate exceeds the standard, a second cleaning is performed. If the standard is met, the trajectory continues to be executed. Operation completed and exited: After the target area is cleaned, the execution mode exit process is completed, the robotic arm is retracted, and the chassis navigates to the next cleaning area or returns to the charging position.

[0111] The embodiments described above are merely some preferred embodiments of the present invention, and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A control method for a robotic arm-type intelligent furniture cleaning robot based on multimodal perception, characterized in that, The method includes multimodal radar fusion modeling and curvature semantic localization method adapted to the robotic arm. The robotic arm-type intelligent furniture cleaning robot is a three-degree-of-freedom robotic arm with a rotary joint, an upper arm telescopic joint and a lower arm telescopic joint. Its chassis is equipped with LiDAR and millimeter-wave radar. The LiDAR is used to build the basic geometric boundary framework of the environment, and the millimeter-wave radar is used to penetrate and detect the space occupancy behind the obstruction. The fusion modeling constructs a ternary semantic map that considers the distinction between soft and hard obstacles. During the robot localization process, curvature semantic classification is performed on the LiDAR point cloud to extract key parameters of the surface to be worked on, which are then used by the robot to adjust its work posture.

2. The control method for a robotic arm-type intelligent furniture cleaning robot based on multimodal perception according to claim 1, characterized in that, The fusion modeling construction of a ternary semantic map that considers the distinction between soft and hard obstacles includes establishing transparency attribute labeling rules: when the lidar feedback area is "occupied" but the millimeter-wave radar feedback area is "idle", the area is labeled with the semantic tag "soft occlusion"; when both feedback is "occupied", it is labeled as "rigid obstacle"; when both feedback is "idle", it is labeled as "accessible / workable space", forming a ternary semantic map of "rigid obstacle-soft occlusion-idle space" and matching it with differentiated cleaning strategies.

3. The control method for a robotic arm-type intelligent furniture cleaning robot based on multimodal perception according to claim 1, characterized in that, After distortion correction of the LiDAR, feature values ​​are extracted based on the coordinate components of the laser point and its neighboring points to quantify the local curvature. Based on the magnitude of the feature values, geometric semantic classification is performed on the surface of the home environment to distinguish between planar, curved, and corner areas. Normal vectors of planar and curved surfaces are extracted. During the robot's coordinate positioning process, the geometric semantic label and normal vector of the current working surface are output simultaneously to guide the cleaning head at the end of the robot arm to adjust its working posture.

4. The control method for a robotic arm-type intelligent furniture cleaning robot based on multimodal perception according to claim 1, characterized in that, The method also includes AI visual recognition technology to assist robot cleaning, specifically including: acquiring an image of the surface to be cleaned, preprocessing the image and inputting it into a multi-task recognition network model, wherein the multi-task recognition network model uses depthwise separable convolution to extract features from the input image to obtain multi-scale feature maps; After at least one feature extraction stage, each feature channel is weighted using a channel attention mechanism; the multi-scale feature map is fused using a feature pyramid network and a path aggregation network to obtain a deep feature vector; the deep feature vector is then concatenated with a multi-dimensional physical feature vector directly extracted from the preprocessed image to obtain a fused feature; wherein, the multi-dimensional physical feature vector includes: RGB channel mean, RGB channel variance, image entropy, contrast, correlation, aspect ratio of the bounding rectangle of the object, circularity of the object, actual area of ​​the object, pollutant distribution density, material gloss, and the second and third moments of the color moments; based on the fused feature, material classification results, object detection results, and surface tilt angle estimation results are output in parallel.

5. The control method for a robotic arm-type intelligent furniture cleaning robot based on multimodal perception according to claim 4, characterized in that, Acquire the RGB image and depth image of the surface to be processed, and preprocess the images, including: The RGB image and the depth image are coordinate registered to generate a pseudo-color depth map. A depth threshold is used to distinguish between the surface-attached target and the background, and candidate target areas are initially screened. Bilateral filtering is used to reduce noise in the image, and morphological dilation is used to enhance the salience of small components; Perspective correction is performed on the image based on gyroscope tilt data, and effective regions of interest are extracted through contour detection and area thresholding.

6. The control method for a robotic arm-type intelligent furniture cleaning robot based on multimodal perception according to claim 4, characterized in that, The multi-task recognition network model is trained in the following way: The training dataset includes sample images, each of which is labeled with a material category label, a foreign object label, and an angle label. The material category label is selected from different categories of furniture materials. The angle label includes various angles. The foreign object label includes hard, fragile, flexible, and contaminant categories. Among them, the hard, fragile, and flexible categories are all items that need to be avoided, and the contaminant category is contaminants that need to be removed. For the material classification branch of the multi-task recognition network model, the Focal loss function is used, the EIOU loss function is used for the item detection branch, and the MSE loss function is used for furniture surface tilt angle estimation.

7. The control method for a robotic arm-type intelligent furniture cleaning robot based on multimodal perception according to claim 1, characterized in that, The method also includes a forearm power-off gravity-adhesion energy-saving and cleaning control mode, specifically including: Step 1: Obtain the material type, curvature, contaminant distribution density, and obstacle information of the surface to be worked on. When the material type belongs to a preset low-damage-risk material set, the curvature is less than the curvature threshold, the contaminant distribution density is less than the density threshold, and the obstacle information meets the preset safety conditions, the energy-saving mode is triggered. Step 2: Plan the reference pose of the robotic arm end effector parallel to the working surface, obtain the reference stroke of the robotic arm extension joint through inverse kinematics calculation, and drive the arm to the reference stroke and lock it. Step 3: De-energize the drive motor of the robotic arm's forearm extension joint, release the axial degree of freedom of the forearm, and let the forearm fall naturally under the action of gravity until the end cleaner contacts the working surface, establishing a stable normal contact force; Step 4: After the contact force stabilizes, start the cleaning function, and complete the cleaning trajectory by coordinating the rotation of the robotic arm's rotary joint with the movement of the chassis, during which the forearm motor remains de-energized. Step 5: After cleaning is complete, power on the forearm motor to retract the forearm, raise the upper arm, and exit the energy-saving mode.

8. The control method for a robotic arm-type intelligent furniture cleaning robot based on multimodal perception according to claim 1, characterized in that, The method also includes a continuous cleaning mode with arm-chassis linkage triggered by travel limit, specifically including: Step 1): Calculate the stroke percentage of each joint of the robotic arm in real time. When the stroke percentage of any joint reaches the first warning threshold, trigger the warning and start pre-planning the path; when it reaches the second linkage threshold, trigger the linkage mode. Step 2): Based on the type of joint reaching its limit and the deviation between the target path point and the maximum reachable range of the robotic arm, determine the compensation direction and compensation distance of the chassis so that the stroke ratio of each joint of the robotic arm returns to the preset optimal range after compensation. Step 3): Establish a synchronous matching model with constant absolute end velocity. Solve the joint compensation speed by using the Jacobian matrix pseudo-inverse to control the coordination between the chassis movement speed and the robot arm joint speed, so that the end cleaner maintains a constant linear velocity during chassis movement. Step 4): After the chassis moves to the target position, the travel percentage is checked and returned to the optimal range. The linkage mode is exited and the robotic arm continues to execute the cleaning trajectory. If the travel limit is triggered again, the above steps are repeated to achieve uninterrupted continuous cleaning.

9. The control method for a robotic arm-type intelligent furniture cleaning robot based on multimodal perception according to claim 8, characterized in that, In step 2), determining the chassis's compensation direction based on the joint type that has reached its limit includes: establishing a chassis coordinate system with the origin at the chassis's geometric center, the X-axis representing the robot's forward / backward direction (vertical), and the Y-axis representing the robot's left / right direction (lateral); when the joint that has reached its limit is a rotary joint, the chassis is determined to perform lateral translation compensation along the Y-axis; when the joint that has reached its limit is a telescopic joint, the chassis is determined to perform forward / backward translation compensation along the X-axis.

10. The control method for a robotic arm-type intelligent furniture cleaning robot based on multimodal perception according to claim 8, characterized in that, Step 3) specifically involves: based on the real-time speed of the chassis. Joint compensation speed is calculated by solving the pseudo-inverse of the Jacobian matrix. To ensure the end speed Constant: In the formula As the pseudo-inverse of the Jacobian matrix, the S-shaped velocity curve is used to plan the chassis acceleration and deceleration.

Images