Intelligent cleaning method of sweeping robot based on image recognition

Abstract

The invention relates to a sweeping robot intelligent cleaning method based on image recognition, and the method comprises the steps: obtaining an RGB image and depth sensor data through an image sensor, extracting a texture feature, a color transition feature and a depth mutation feature, and generating a first multi-mode feature set; fusing the first multi-modal feature set by adopting a multi-head attention mechanism, and generating a second multi-modal feature set according to the illumination intensity distribution and the material reflectivity distribution; processing the drop risk label set through time sequence analysis, and fusing the illumination change characteristics to generate stable drop risk distribution; extracting a high-risk area from the stabilized drop risk distribution, generating an obstacle distribution diagram, and generating a motion control instruction set based on a path planning algorithm; and executing navigation adjustment according to the motion control instruction set, fusing the material boundary characteristics to correct the depth deviation, and generating an avoidance path.

Description

Technical Field

[0001] The present application relates to the field of graphic data technology, and in particular to an intelligent cleaning method of a sweeping robot based on image recognition. Background Art

[0002] Robot vacuums face unique visual recognition challenges when cleaning mixed carpeted and wooden floors. Sudden depth changes at the intersection of carpet and wooden flooring can easily be misjudged as a continuous surface. Differences in reflectivity between the two materials, such as dark carpets absorbing light and wooden flooring having high reflectivity, can lead to deviations in depth sensor data. Furthermore, the contrast between the rough texture of carpets and the regular texture of wooden flooring can interfere with edge detection. These factors combine to complicate fall risk identification.

[0003] During the fusion process, the multi-head attention mechanism needs to consider the strong absorption of light by dark floors and the reflection differences between different materials, so as to accurately identify edge depth mutations and texture continuity. At the same time, the stability of the fall risk identification results is affected by natural light fluctuations and local shadows, and needs to be optimized through time series analysis. After obtaining indications of steep stair edges or slightly convex thresholds, how to effectively integrate this information into the A-star path planning algorithm and generate highly adaptable speed and angle control instructions is the core issue in achieving safe navigation.

[0004] In addition, when performing obstacle avoidance operations, it is also necessary to consider the interference of special materials such as metal, wood and glass on sensor data, and correct positioning errors in real time to ensure that the robot can accurately perform fall edge avoidance operations. Summary of the Invention

[0005] In order to solve the problems existing in the above-mentioned prior art, the purpose of this application is to provide an intelligent cleaning method of a sweeping robot based on image recognition.

[0006] This application addresses the difficulty of visual recognition in mixed carpet and wooden floor areas and provides an intelligent cleaning method for a sweeping robot based on image recognition, including:

[0007] S101, extracting texture contrast features and color transition region features at the interface between the carpet and the wooden floor from the RGB image using a convolutional neural network, and extracting depth mutation features at the interface using depth sensor data to generate a first multimodal feature set;

[0008] S102. Using a multi-head attention mechanism to fuse the first multimodal feature set, obtain the light intensity distribution based on the RGB image brightness analysis, obtain the material reflectance distribution based on the texture features, generate an attention allocation matrix, enhance the edge depth weight and optimize the texture continuity for dark ground areas with brightness below a preset threshold, and generate a second multimodal feature set;

[0009] Among them, a multi-head attention mechanism is used to fuse the first multimodal feature set, including calculating the weight distribution matrix of each modal feature through a multi-head attention module, wherein the depth feature weight is enhanced for the dark carpet area and the texture continuity weight is enhanced for the wooden floor area;

[0010] S103. Extract edge depth mutation features from the second multimodal feature set, calculate their values and compare them with a dynamic depth threshold. The dynamic depth threshold estimates the home fall height range of 10 to 20 centimeters based on the depth sensor data. If the height is lower than the threshold, perform binary classification by fusing color transition region features and texture continuity features through a fully connected layer to generate a fall risk label set.

[0011] S104, analyzing the continuous frame sequence of the fall risk label set through a pre-trained recurrent neural network, detecting dynamic fluctuations of natural light and local shadows based on the light intensity distribution, and smoothing the time series using a Kalman filter to generate a stabilized fall risk distribution;

[0012] S105. Extract high-risk areas from the stabilized fall risk distribution, apply the Canny edge detection algorithm to locate stair edges and threshold convexities, generate an obstacle distribution map, and generate a motion control instruction set using the A-star path planning algorithm combined with a path cost function. The path cost function is calculated based on the distance and the obstacle distribution map weight. The motion control instruction set includes speed parameters and angle parameters.

[0013] S106. Extract speed parameters and angle parameters from the motion control instruction set, perform navigation adjustments through the embedded control system, combine depth sensor data with the smoothness characteristics of the material interface to correct the depth deviation of the wood and carpet materials, use a real-time path update algorithm to correct the positioning error, and complete the fall edge avoidance operation.

[0014] Preferably, in step S101, the RGB image and depth sensor data are acquired by an image sensor, texture features, color transition features, and depth mutation features are extracted, and a first multimodal feature set is generated, including:

[0015] The convolutional neural network is used to extract the roughness features of carpet texture, regularity features of wooden floor texture and color transition area features from RGB images.

[0016] The depth sensor data is integrated to extract the depth mutation feature of the falling edge and the smoothness feature of the material interface, and generate the first multimodal feature set;

[0017] Principal component analysis is used to reduce the dimensionality of the first multimodal feature set to obtain the second feature set;

[0018] If the variance of the feature vector of the second feature set is greater than a preset threshold, the feature vector is classified using the K-means clustering algorithm to obtain a feature classification result;

[0019] According to the feature classification result, a template matching method is used to generate a first region segmentation map;

[0020] If there is a boundary overlap region in the first region segmentation map, a second region segmentation map is generated through morphological processing;

[0021] The second region segmentation map is deeply verified using depth sensor data to generate a final multimodal segmentation map.

[0022] Preferably, in step S102, a multi-head attention mechanism is used to fuse the first multimodal feature set, and a second multimodal feature set is generated according to the light intensity distribution and the material reflectance distribution, including:

[0023] Obtain light intensity distribution and texture features through RGB image decomposition to generate an initial multimodal feature set;

[0024] A multi-head attention mechanism is used to fuse the initial multimodal feature set and construct an allocation matrix to determine the weight distribution;

[0025] If the light intensity is lower than the preset threshold, the dark ground area is extracted and the edge depth weight is enhanced to obtain enhanced features;

[0026] Analyze the material reflectance distribution according to the texture features, optimize the texture continuity of the enhanced features, and generate a second multimodal feature set;

[0027] Extracting deep features of the second multimodal feature set through a convolutional neural network to generate a feature vector;

[0028] If the texture continuity of the feature vector is lower than the preset standard, the allocation matrix is iteratively optimized to obtain the optimized feature set;

[0029] Generate the final multimodal feature representation based on the optimized feature set.

[0030] Preferably, in step S103, edge depth mutation features are extracted from the second multimodal feature set, compared with the dynamic depth threshold, and the color transition features and texture continuity features are integrated to generate a fall risk label set, including:

[0031] Acquire an original depth image from the depth sensor data, remove noise using a preprocessing algorithm, and obtain a first depth feature set;

[0032] Extract edge depth mutation features from the first depth feature set, calculate the mutation value, and obtain an edge mutation feature set;

[0033] The dynamic depth threshold is calculated using the dynamic depth threshold formula T_d = f(D_s, H_r), where T_d represents the dynamic depth threshold, D_s represents the depth sensor data, and H_r represents the home fall height range;

[0034] If the value of the edge mutation feature set is lower than T_d, a low-risk preliminary label set is generated;

[0035] Extract color transition features and texture continuity features from the low-risk preliminary label set, and fuse them through a fully connected layer to generate a fused feature set;

[0036] The fused feature set is classified by a binary classifier to generate a fall risk label set.

[0037] Preferably, in step S104, the fall risk label set is processed by time series analysis, and the illumination change feature is integrated to generate a stabilized fall risk distribution, including:

[0038] Obtain continuous frame sequences from the video stream, use the inter-frame difference method to extract dynamic change features, and obtain time series change data;

[0039] By processing time-series changing data through a pre-trained recurrent neural network, the fall risk label set is analyzed to obtain a preliminary fall risk distribution.

[0040] If the variance of the preliminary fall risk distribution exceeds a preset threshold, the Kalman filter is used to smooth the time series to obtain a smoothed risk distribution;

[0041] Based on the smoothed risk distribution, the gradient analysis method is used to extract the dynamic fluctuation of natural light and local shadow features to obtain the illumination change feature set.

[0042] The enhanced risk distribution is generated by fusing the illumination change feature set and the fall risk distribution through a convolutional neural network.

[0043] If the risk value of a local area in the enhanced risk distribution is higher than the preset threshold, the area is weighted to generate the final stabilized fall risk distribution.

[0044] Preferably, in step S105, extracting high-risk areas from the stabilized fall risk distribution, generating an obstacle distribution map, and generating a motion control instruction set based on a path planning algorithm include:

[0045] Obtain original point cloud data from environmental sensors, downsample it using a voxel filtering algorithm to obtain downsampled point cloud data;

[0046] The point cloud normal vector and height difference are calculated by downsampling the point cloud data. If the angle between the normal vector and the vertical direction is greater than the preset threshold and the height difference exceeds the preset threshold, it is marked as a high-risk area;

[0047] The Canny edge detection algorithm is used to process the image data corresponding to the high-risk area, identify the edges of the stairs and the convex corners of the threshold, and generate an edge feature map;

[0048] Generate an obstacle distribution map by fusing the high-risk area distribution based on the edge feature map;

[0049] The optimal path is calculated based on the obstacle distribution map through the A-star path planning algorithm, where the path cost function C(p) = w_dD(p) + w_oO(p), D(p) represents the path distance, O(p) represents the obstacle weight, w_d and w_o are the distance and obstacle weight coefficients respectively, and a motion control instruction set is generated.

[0050] Preferably, in step S106, performing navigation adjustment according to the motion control instruction set, fusing material boundary features to correct depth deviation, and generating a fall edge avoidance path includes:

[0051] Obtain speed parameters and angle parameters from the motion control instruction set, perform adjustments through the navigation module, and obtain a preliminary navigation path;

[0052] The depth sensor collects environmental data, detects the depth change at the interface between wood and carpet materials, and obtains the material interface characteristics;

[0053] If the smoothed feature value of the material boundary feature exceeds a preset threshold, the depth deviation of the wood and carpet materials is corrected to obtain the corrected depth data;

[0054] The Kalman filter algorithm is used to fuse the corrected depth data and the navigation path. If the positioning error exceeds the preset threshold, the path parameters are updated to obtain the optimized path.

[0055] By optimizing the path and adjusting the speed and angle parameters, a falling edge avoidance instruction is generated to obtain an avoidance path;

[0056] If the final navigation result shows that the risk of falling still exists, the depth sensor is used to recollect data and repeat the correction and path update.

[0057] The intelligent cleaning method of a sweeping robot based on image recognition described in this application has the advantages of extracting ground texture, color and depth features by fusing RGB images and depth sensor data, generating a multimodal feature set, using a multi-head attention mechanism to optimize feature fusion, enhancing edge depth weights for dark ground, combining dynamic depth thresholds and texture continuity to classify fall risks, and analyzing continuous frame sequences through a recurrent neural network to generate a stabilized risk distribution. Finally, edge detection is applied to locate obstacles, and a safe path is planned in combination with the A-star algorithm. Navigation adjustments are performed through an embedded system to achieve intelligent fall avoidance in a home environment.

[0058] The present invention effectively solves the difficulty of visual recognition to prevent falls when the sweeping robot is cleaning a mixed area of carpeted and wooden floors, as well as the problem of identifying fall risks under complex lighting and diverse ground material conditions, thereby improving the safety and adaptability of the robot in the home environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is the process of an intelligent cleaning method of a sweeping robot based on image recognition described in this application Figure 1 ;

[0060] Figure 2 This is the process of an intelligent cleaning method of a sweeping robot based on image recognition described in this application Figure 2 . DETAILED DESCRIPTION

[0061] like Figure 1-Figure 2 As shown, the present application describes an intelligent cleaning method for a sweeping robot based on image recognition, comprising:

[0062] S101, extracting texture contrast features and color transition region features at the interface between the carpet and the wooden floor from the RGB image using a convolutional neural network, and extracting depth mutation features at the interface using depth sensor data to generate a first multimodal feature set;

[0063] S102. Using a multi-head attention mechanism to fuse the first multimodal feature set, obtain the light intensity distribution based on the RGB image brightness analysis, obtain the material reflectance distribution based on the texture features, generate an attention allocation matrix, enhance the edge depth weight and optimize the texture continuity for dark ground areas with brightness below a preset threshold, and generate a second multimodal feature set;

[0064] S103. Extract edge depth mutation features from the second multimodal feature set, calculate their values and compare them with a dynamic depth threshold. The dynamic depth threshold estimates the home fall height range of 10 to 20 centimeters based on the depth sensor data. If the height is lower than the threshold, perform binary classification by fusing color transition region features and texture continuity features through a fully connected layer to generate a fall risk label set.

[0065] S104, analyzing the continuous frame sequence of the fall risk label set through a pre-trained recurrent neural network, detecting dynamic fluctuations of natural light and local shadows based on the light intensity distribution, and smoothing the time series using a Kalman filter to generate a stabilized fall risk distribution;

[0066] S105. Extract high-risk areas from the stabilized fall risk distribution, apply the Canny edge detection algorithm to locate stair edges and threshold convexities, generate an obstacle distribution map, and generate a motion control instruction set using the A-star path planning algorithm combined with a path cost function. The path cost function is calculated based on the distance and the obstacle distribution map weight. The motion control instruction set includes speed parameters and angle parameters.

[0067] S106. Extract speed parameters and angle parameters from the motion control instruction set, perform navigation adjustments through the embedded control system, combine depth sensor data with the smoothness characteristics of the material interface to correct the depth deviation of the wood and carpet materials, use a real-time path update algorithm to correct the positioning error, and complete the fall edge avoidance operation.

[0068] like Figure 1-Figure 2 As shown, in step S101, the roughness features of the carpet texture, the regularity features of the wooden floor texture, and the color transition area features are extracted from the RGB image through a convolutional neural network, and the depth mutation features of the falling edge and the smoothness features of the material boundary are extracted in combination with the depth sensor data to generate a first multimodal feature set.

[0069] Furthermore, in step S101, texture contrast features and color transition region features at the interface between the carpet and the wooden floor are extracted from the RGB image using a convolutional neural network, and depth mutation features at the interface are extracted in combination with depth sensor data to generate a first multimodal feature set.

[0070] The RGB image decomposition algorithm is used to obtain the light intensity distribution and texture features to generate the initial multimodal feature set;

[0071] The initial multimodal feature set is fused through the multi-head attention mechanism, the allocation matrix is constructed, the weight distribution is determined, and the enhanced features are obtained;

[0072] If the light intensity is lower than the preset threshold, the dark ground area is extracted and the edge depth weight is enhanced to obtain enhanced features;

[0073] If the illumination intensity is higher than or equal to the preset threshold, the original weight is retained and the enhanced feature is obtained;

[0074] Analyze the material reflectance distribution based on the enhanced features, optimize the texture continuity, and generate a second multimodal feature set;

[0075] Extracting deep features of the second multimodal feature set through a convolutional neural network to generate a feature vector;

[0076] If the texture continuity of the feature vector is lower than the preset standard, the allocation matrix is iteratively optimized and the weight enhancement is adjusted to obtain the optimized feature set;

[0077] If the texture continuity of the feature vector is higher than or equal to the preset standard, the optimized feature set is directly output;

[0078] Generate the final multimodal feature representation based on the optimized feature set, and output the feature set that integrates illumination and texture;

[0079] The original depth image is obtained through the depth sensor data, and the noise is removed by using a preprocessing algorithm to obtain the first depth feature set.

[0080] Specifically, in step S101, a convolutional neural network is used to extract the roughness features of the carpet texture from the RGB image. The VGG16 network structure is used to extract the local contrast features of the carpet texture, and its roughness value is calculated to be 0.85. At the same time, the regularity features of the wooden floor texture are extracted, and its periodicity is analyzed using a Gabor filter, and the regularity is calculated to be 0.92. The color transition area features are also extracted, and the Canny edge detection algorithm is used to obtain the color gradient change area.

[0081] Combined with the depth sensor data, the Sobel operator is used to extract the depth mutation characteristics of the drop edge, and the mutation value is calculated to be 15mm. The smoothness characteristics of the material interface are also extracted. The Gaussian smoothing filter algorithm is used to calculate the smoothness to be 0.78, generating the first multimodal feature set.

[0082] Using the RGB image decomposition algorithm, the light intensity distribution is decomposed through the Retinex theory, the light intensity is calculated to be 50 lux, and texture features are extracted to generate the initial multimodal feature set;

[0083] The initial multimodal feature set is fused through the multi-head attention mechanism, and the distribution matrix is constructed using the Softmax function. The weight distribution is determined to be [0.4, 0.3, 0.3] to obtain enhanced features.

[0084] If the light intensity is lower than the preset threshold of 60 lux, the dark ground area is extracted, and the edge depth weight is enhanced using histogram equalization. The calculated weight value is 0.6 to obtain the enhanced feature;

[0085] If the illumination intensity is higher than or equal to the preset threshold, the original weight is retained and the enhanced feature is obtained;

[0086] The material reflectance distribution is analyzed based on the enhanced features, and the K-means clustering algorithm is used to optimize the texture continuity to generate the second multimodal feature set;

[0087] The deep features of the second multimodal feature set are extracted through a convolutional neural network, and a ResNet50 network structure is used to generate a feature vector;

[0088] If the texture continuity of the feature vector is lower than the preset standard of 0.8, the allocation matrix is iteratively optimized and the weight enhancement is adjusted using the gradient descent method to obtain the optimized feature set;

[0089] If the texture continuity of the feature vector is higher than or equal to the preset standard, the optimized feature set is directly output;

[0090] Generate the final multimodal feature representation based on the optimized feature set, and output the feature set that integrates illumination and texture;

[0091] The original depth image is obtained through the depth sensor data, and the median filtering algorithm is used to remove noise to obtain the first depth feature set.

[0092] like Figure 1-Figure 2 As shown, in step S102, a multi-head attention mechanism is used to fuse the first multimodal feature set, the light intensity distribution is obtained according to the RGB image brightness analysis, the material reflectance distribution is obtained according to the texture features, and an attention allocation matrix is generated. The edge depth weight is enhanced and the texture continuity is optimized for dark ground areas with brightness lower than a preset threshold to generate a second multimodal feature set.

[0093] Furthermore, in step S102, the illumination intensity distribution and texture features are obtained by decomposing the RGB image to generate an initial multimodal feature set;

[0094] A multi-head attention mechanism is used to fuse the initial multimodal feature set, construct the attention allocation matrix, and determine the weight distribution;

[0095] If the light intensity is lower than the preset threshold, the dark ground area is extracted and the edge depth weight is enhanced to obtain enhanced features;

[0096] Analyze the material reflectance distribution according to the texture features, optimize the texture continuity of the enhanced features, and generate a second multimodal feature set;

[0097] Extracting deep features of the second multimodal feature set through a convolutional neural network to generate a feature vector;

[0098] If the texture continuity of the feature vector is lower than the preset standard, the allocation matrix is iteratively optimized and the weight enhancement is adjusted to obtain the optimized feature set;

[0099] Perform principal component analysis and dimensionality reduction processing based on the optimized feature set, retain the main feature components, and generate a reduced dimensionality feature set;

[0100] If the variance of the feature vector of the reduced dimension feature set is greater than the preset threshold, the feature vector is classified using the K-means clustering algorithm to obtain the feature classification result;

[0101] According to the feature classification results, the template matching method is used to perform region segmentation and generate the final multimodal segmentation map.

[0102] Specifically, in step S102, the illumination component and the reflection component are separated by using the Retinex theory through RGB image decomposition, the brightness threshold T is set to 50 lux, and the texture features are extracted by using the LBP operator to calculate the local binary pattern to generate an initial multimodal feature set including the illumination intensity distribution and texture features;

[0103] A multi-head attention mechanism is used to fuse the initial feature set. The number of attention heads is set to N = 8. The weight distribution matrix is calculated by the Softmax function to determine the weight distribution of each modal feature.

[0104] The multi-head attention mechanism adopts a multi-head attention structure, sets 8 attention heads, and generates a weight distribution matrix through the Softmax function;

[0105] For dark carpet areas (RGB brightness <50 lux), the gradient weight of the depth sensor data is increased to 1.5 times to compensate for the depth error caused by light absorption;

[0106] For wooden floor areas (texture regularity > 0.9), the texture continuity weight is increased to 1.2 times to enhance edge smoothness judgment;

[0107] If an area with light intensity lower than T is detected, Canny edge detection is used to extract the dark ground contour, and the gradient value of the depth sensor data is multiplied by the weight coefficient α=1.5 to enhance the edge depth feature;

[0108] The material reflectance is calculated based on the texture features using the Phong illumination model, and the texture continuity of the features is enhanced through bicubic interpolation optimization to output the second multimodal feature set.

[0109] The ResNet50 network is used to extract the deep features of the second feature set, and the last fully connected layer outputs a 1024-dimensional feature vector;

[0110] If the SSIM texture similarity of the feature vector is lower than the threshold of 0.7, the attention matrix weight is adjusted by backpropagation, and the number of iterations K = 3 times to obtain the optimized feature set;

[0111] Perform PCA dimensionality reduction on the optimized feature set, retain the principal components with cumulative contribution rate ≥ 95%, and generate a 20-dimensional reduced dimension feature set;

[0112] If the feature set variance exceeds the threshold σ2 = 0.1, the K-means clustering algorithm is used to set the number of clusters K = 3, the Euclidean distance is used as the similarity measure, and the material classification label is output;

[0113] The Sobel operator is used to perform template matching segmentation according to the classification label, and the morphological closing operation is combined to eliminate regional gaps to generate the final multimodal segmentation map.

[0114] like Figure 1-Figure 2 As shown, in step S103, the edge depth mutation feature is extracted from the second multimodal feature set, and its value is calculated and compared with the dynamic depth threshold. The dynamic depth threshold estimates the home fall height range of 10 to 20 cm based on the depth sensor data. If it is lower than the threshold, the color transition area feature and the texture continuity feature are fused through the fully connected layer to perform binary classification to generate a fall risk label set.

[0115] Furthermore, in step S103, the first multimodal feature set is subjected to dimensionality reduction processing by a principal component analysis method, and the main feature components are retained to obtain a second multimodal feature set;

[0116] Extracting edge depth mutation features from the second multimodal feature set, calculating their mutation values, and obtaining an edge mutation feature set;

[0117] According to the depth sensor data and the home falling height range of 10 to 20 cm, the dynamic depth threshold is calculated by the dynamic depth threshold formula T_d = f(D_s, H_r) to obtain the dynamic depth threshold T_d;

[0118] If the value of the edge mutation feature set is lower than the dynamic depth threshold T_d, the corresponding color transition area features and texture continuity features are obtained from the second multimodal feature set to obtain a preliminary fusion feature set;

[0119] The preliminary fusion feature set is fused through the fully connected layer to obtain the fusion feature set;

[0120] A binary classifier is used to classify the fused feature set to obtain the fall risk label set;

[0121] According to the fall risk label set, the area corresponding to the high-risk label is obtained, and the edge depth mutation feature is recalculated using the depth sensor data to obtain the updated edge mutation feature set;

[0122] If the value of the updated edge mutation feature set is lower than the dynamic depth threshold T_d, the area corresponding to the high-risk label is smoothed and optimized through morphological processing to obtain the final low-risk label;

[0123] If the value of the updated edge mutation feature set is higher than the dynamic depth threshold T_d, the updated edge mutation feature set is integrated with the second region segmentation map through feature fusion processing to obtain the final high-risk label.

[0124] Specifically, in step S103, the first multimodal feature set is subjected to dimensionality reduction processing by principal component analysis, and the feature components with a cumulative contribution rate exceeding 95% are retained using the PCA algorithm, and minor features with a variance accounted for less than 5% are eliminated to obtain a second multimodal feature set;

[0125] Extract edge depth mutation features from the second multimodal feature set, calculate the depth gradient using the Sobel operator, set the mutation threshold Δd = 5 cm, and filter pixels with gradient values greater than the threshold to obtain the edge mutation feature set;

[0126] Based on the depth sensor data D_s and the home drop height range H_r = [10cm, 20cm], the dynamic depth threshold formula T_d = 0.8 × max(D_s) + 0.2 × mean(H_r) is used to calculate T_d = 15cm, where D_s is the raw depth data collected by the depth sensor in real time (unit: cm), H_r is the preset home drop height range, max(D_s) is the maximum value of the depth sensor data in the current frame, and mean(H_r) is the average value of H_r.

[0127] If the gradient values of the edge mutation feature set are all lower than T_d, the hue transition region features of the HSV color space (threshold ΔH = 30°) and the LBP texture continuity features (window size 3×3) are extracted from the second multimodal feature set to obtain the preliminary fusion feature set;

[0128] The preliminary fusion feature set is fused through the fully connected layer, the hidden layer dimension is set to 256, the activation function is ReLU, and a 256-dimensional fusion feature vector is obtained;

[0129] The fused feature vector is classified using the SVM binary classifier, the kernel function is RBF, and the penalty coefficient C = 1.0 to obtain the fall risk label set;

[0130] The high-risk area is screened according to the label set, the depth gradient of the area is recalculated, and the updated edge mutation feature set is obtained after removing the noise using bilateral filtering;

[0131] If the maximum gradient value of the updated feature set is lower than T_d, a 3×3 circular structure element is used to perform a morphological closing operation to eliminate small holes and output the final low-risk label;

[0132] If the maximum gradient value exceeds T_d, the updated edge features are weightedly fused with the boundary coordinates of the region segmentation map (weight α = 0.7) to output the final high-risk label.

[0133] like Figure 1-Figure 2As shown, in step S104, the continuous frame sequence of the fall risk label set is analyzed by a pre-trained recurrent neural network, the dynamic fluctuation of natural light and local shadows are detected for the light intensity distribution, and the Kalman filter is used to smooth the time series to generate a stabilized fall risk distribution.

[0134] Furthermore, in step S104, a continuous frame sequence is obtained from the video stream, and a frame difference method is used to extract dynamic change features to obtain time series change data of the frame sequence;

[0135] By processing time-series changing data through a pre-trained recurrent neural network, the fall risk label set is analyzed to obtain a preliminary fall risk distribution.

[0136] If the variance of the preliminary fall risk distribution exceeds a preset threshold, the Kalman filter is used to smooth the time series to obtain a smoothed risk distribution;

[0137] Based on the smoothed risk distribution, the light intensity distribution is detected, and the gradient analysis method is used to extract the dynamic fluctuation of natural light and local shadow features to obtain the light change feature set;

[0138] The illumination change feature set is processed by convolutional neural network and integrated with the fall risk distribution to obtain an enhanced risk distribution.

[0139] If the risk value of a local area in the enhanced risk distribution is higher than the preset threshold, the area is weighted to obtain the final stabilized fall risk distribution;

[0140] According to the final stabilized fall risk distribution, cluster analysis method is used to divide the risk levels and obtain the distribution of high-risk areas;

[0141] Collect environmental data through depth sensors, integrate the distribution of high-risk areas, detect depth change characteristics in the environment, and obtain an environmental risk feature set;

[0142] According to the environmental risk feature set, the conditional random field model is used to optimize the distribution of high-risk areas and obtain the refined risk area distribution.

[0143] Specifically, in step S104, a continuous frame sequence is obtained from the video stream at a sampling rate of 30 fps, the pixel changes of adjacent frames are calculated using a three-frame difference method, a dynamic threshold of 0.15 is set to extract the contour of the moving object, and time-series change data including motion vectors is generated;

[0144] The time series data is input into the LSTM network. The network structure contains 64 hidden units and outputs 6 categories of fall risk labels through the Softmax layer. When the probability distribution variance exceeds 0.25, the Kalman filter is triggered. The process noise Q = 0.1 and the observation noise R = 0.5 are set for smoothing.

[0145] Based on the smoothed risk distribution, the regional illumination gradient is calculated in the HSV color space. The Sobel operator convolution is performed on the luminance channel to detect dynamically fluctuating areas with gradient values greater than 30 lux / frame. Shadow blocks with an area exceeding 15% are extracted.

[0146] The ResNet18 network is used to process shadow features. After feature concatenation with the risk distribution, the features are fused through a 1×1 convolutional layer. The regions with risk values exceeding 0.8 in the output enhanced distribution are weighted 1.5 times.

[0147] The DBSCAN clustering algorithm was used, with a neighborhood radius of ε = 0.3 m and a minimum number of samples of 5 to divide the high-risk clusters;

[0148] Synchronously acquire depth sensor point cloud data and calculate the Z-axis standard deviation within a 2m range. Use bilateral filtering for correction when the depth mutation at the interface between wood and carpet exceeds 50mm.

[0149] The correction data and risk clusters are input into the CRF model, and the potential function is defined to include 10 feature dimensions. After L-BFGS optimization, the risk area boundary with sub-meter accuracy is output.

[0150] like Figure 1-Figure 2 As shown, in step S105, high-risk areas are extracted from the stabilized fall risk distribution, the Canny edge detection algorithm is applied to locate the edge of the stairs and the slightly convex threshold, and an obstacle distribution map is generated. The motion control instruction set is generated according to the A-star path planning algorithm combined with the path cost function. The path cost function is calculated based on the distance and the weight of the obstacle distribution map. The motion control instruction set includes speed parameters and angle parameters.

[0151] Furthermore, in step S105, high-risk areas are extracted from the stabilized fall risk distribution, and the Canny edge detection algorithm is applied to locate the stair edges and threshold convexities to generate an obstacle distribution map;

[0152] According to the obstacle distribution map, the optimal path is calculated using the A-star path planning algorithm combined with the path cost function. The path cost function is calculated based on the distance and the weight of the obstacle distribution map to obtain the motion control instruction set, which includes speed parameters and angle parameters.

[0153] Obtain speed and angle parameters from the motion control instruction set, and perform adjustments through the navigation module in the embedded system to obtain a preliminary navigation path;

[0154] The depth sensor collects environmental data, detects the depth change at the interface between wood and carpet materials, and obtains the material interface characteristics;

[0155] Smooth features are extracted based on the material boundary features. If the smooth feature value exceeds the preset threshold, the depth deviation of the wood and carpet materials is corrected to obtain the corrected depth data;

[0156] The Kalman filter algorithm is used to fuse the corrected depth data and the preliminary navigation path. If the positioning error exceeds the preset threshold, the path parameters are updated to obtain the optimized path.

[0157] Adjust the speed and angle parameters according to the optimized path. If the adjusted parameters match the depth data, generate a fall edge avoidance instruction and obtain an avoidance path.

[0158] Obtain the avoidance path, perform navigation adjustments through the embedded system, determine whether the falling edge is avoided, and obtain the final navigation result;

[0159] If the final navigation result shows that the risk of falling still exists, the depth sensor is used to recollect data, repeat the correction and path update, and obtain a new navigation path.

[0160] Specifically, in step S105, high-risk areas are extracted from the stabilized fall risk distribution, and areas with a risk threshold greater than 0.8 are set as high-risk areas. The Canny edge detection algorithm is applied with a Gaussian filter kernel size of 5×5 and a high-low threshold ratio of 1:3 to detect stair edges and threshold convexities, generating a binary obstacle distribution map.

[0161] Based on the obstacle distribution map, the A-star path planning algorithm is used. The heuristic function is set to Manhattan distance, and the path cost function weight coefficients are 0.6 for distance and 0.4 for obstacle density. The optimal path is calculated and a motion control instruction set containing speed parameters (0.5m / s) and angle parameters (30°) is output.

[0162] The embedded system navigation module parses the instruction set and uses the PID controller to adjust the motor speed and steering angle to generate a preliminary navigation path.

[0163] The depth sensor collects environmental point cloud data at a frequency of 30Hz, detects the interface between wood and carpet materials based on a Z-axis height difference exceeding 2cm, and extracts the depth mutation characteristics at the interface;

[0164] Gaussian smoothing is performed on the material boundary features with a window size of 7×7. If the standard deviation after smoothing exceeds the threshold of 0.5, linear interpolation is used to correct the depth deviation and the corrected depth data is output;

[0165] The Kalman filter algorithm fuses the corrected depth and navigation path, and the predicted position covariance is set to 0.1. If the positioning error exceeds 10 cm, the path node coordinates are updated to generate an optimized path;

[0166] The speed parameters (0.3 m / s) and angle parameters (45°) are recalculated based on the optimized path. If the matching error with the real-time data from the depth sensor is less than 5%, an avoidance path including an emergency stop command is generated.

[0167] After the embedded system executes the avoidance path, it detects whether the obstacle distance is greater than the safety threshold of 50cm and outputs the final navigation result;

[0168] If it is detected that the fall risk still exists, the depth sensor is triggered to re-collect data, and the Kalman filter and path update are iteratively executed until a risk-free navigation path is output.

[0169] like Figure 1-Figure 2 As shown, in step S106, speed parameters and angle parameters are extracted from the motion control instruction set, navigation adjustment is performed through the embedded control system, and the depth deviation of the wood and carpet materials is corrected according to the smooth characteristics of the material interface combined with the depth sensor data. The real-time path update algorithm is used to correct the positioning error and complete the fall edge avoidance operation.

[0170] Furthermore, in step S106, speed parameters and angle parameters are extracted from the motion control instruction set, and preliminary adjustments are performed through the navigation module of the embedded control system to obtain a preliminary navigation path;

[0171] The depth sensor collects environmental data and detects the depth change at the interface between wood and carpet to obtain the material interface characteristics. Smooth features are extracted based on the material interface characteristics. If the smooth feature value exceeds the preset threshold, the depth deviation of the wood and carpet materials is corrected to obtain the corrected depth data.

[0172] The Kalman filter algorithm is used to fuse the corrected depth data and the preliminary navigation path. If the positioning error exceeds the preset threshold, the path parameters are updated to obtain the optimized path.

[0173] By optimizing the path, the speed and angle parameters are adjusted. If the adjusted parameters match the corrected depth data, a fall edge avoidance instruction is generated to obtain an avoidance path.

[0174] Obtain the avoidance path, perform navigation adjustments through the embedded control system, determine whether the falling edge is avoided, and obtain the final navigation result;

[0175] If the final navigation result shows that the risk of falling still exists, the environment data is collected again through the depth sensor, and the correction and path update are repeated to obtain a new navigation path;

[0176] Extracting an original depth image from the depth sensor data corresponding to the new navigation path, removing noise using a preprocessing algorithm, and obtaining a first depth feature set;

[0177] Extract the edge depth mutation feature from the first depth feature set, calculate its mutation value, calculate the dynamic depth threshold through the dynamic depth threshold formula T_d = f(D_s, H_r), compare the mutation value with T_d, and if it is lower than T_d, determine the final low-risk label.

[0178] Specifically, in step S106, the speed parameters (linear speed 0.5 m / s, angular speed 30° / s) are extracted from the motion control instruction set, and the PID control algorithm is executed by the navigation module of the embedded control system to adjust the motor speed and steering angle to generate a preliminary navigation path including a coordinate sequence;

[0179] The depth sensor collects environmental point cloud data at a frequency of 30Hz, and uses the region growing algorithm to segment the wood and carpet material areas. The mean depth difference at the intersection is calculated (the depth of the wood area is 1.2cm, and the depth of the carpet area is 0.8cm), and the material boundary feature matrix is output.

[0180] Gaussian smoothing (σ = 0.5) is performed based on the material boundary feature matrix. When the smoothed gradient value exceeds the threshold of 0.3, a -0.4 cm compensation coefficient is applied to the depth data of the wood area, and the corrected depth point cloud is output;

[0181] The Kalman filter algorithm (process noise Q = 0.1, observation noise R = 0.05) is used to fuse the corrected depth point cloud and navigation path coordinates. If the diagonal elements of the positioning error covariance matrix are greater than 0.2m 2 , then the path nodes are updated through the A* algorithm to generate an optimized path including the new heading angle of 45°;

[0182] The motor control variables are calculated based on the optimized path. When the adjusted speed of 0.4 m / s matches the carpet area height of 0.7 cm detected in real time by the depth sensor, the edge avoidance strategy is triggered, and an avoidance path including an emergency braking command is output.

[0183] When executing the avoidance path, the edge features are monitored in real time through the depth sensor. If the depth mutation value detected in 5 consecutive frames is less than the dynamic threshold T_d = 1.5cm (where H_r is set to 15cm), it is determined to be a successful avoidance and the final navigation coordinates are output;

[0184] When a residual risk area is detected, the depth sensor acquisition process is restarted, bilateral filtering is used to remove noise, and new material boundary features are extracted. Kalman filtering and path optimization are then performed iteratively.

[0185] The re-collected depth image is subjected to median filtering (3×3 kernel) to extract the depth second-order derivative features of edge pixels. When the mutation value exceeds the dynamic threshold T_d, a secondary verification is performed using an SVM classifier (kernel function RBF, C=1.0) combined with HSV color space features (saturation threshold>0.6) and LBP texture features (contrast<40), and finally a set of high-risk area coordinates is output.

[0186] In this embodiment, C(p)=w d D(p)+w o O(p), where D(p) is the total distance of path p (in meters), O(p) is the cumulative weight of path p passing through the obstacle area (calculated based on the density of the obstacle distribution map), wd is the distance weight coefficient, and wo is the obstacle weight coefficient.

[0187] Those skilled in the art can make various other corresponding changes and deformations based on the technical solutions and concepts described above, and all of these changes and deformations should fall within the scope of protection of the claims of this application.

Claims

1. An intelligent cleaning method for a sweeping robot based on image recognition, characterized in that: include: The image sensor acquires RGB images and the depth sensor acquires depth data, extracts the texture contrast features of the carpet and the wooden floor, the color transition features at the junction, and the depth mutation features, and obtains the first multimodal feature set; The multi-head attention mechanism is used to fuse the first multimodal feature set, and the second multimodal feature set is obtained according to the light intensity distribution and material reflectivity distribution; From the second multimodal feature set, edge depth mutation features are filtered according to the dynamic depth threshold T_d = 10-20cm, and the color transition difference and texture continuity interruption features of carpet and wooden floor are integrated to obtain the fall risk label set; By processing the fall risk label set through time series analysis and integrating the illumination change characteristics, a stabilized fall risk distribution is obtained. Extract high-risk areas from the stabilized fall risk distribution, generate an obstacle distribution map, and obtain a motion control instruction set based on the path planning algorithm; Navigation adjustments are performed according to the motion control instruction set, and the depth deviation is corrected by integrating the material boundary features to obtain the avoidance path.

2. The intelligent cleaning method of a sweeping robot based on image recognition according to claim 1, characterized in that: The obtaining of the first multimodal feature set includes: The convolutional neural network is used to extract the roughness features of carpet texture, the regularity features of wooden floor texture and the color transition area features from the RGB image. The depth mutation features of the falling edge and the smoothness features of the material boundary are extracted by combining the depth sensor data to obtain the first multimodal feature set.

3. The intelligent cleaning method of a sweeping robot based on image recognition according to claim 1, characterized in that: The obtaining of the second multimodal feature set includes: The light intensity distribution and texture features are obtained by RGB image decomposition to generate an initial multimodal feature set. The multi-head attention mechanism is used to construct an attention allocation matrix and determine the weight distribution. If the light intensity is lower than the preset threshold, the dark ground area is extracted and the edge depth weight is enhanced to obtain enhanced features. The material reflectance distribution is analyzed based on the texture features to obtain the second multimodal feature set.

4. The intelligent cleaning method of a sweeping robot based on image recognition according to claim 1, characterized in that: The obtained fall risk label set includes: Principal component analysis is performed on the first multimodal feature set to reduce its dimensionality to obtain a second multimodal feature set. Edge depth mutation features are extracted from the second multimodal feature set, and their mutation values are calculated to obtain an edge mutation feature set. A dynamic depth threshold is calculated based on depth sensor data and a preset home fall height range. The relationship between the value of the edge mutation feature set and the dynamic depth threshold is determined to obtain a preliminary fused feature set. Feature fusion is performed on the preliminary fused feature set to obtain a fused feature set. A binary classifier is used to classify the fused feature set to obtain a fall risk label set.

5. The intelligent cleaning method of a sweeping robot based on image recognition according to claim 1, characterized in that: The method of processing the fall risk label set through time series analysis and integrating the illumination change characteristics to obtain a stabilized fall risk distribution includes: The inter-frame difference method is used to extract dynamic change features to obtain the time-series change data of the frame sequence. The time-series change data is processed to analyze the fall risk label set to obtain a preliminary fall risk distribution. It is judged whether the variance of the preliminary fall risk distribution exceeds the preset threshold to obtain a smoothed risk distribution. The light intensity distribution is detected to obtain the light change feature set. The light change feature set is processed through a convolutional neural network to obtain an enhanced risk distribution. If the local area risk value of the enhanced risk distribution is higher than the preset threshold, a stabilized fall risk distribution is obtained.

6. The intelligent cleaning method of a sweeping robot based on image recognition according to claim 1, characterized in that: The method of extracting high-risk areas from the stabilized fall risk distribution, generating an obstacle distribution map, and obtaining a motion control instruction set based on a path planning algorithm includes: High-risk areas are extracted from the stabilized fall risk distribution, and the Canny edge detection algorithm is applied to locate the edges of stairs and the slightly convex thresholds to generate an obstacle distribution map. Based on the obstacle distribution map, the A-star path planning algorithm is used in combination with the path cost function to calculate the optimal path. The path cost function is calculated based on the distance and the weight of the obstacle distribution map to obtain the motion control instruction set, which includes speed parameters and angle parameters.

7. The intelligent cleaning method of a sweeping robot based on image recognition according to claim 1, characterized in that: The navigation adjustment is performed according to the motion control instruction set, and the depth deviation is corrected by fusing the material boundary features to obtain the avoidance path, including: Speed and angle parameters are extracted from the motion control instruction set. Preliminary adjustments are performed through the navigation module to obtain a preliminary navigation path. Environmental data is then collected through a depth sensor to obtain material boundary features. Smooth features are extracted based on the material boundary features. If the smooth feature value exceeds a preset threshold, the depth deviation of the wood and carpet materials is corrected to obtain corrected depth data. If the positioning error between the depth data and the preliminary navigation path exceeds a preset threshold, the path parameters are updated to obtain an optimized path. The speed and angle parameters are adjusted through the optimized path. If the adjusted parameters match the corrected depth data, a fall edge avoidance instruction is generated.

Images