Method, device and system for monitoring suction, scrabbling and falling-off of cleaning robot

Abstract

The invention belongs to the field of computer vision, and particularly relates to a method, a device and a system for monitoring suction-scrabbling falling of a cleaning robot, and the method comprises the following steps: 1) a rear-view camera arranged on the cleaning robot collects suction-scrabbling images, and carries out data labeling; 2) building a deep learning labeling and training environment in a server or a workstation, selecting a YOLOv8 target detection network model, sending labeled suction and scrabbling image data into the model for training, and performing format conversion and quantization on the model after the training is completed; 3) inputting an image acquired in real time into the trained target detection model, and judging whether the suction rake falls off or not; (4) if the suction scrabbling device falls off, the upper computer sends falling abnormal information to a robot controller of the cleaning robot, and the robot controller controls a hub motor to brake emergently; and meanwhile, the upper computer transmits the abnormal information of the suction-scrabbling falling and the position coordinate information of the robot in the global map to the operation and maintenance personnel through the cloud platform and then transmits the abnormal result and the position information of the suction-scrabbling falling to the operation and maintenance personnel.

Description

Technical Field

[0001] The invention belongs to the field of computer vision, and in particular relates to a method, device and system for monitoring the falling off of a vacuum cleaner of a cleaning robot. Background Art

[0002] With the rise of artificial intelligence technology, intelligent robots are playing an increasingly important role in all walks of life. Cleaning robots can operate autonomously 24 hours a day and are welcomed by owners of factories, supermarkets, airports, warehouses, etc. However, during autonomous operation, since the length of the suction shovel exceeds the width of the robot body, there is a certain probability that the robot will hook onto obstacles. In serious cases, the suction shovel structure will be hit and fall off. At present, cleaning robots rarely monitor and handle abnormalities on the suction shovel, which undoubtedly increases a safety risk. If a robot without a suction shovel structure continues to operate, it will perform ineffective operations. In addition, current cleaning robots need to be equipped with additional hardware sensors for effective monitoring. Summary of the invention

[0003] The purpose of the present invention is to provide a method, device and system for monitoring the detachment of the suction scraper of a cleaning robot, which uses deep learning image technology to detect whether the suction scraper has detached. If detachment is detected, autonomous emergency braking is performed, and the abnormal information and the coordinate position information of the robot in the global map are sent to the operation and maintenance personnel for manual processing, so as to solve the safety hazards and invalid operation caused by the suction scraper of the cleaning robot falling off due to impact.

[0004] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a method for monitoring the detachment of a cleaning robot suction scraper, comprising the following steps:

[0005] 1) The rear-view camera set on the cleaning robot collects the suction image and performs data annotation;

[0006] 2) Build a deep learning annotation and training environment on the server or workstation, select the YOLOv8 target detection network model, send the annotated suction image data into the model for training, and convert and quantize the model after training;

[0007] 3) Input the real-time collected images into the trained target detection model to determine whether the suction pick has fallen off;

[0008] 4) If the vacuum cleaner falls off, the host computer will send the abnormal information of the falling off to the robot controller of the cleaning robot. The robot controller will control the hub motor to perform emergency braking. At the same time, the host computer will transmit the abnormal information of the vacuum cleaner falling off and the position coordinate information of the robot in the global map to the cloud platform, and the cloud platform will send the abnormal result and position information of the vacuum cleaner falling off to the operation and maintenance personnel.

[0009] The rear-view camera is a monocular camera or a depth camera, that is, an RGB image is collected by the monocular camera, or an infrared image or its built-in RGB image is collected by the depth camera.

[0010] The data annotation is specifically as follows:

[0011] 1-1) For each image, determine whether there are suction cup features; if the image contains more than 1 / 3 of the suction cup features, perform the annotation operation;

[0012] 1-2) Draw a rectangular box and frame it along the four-week contour of the suction cup to ensure that the framed range accurately includes the main feature part of the suction cup;

[0013] 1-3) In the label input area provided by the annotation tool, enter English as the label;

[0014] 1-4) After confirming that the annotation is correct, save the annotation result and generate a suction cup marking file that records the image coordinate position and the corresponding English label information.

[0015] The specific content of step 2) is as follows:

[0016] 2-1) Divide the annotated suction cup image dataset into a training set, a validation set, and a test set according to the ratio of 70%:20%:10%; among them, the training set is used to update the model parameters, the validation set is used to evaluate the performance of the model during training, and the test set is used to finally evaluate the generalization ability of the model;

[0017] 2-2) Perform data augmentation operations on the training set data, including random rotation, flipping, scaling, cropping, and brightness adjustment, to increase the diversity of the data and improve the generalization ability of the model; then load the divided dataset into the model for training through the data loader of the deep learning framework;

[0018] 2-3) Define the loss function: Obtain the total loss function according to the output of the YOLO model and the true label;

[0019] 2-4) Update the model parameters through the Adam optimizer;

[0020] 2-5) Perform multiple rounds of training, and each round of training includes the processes of forward propagation, loss calculation, backpropagation, and parameter update;

[0021] 2-6) After each round of training, use the validation set to evaluate the accuracy and recall rate of the model. When both the accuracy and recall rate are greater than 0.9, it is considered that the model meets the standard and the training is stopped.

[0022] The specific content of step 2-3) is as follows:

[0023] The total loss function is:

[0024] L = L cls + L reg

[0025] Among them, L cls is the classification loss function, and L reg is the regression loss function;

[0026] The classification loss function uses binary cross - entropy loss, that is:

[0027]

[0028] Among them, N is the number of samples, y i is the true label, is the probability value predicted by the model;

[0029] The regression loss function L reg adopts an improved intersection - over - union loss, and measures the overlap degree and relative position relationship between two boxes through the parameter relationship of the coordinates, width and height, and diagonal distance of the predicted box and the true box.

[0030] Using the validation set to evaluate the accuracy and recall rate of the model, specifically:

[0031] The calculation formula of the accuracy rate P is:

[0032]

[0033] The calculation formula of the recall rate R is:

[0034]

[0035] Among them, TP is the true positive example, that is, the number of positive samples correctly predicted by the model, FP is the false positive example, that is, the number of positive samples wrongly predicted by the model, and FN is the false negative example, that is, the number of negative samples wrongly predicted by the model.

[0036] The step 3), specifically:

[0037] 3 - 1) Size adjustment: Adjust the input image to be detected to the same size specification as when training the model, that is, perform size adjustment through bilinear interpolation. For any point (x1, y1) in the image, obtain the new coordinates (x ′ , y ′ ) and the corresponding pixel value I ′ (x ′ , y ′ );

[0038] 3 - 2) Normalize the pixel values of the image and map them to a specific interval [0, 1], then the normalization formula is:

[0039]

[0040] Among them, x is the original pixel value, (x_min) and (x_max) are the x min minimum value and x max maximum value of the pixel values in the image, and x norm is the pixel value after normalization;

[0041] 3-3) Input the preprocessed image into the trained object detection model for feature extraction. The convolutional layer uses the convolutional kernel to slide on the image for convolution operation;

[0042] For the input image I and the convolutional kernel K, the calculation formula for the convolution operation to generate the output feature map O is:

[0043]

[0044] Among them, (x, y) is the coordinate position in the output feature map;

[0045] 3-4) The model makes object predictions based on the extracted features, outputs possible detection boxes, and each detection box contains position information, category information, and confidence scores; and uses the non-maximum suppression algorithm to remove redundant detection boxes;

[0046] a. Sort all the detection boxes in descending order of confidence;

[0047] b. Select the detection box with the highest confidence as the current retained box;

[0048] c. Calculate the intersection over union IoU of the remaining detection boxes and the retained box to measure the overlap degree of the two detection boxes, that is:

[0049]

[0050] For the bounding box A = (x A1 , y A1 , x A2 , y A2 ) and B = (x B1 , y B1 , x B2 , y B2 ); According to the upper left corner coordinates (max(x A1 , x B1 ), max(y A1 , y B1 )) of the overlapping part and the lower right corner coordinates (min(x A2 , x B2 ), min(y A2 , y B2)) Obtain Area(A∩B) and Area(A∪B);

[0051] 3-5) Remove the detection boxes with an intersection-over-union ratio greater than the set threshold until all detection boxes have been processed, and finally obtain a set of filtered, non-overlapping detection boxes with relatively high confidence.

[0052] Step 4) includes the following steps:

[0053] 4-1) When the target detection algorithm fails to detect the suction cup feature or the confidence of the detected suction cup feature is lower than the set threshold, start the suction cup dropping processing flow;

[0054] 4-2) The host computer generates an abnormal information instruction including the suction cup dropping, and sends the instruction to the robot controller of the cleaning robot through a pre-set communication protocol. The communication protocol stipulates the data frame format, including a start bit, data bits, a check bit, and a stop bit. The host computer assembles the data frame according to the protocol and then sends the data frame to the robot controller through the communication interface;

[0055] 4-3) The host computer obtains the position coordinate information of the robot in the global map through the robot's positioning system based on the SLAM algorithm, packs the abnormal information of the suction cup detachment and the position coordinate information, and sends the data to the cloud platform according to the communication protocol agreed with the cloud platform;

[0056] 4-4) After receiving the suction cup dropping abnormal information instruction sent by the host computer, the robot controller first parses the data frame and extracts the abnormal information identifier according to the communication protocol;

[0057] 4-5) The controller immediately generates a control signal according to the pre-programmed logic to control the hub motor to brake emergently;

[0058] 4-6) After receiving the abnormal result and position information of the suction cup detachment sent by the host computer, the cloud platform parses and stores the data, records it in the database for subsequent query and analysis, and forwards the abnormal result and position information of the suction cup detachment to the operation and maintenance personnel according to the communication method with the operation and maintenance personnel's terminal.

[0059] A monitoring device for a method of monitoring the detachment of the suction cup of a cleaning robot, comprising: a robot body, a rear-view camera, a suction cup, a rear hub, a front hub, and a front-view camera;

[0060] The rear-view camera is installed on the robot body and is used to collect image data of the rear of the robot including the suction cup; the rear-view camera is: a monocular camera or a depth camera, and has the function of adjusting the shooting angle and focal length to meet the comprehensive shooting requirements of the suction cup in different working scenarios;

[0061] The rear wheel hub and the front wheel hub are respectively installed at the front and rear parts of the robot body, and are used to support the robot body and realize the movement of the robot. The rear wheel hub is connected to a hub motor, and the hub motor performs emergency braking operation according to the control signal received from the robot controller.

[0062] The forward-looking camera is installed in front of the robot body to collect image data of the environment in front of the robot, assist the robot body in navigation and positioning during movement, and provide data support for the construction of the global map.

[0063] A monitoring system for a cleaning robot suction scraper shedding monitoring method, comprising: an image acquisition and processing module, a model building module, a judgment and decision module, an abnormality handling module, a robot controller, a host computer and a cloud platform;

[0064] The image acquisition and processing module is used to acquire image data from the rear-view camera in real time and pre-process the acquired images, including image resizing and pixel value normalization operations, to meet the input requirements of the subsequent target detection algorithm; and input the pre-processed image data into the target detection model pre-trained by the model building module;

[0065] The model building module is used to adopt the target detection network structure of YOLOv8. During the training process, the labeled image data containing the normal state and the detached state of the suction pick are used. By continuously adjusting the model parameters, the accuracy and recall rate of the model when detecting the suction pick meet the set standards;

[0066] The judgment and decision module is used to receive the suction and pick detection results output by the target detection model, and compare the confidence level in the detection results with a preset threshold value; when the suction and pick confidence level is detected to be lower than the preset threshold value, it is determined that the suction and pick have fallen off, and the corresponding abnormal processing flow is triggered;

[0067] The exception handling module is used for generating an abnormal information instruction of the suction wand falling off by the upper computer when the judgment and decision module determines that the suction wand has fallen off, and sending the instruction to the robot controller through a specific communication protocol;

[0068] The robot controller is used to parse the instruction content after receiving the abnormal information instruction, generate a control signal according to the pre-programmed logic, and send it to the hub motor connected to the rear wheel hub and the front wheel hub to control the hub motor to emergency brake and stop the robot from moving;

[0069] The host computer is used to simultaneously obtain the position coordinate information of the robot in the global map, package the abnormal information of the suction pick falling off with the position coordinate information, and send the data to the cloud platform according to the communication protocol agreed with the cloud platform;

[0070] The cloud platform is used to receive and store the above data. After parsing the data, the abnormal results and location information of the suction scraper detachment are sent to the operation and maintenance personnel through the SMS notification interface and the instant messaging software interface.

[0071] The present invention has the following beneficial effects and advantages:

[0072] 1. The present invention can autonomously monitor whether the suction pick of the cleaning robot falls off, autonomously perform emergency braking and report abnormalities.

[0073] 2. The present invention does not require the installation of additional hardware sensors. The robot's own recognition and obstacle avoidance sensors can be used to effectively monitor the suction scraper falling, effectively reducing the monitoring cost.

[0074] 3. The deep learning image recognition technology used in the present invention has much higher accuracy and detection rate than traditional image processing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 A side view of the installation position of the robot rear view camera and the suction pick of the present invention;

[0076] Figure 2 A top view of the installation position of the robot rear-view camera and the suction pick of the present invention;

[0077] Figure 3 This is a flow chart of the method for suction detection and processing using a monocular camera in the present invention;

[0078] Figure 4 This is a flow chart of the method for suction detection and processing using a depth camera in the present invention;

[0079] Figure 5 Flowchart of the method for training a suction detection model based on deep learning;

[0080] Among them, 1 is the robot body, 2 is the rear-view camera, 3 is the suction pick, 4 is the rear wheel hub, 5 is the front wheel hub, and 6 is the front-view camera. DETAILED DESCRIPTION

[0081] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0082] like Figures 1 to 2 As shown, it is a schematic diagram of the installation position of the robot rear-view camera and the suction scraper of the present invention, and a monitoring device of a cleaning robot suction scraper shedding monitoring method of the present invention comprises: a robot body 1, a rear-view camera 2, a suction scraper 3, a rear wheel hub 4, a front wheel hub 5, and a front-view camera 6;

[0083] A suction scraper 3 or a mop is installed at the bottom of the robot body 1 for cleaning and sewage absorption;

[0084] The rear-view camera 6 is installed obliquely downward on the robot body 1 at an installation height of 60 - 90 cm and an oblique downward angle range of 30 - 60°, and is used to collect image data of the area behind the robot including the suction cup.

[0085] The rear-view camera 2 is: a monocular camera or a depth camera, and has the function of adjusting the shooting angle and focal length to meet the comprehensive shooting requirements of the suction cup in different working scenarios, that is: collecting RGB images through the monocular camera, or collecting infrared images or the built-in RGB images using the depth camera.

[0086] The rear wheels 4 and the front wheels 5 are respectively installed at the front and rear parts of the robot body 1, used to support the robot body 1 and realize the movement of the robot, and the rear wheels 4 are connected with hub motors, and the hub motors perform emergency braking operations according to the control signals received from the robot controller.

[0087] The front-view camera 6 is installed in front of the robot body, used to collect image data of the environment in front of the robot, assist the navigation and positioning of the robot body during movement, and provide data support for the construction of the global map.

[0088] As Figures 3 to 4 shown, this is an embodiment of the suction cup detection and processing method using two different cameras for the rear-view camera of the present invention. The difference is that the monocular camera determines that the suction cup 3 has fallen off when the suction cup 3 is not detected, while the depth camera determines that the suction cup 3 has fallen off when the suction cup 3 is detected. This embodiment is discussed with the depth camera processing as the target.

[0089] Embodiment 1:

[0090] As Figure 3 shown, the rear-view camera 2 is installed with a relatively inexpensive monocular camera, which can be an image acquisition sensor shared by multiple target recognition tasks, such as recognizing "stains", "sewage", "garbage", "suction cups", etc. behind the vehicle. The greatest advantage is cost savings, and multiple target tasks can be detected. The installation angle of view of this monocular camera needs to cover the position of the suction cup, that is, when the suction cup is working properly, the suction cup appears within the image field of view. When the suction cup has not fallen off, the position of the suction cup rectangular frame detected in the image always remains within a certain range; if the suction cup has fallen off, the position of the suction cup rectangular frame detected in the image will change greatly or be outside the image detection range, and it can be determined that the suction cup has fallen off.

[0091] Embodiment 2:

[0092] As Figure 4As shown in the figure, the rear-view camera 2 is a depth camera. The main function of the depth camera is to avoid obstacles using point cloud data. It will output an infrared image by itself, or come with an RGB sensor and will also output an RGB image. Whether it is an infrared image or an RGB image, it can be used as the image source for suction cup detection. The advantage of doing this is that the original obstacle avoidance sensor is used for the suction cup detection task, and there is no need to install a dedicated sensor for suction cup detection, saving costs. The viewing angle of the installation position of the depth camera should not cover the suction cup position, that is, the suction cup is not within the image field of view during normal operation, and appears within the image field of view when the suction cup falls off (if the suction cup is within the field of view of the depth camera when it has not fallen off, the point cloud will regard the suction cup as an obstacle, thus affecting the driving and operation of the robot). When the suction cup falls off, the position of the suction cup rectangular frame detected by the image will appear within the image detection range, and it is immediately determined that the suction cup has fallen off.

[0093] As Figures 4 to 5 shown, the following is the flowchart of the method for suction cup detection and processing using a depth camera in the present invention. A method for monitoring the falling off of the suction cup of a cleaning robot in the present invention includes the following steps:

[0094] 1) Set the rear-view camera of the cleaning robot to collect suction cup images and perform data annotation;

[0095] 1-1) For each image, determine whether there are suction cup features in it; if the image contains more than 1 / 3 of the suction cup features, perform the annotation operation;

[0096] 1-2) Draw a rectangular frame and select along the surrounding contour of the suction cup to ensure that the selected range accurately includes the main feature part of the suction cup;

[0097] 1-3) In the label input area provided by the annotation tool, enter English as the label;

[0098] 1-4) After confirming that the annotation is correct, save the annotation result and generate a suction cup mark file recording the image coordinate position and the corresponding English label information.

[0099] In this embodiment, a. Manual annotation, such as using the annotation tool labelimg; b. First train a version of the suction cup detection model with some of the annotated data, and then use this model to detect the unannotated suction cup data, or detect the automatically detected suction cup data and convert it into annotated data; c. Use a large model automatic annotation tool, such as Grounded-SAM;

[0100] In this embodiment, only the images containing the suction feature in the images are marked. The state when the suction is normally installed or detached is acceptable, as long as there is a suction in the image. As long as the image contains more than 1 / 3 of the suction feature, it can be marked. In the annotation tool, select the four-sided contour of the suction, and then label this contour with the English word "suction", and a marked file with the image coordinate position of "suction" and the label of "suction English" will be formed.

[0101] 2) Set up a deep learning annotation and training environment on the server or workstation. Select the YOLOv8 object detection network model, and send the annotated suction image data into the model for training. After the training is completed, perform format conversion and quantization on the model;

[0102] In step 2), to detect the model cliff, this embodiment uses the yolo model;

[0103] Adjust the parameters for training. As long as the accuracy and recall rate reach relatively good indicators, with the accuracy and recall rate greater than 0.9 in one version, it is considered a usable model;

[0104] The model needs format conversion and model quantization. The purpose is to enable the model to run on the robot processor. Model quantization can make the detection speed of each frame of the model faster;

[0105] Among them, the method of step 2) is specifically as follows:

[0106] 2-1) Divide the annotated suction image dataset into a training set, a validation set, and a test set according to the ratio of 70%:20%:10%; among them, the training set is used for updating the parameters of the model, the validation set is used for evaluating the performance of the model during the training process, and the test set is used for finally evaluating the generalization ability of the model;

[0107] 2-2) Perform data augmentation operations on the training set data, including random rotation, flipping, scaling, cropping, and brightness adjustment, to increase the diversity of the data and improve the generalization ability of the model; then load the divided dataset into the model for training through the data loader of the deep learning framework;

[0108] 2-3) Define the loss function: According to the output of the YOLO model and the true label, obtain the total loss function;

[0109] The total loss function is:

[0110] L = L cls +L reg

[0111] Among them, L cls is the classification loss function, and L reg is the regression loss function;

[0112] The classification loss function uses binary cross - entropy loss, that is:

[0113]

[0114] where N is the number of samples, y i is the true label, is the probability value predicted by the model;

[0115] The regression loss function L reg adopts an improved intersection - over - union loss. By the parametric relationships of the coordinates, widths, heights, and diagonal distances between the predicted bounding box and the ground - truth bounding box, it measures the overlap degree and relative position relationship between the two boxes.

[0116] 2 - 4) Update the model parameters through the Adam optimizer;

[0117] 2 - 5) Conduct multiple rounds of training. Each round of training includes the processes of forward propagation, loss calculation, backpropagation, and parameter update;

[0118] 2 - 6) After each round of training, use the validation set to evaluate the accuracy and recall rate of the model. When both the accuracy and recall rate are greater than 0.9, it is considered that the model meets the standard and stop training.

[0119] Use the validation set to evaluate the accuracy and recall rate of the model. Specifically:

[0120] The formula for calculating the accuracy P is:

[0121]

[0122] The formula for calculating the recall rate R is:

[0123]

[0124] where TP is the true positive, that is, the number of positive samples correctly predicted by the model, FP is the false positive, that is, the number of positive samples wrongly predicted by the model, and FN is the false negative, that is, the number of negative samples wrongly predicted by the model.

[0125] 3) Input the real - time collected image into the trained object - detection model to determine whether the suction cup has fallen off;

[0126] 3 - 1) Size adjustment: Adjust the input image to be detected to the same size specification as when training the model, that is, perform size adjustment through bilinear interpolation. For any point (x1, y1) in the image, obtain the new coordinates (x ′ , y ′ ) and the corresponding pixel value I ′ (x ′ , y ′ );

[0127] 3-2) Normalize the pixel values of the image and map them to a specific interval [0,1]. The normalization formula is as follows:

[0128]

[0129] where x is the original pixel value, (x_{min}) and (x_{max}) are the x min minimum value and x max maximum value of the pixel values in the image, and x norm is the normalized pixel value;

[0130] 3-3) Input the preprocessed image into the trained object detection model for feature extraction. The convolutional layer uses a convolutional kernel to slide over the image for convolution operations;

[0131] For the input image I and the convolutional kernel K, the calculation formula for the convolution operation to generate the output feature map O is:

[0132]

[0133] where (x,y) is the coordinate position in the output feature map;

[0134] 3-4) The model makes object predictions based on the extracted features, outputs possible detection boxes, and each detection box contains location information, class information, and a confidence score; and removes redundant detection boxes through the non-maximum suppression algorithm;

[0135] a. Sort all detection boxes in descending order of confidence;

[0136] b. Select the detection box with the highest confidence as the current retained box;

[0137] c. Calculate the intersection over union (IoU) of the remaining detection boxes and the retained box to measure the overlap degree of the two detection boxes, that is:

[0138]

[0139] For bounding box A=(x A1 ,y A1 ,x A2 ,y A2 ) and B=(x B1 ,y B1 ,x B2 ,y B2 ); according to the upper left corner coordinates (max(x A1 ,x B1 ),max(y A1 ,y B1 )) of the overlapping part and the lower right corner coordinates (min(xA2 ,x B2 ),min(y A2 ,y B2 ))Get Area(A∩B) and Area(A∪B);

[0140] 3-5) Remove the detection frames whose IoU ratio is greater than the set threshold until all detection frames are processed, and finally obtain a set of screened, non-overlapping and high-confidence detection frames.

[0141] 4) If the vacuum cleaner falls off, the host computer will send the abnormal information of the falling off to the robot controller of the cleaning robot. The robot controller will control the hub motor to perform emergency braking. At the same time, the host computer will transmit the abnormal information of the vacuum cleaner falling off and the position coordinate information of the robot in the global map to the cloud platform, and the cloud platform will send the abnormal result and position information of the vacuum cleaner falling off to the operation and maintenance personnel.

[0142] Step 4) comprises the following steps:

[0143] 4-1) When the target detection algorithm fails to detect the suction feature or the confidence of the detected suction feature is lower than the set threshold, the suction drop processing flow is started;

[0144] 4-2) The host computer generates an abnormal information instruction containing the vacuum cleaner falling off, and sends the instruction to the robot controller of the cleaning robot through a pre-set communication protocol. The communication protocol specifies the data frame format, including the start bit, data bit, check bit and stop bit. The host computer assembles the data frame according to the protocol, and then sends the data frame to the robot controller through the communication interface;

[0145] 4-3) The host computer obtains the position coordinate information of the robot in the global map through the robot's positioning system based on the SLAM algorithm, packages the abnormal information of the suction pick falling off with the position coordinate information, and sends the data to the cloud platform according to the communication protocol agreed with the cloud platform;

[0146] 4-4) After receiving the suction and scraping abnormal information instruction sent by the host computer, the robot controller first parses the data frame and extracts the abnormal information identifier according to the communication protocol;

[0147] 4-5) The controller immediately generates a control signal according to the pre-programmed logic to control the wheel hub motor to perform emergency braking;

[0148] 4-6) After the cloud platform receives the abnormal results and location information of the suction pick detachment sent by the host computer, it parses and stores the data and records it in the database for subsequent query and analysis, and forwards the abnormal results and location information of the suction pick detachment to the operation and maintenance personnel in accordance with the communication method with the operation and maintenance personnel terminal.

[0149] Based on a cleaning robot suction scraper falling-off monitoring method provided by the present invention, a monitoring system of the cleaning robot suction scraper falling-off monitoring method is provided on the robot body 1, and a corresponding method flow is implemented based on the system, and the system includes: an image acquisition and processing module, a model building module, a judgment and decision module, an abnormality handling module, a robot controller, a host computer and a cloud platform;

[0150] The image acquisition and processing module is used to acquire image data from the rear-view camera in real time and pre-process the acquired images, including image resizing and pixel value normalization operations, to meet the input requirements of the subsequent target detection algorithm; and input the pre-processed image data into the target detection model pre-trained by the model building module;

[0151] The model building module is used to adopt the target detection network structure of YOLOv8. During the training process, the labeled image data containing the normal state and the detached state of the suction pick are used. By continuously adjusting the model parameters, the accuracy and recall rate of the model when detecting the suction pick meet the set standards;

[0152] The judgment and decision module is used to receive the suction and pick detection results output by the target detection model, and compare the confidence level in the detection results with a preset threshold value; when the suction and pick confidence level is detected to be lower than the preset threshold value, it is determined that the suction and pick have fallen off, and the corresponding abnormal processing flow is triggered;

[0153] The exception handling module is used for generating an abnormal information instruction of the suction wand falling off by the upper computer when the judgment and decision module determines that the suction wand falls off, and sending the instruction to the robot controller through a specific communication protocol;

[0154] The robot controller is used to parse the instruction content after receiving the abnormal information instruction, generate a control signal according to the pre-programmed logic, and send it to the hub motor connected to the rear wheel hub and the front wheel hub to control the hub motor to emergency brake and stop the robot from moving;

[0155] The host computer is used to simultaneously obtain the position coordinate information of the robot in the global map, package the abnormal information of the suction pick falling off with the position coordinate information, and send the data to the cloud platform according to the communication protocol agreed with the cloud platform;

[0156] The cloud platform is used to receive and store the above data. After parsing the data, the abnormal results and location information of the suction scraper detachment are sent to the operation and maintenance personnel through the SMS notification interface and the instant messaging software interface.

[0157] In summary, combined with the embodiments of the present invention, it can be seen that the present invention does not require the installation of additional hardware sensors, and the robot's own recognition and obstacle avoidance sensors can be used to complete the effective monitoring of the suction scraper falling, effectively reducing the monitoring cost; the deep learning image recognition technology has much higher accuracy and detection rate than traditional image processing technology.

[0158] Those skilled in the art will appreciate that the above are only preferred embodiments of the present invention, and the various embodiments of the present disclosure and / or the features described in the claims may be combined or combined in various ways, even if such combinations or combinations are not explicitly described in the present disclosure. It is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art may still modify the technical solutions described in the aforementioned embodiments, or perform equivalent substitutions on some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

[0159] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for monitoring the detachment of a cleaning robot suction scraper, characterized in that: The following steps are involved: 1) The rear-view camera set on the cleaning robot collects the suction image and performs data annotation; 2) Build a deep learning annotation and training environment on the server or workstation, select the YOLOv8 target detection network model, send the annotated suction image data into the model for training, and convert and quantize the model after training; 3) Input the real-time collected images into the trained target detection model to determine whether the suction pick has fallen off; 4) If the vacuum cleaner falls off, the host computer will send the abnormal information of the falling off to the robot controller of the cleaning robot. The robot controller will control the hub motor to perform emergency braking. At the same time, the host computer will transmit the abnormal information of the vacuum cleaner falling off and the position coordinate information of the robot in the global map to the cloud platform, and the cloud platform will send the abnormal result and position information of the vacuum cleaner falling off to the operation and maintenance personnel.

2. A method for monitoring the detachment of a cleaning robot suction scraper according to claim 1, characterized in that: The rear-view camera is a monocular camera or a depth camera, that is, the monocular camera is used to collect RGB images, or the depth camera is used to collect infrared images or its own RGB images.

3. A method for monitoring the detachment of a cleaning robot suction scraper according to claim 1, characterized in that: The data annotation is specifically as follows: 1-1) For each image, determine whether there is a suction feature; if the image contains more than 1 / 3 of the suction features, perform a labeling operation; 1-2) Draw a rectangular frame and select along the four sides of the suction bar to ensure that the selection range accurately includes the main features of the suction bar; 1-3) In the label input area provided by the annotation tool, enter English as the label; 1-4) After confirming that the annotation is correct, save the annotation results and generate a suction and scraping marking file that records the image coordinate position and the corresponding English label information.

4. A method for monitoring the detachment of a cleaning robot suction scraper according to claim 1, characterized in that: The step 2) is specifically: 2-1) The labeled suction image dataset is divided into a training set, a validation set, and a test set in a ratio of 70%:20%:10%; the training set is used to update the parameters of the model, the validation set is used to evaluate the performance of the model during the training process, and the test set is used to finally evaluate the generalization ability of the model; 2-2) Perform data enhancement operations including random rotation, flipping, scaling, cropping, and brightness adjustment on the training set data to increase data diversity and improve the generalization ability of the model; then use the data loader of the deep learning framework to load the divided data set into the model for training; 2-3) Define the loss function: Get the total loss function based on the output of the YOLO model and the true label; 2-4) Update model parameters through Adam optimizer; 2-5) Perform multiple rounds of training, each round of training includes: forward propagation, loss calculation, back propagation and parameter update process; 2-6) After each round of training, the validation set is used to evaluate the accuracy and recall of the model. When both the accuracy and recall are greater than 0.9, the model is considered to have met the standard and training is stopped.

5. A method for monitoring the detachment of a cleaning robot suction scraper according to claim 4, characterized in that: The steps 2-3) are specifically: The total loss function is: L=L cls +L reg Among them, L cls is the classification loss function, L reg is the regression loss function; The classification loss function uses binary cross entropy loss, namely: Where N is the number of samples, y i is the true label, is the probability value predicted by the model; Regression loss function L reg The improved intersection-over-union loss is used to measure the overlap and relative position relationship between the two boxes through the parameter relationship between the coordinates, width, height, and diagonal distance of the predicted box and the real box.

6. A method for monitoring the detachment of a cleaning robot suction scraper according to claim 4, characterized in that: The accuracy and recall of the model evaluated using the validation set are as follows: The calculation formula of accuracy P is: The calculation formula of recall rate R is: Among them, TP is the true positive example, that is, the number of samples correctly predicted by the model as positive, FP is the false positive example, that is, the number of samples incorrectly predicted by the model as positive, and FN is the false negative example, that is, the number of samples incorrectly predicted by the model as negative.

7. A method for monitoring the detachment of a cleaning robot suction scraper according to claim 1, characterized in that: The step 3) is specifically: 3-1) Size adjustment: The input image to be detected is adjusted to the same size as that used when the model was trained. That is, the size is adjusted by bilinear interpolation. For any point (x1, y1) in the image, the scaled new coordinates (x ′ ,y ′ ) and the corresponding pixel value I ′ (x ′ ,y ′ ); 3-2) Normalize the pixel values ​​of the image and map them to a specific interval [0,1]. The normalization formula is: Where x is the original pixel value, (x_{min}) and (x_{max}) are the x and y values ​​of the pixels in the image, respectively. min Minimum value and x max Maximum value, x norm is the normalized pixel value; 3-3) The preprocessed image is input into the trained target detection model for feature extraction, and the convolution layer uses the convolution kernel to slide on the image to perform convolution operation; For the input image I and the convolution kernel K, the calculation formula for the convolution operation to generate the output feature map O is: Among them, (x, y) is the coordinate position in the output feature map; 3-4) The model predicts the target based on the extracted features and outputs possible detection boxes. Each detection box contains location information, category information, and confidence score. The non-maximum suppression algorithm is used to remove redundant detection boxes. a. Sort all detection boxes from high to low according to their confidence; b. Select the detection box with the highest confidence as the current retained box; c. Calculate the intersection over union (IoU) of the remaining detection boxes and the retained box to measure the degree of overlap between the two detection boxes, namely: For the bounding box A = (x A1 ,y A1 ,x A2 ,y A2 ) and B=(x B1 ,y B1 ,x B2 ,y B2 ); According to the coordinates of the upper left corner of the overlapping part (max(x A1 ,x B1 ),max(y A1 ,y B1 )), the lower right corner coordinate (min(x A2 ,x B2 ),min(y A2 ,y B2 ))Get Area(A∩B) and Area(A∪B); 3-5) Remove the detection frames whose IoU ratio is greater than the set threshold until all detection frames are processed, and finally obtain a set of filtered, non-overlapping and high-confidence detection frames.

8. A method for monitoring the detachment of a cleaning robot suction scraper according to claim 1, characterized in that: The step 4) comprises the following steps: 4-1) When the target detection algorithm fails to detect the suction feature or the confidence of the detected suction feature is lower than the set threshold, the suction drop processing flow is started; 4-2) The host computer generates an abnormal information instruction containing the vacuum cleaner falling off, and sends the instruction to the robot controller of the cleaning robot through a pre-set communication protocol. The communication protocol specifies the data frame format, including the start bit, data bit, check bit and stop bit. The host computer assembles the data frame according to the protocol, and then sends the data frame to the robot controller through the communication interface; 4-3) The host computer obtains the position coordinate information of the robot in the global map through the robot's positioning system based on the SLAM algorithm, packages the abnormal information of the suction pick falling off with the position coordinate information, and sends the data to the cloud platform according to the communication protocol agreed with the cloud platform; 4-4) After receiving the suction and scraping abnormal information instruction sent by the host computer, the robot controller first parses the data frame and extracts the abnormal information identifier according to the communication protocol; 4-5) The controller immediately generates a control signal according to the pre-programmed logic to control the wheel hub motor to perform emergency braking; 4-6) After the cloud platform receives the abnormal results and location information of the suction pick detachment sent by the host computer, it parses and stores the data and records it in the database for subsequent query and analysis, and forwards the abnormal results and location information of the suction pick detachment to the operation and maintenance personnel in accordance with the communication method with the operation and maintenance personnel terminal.

9. The monitoring device of the cleaning robot suction scraper falling-off monitoring method according to claim 1, characterized in that: include: Robot body (1), rear-view camera (2), suction pick (3), rear wheel hub (4), front wheel hub (5), front-view camera (6); The rear-view camera (2) is mounted on the robot body (1) and is used to collect image data of the robot rear including the suction pick (3); the rear-view camera (2) is a monocular camera or a depth camera and has the function of adjusting the shooting angle and focal length to meet the comprehensive shooting requirements of the suction pick (3) in different working scenes; The rear wheel hub (4) and the front wheel hub (5) are respectively mounted on the front and rear parts of the robot body (1) and are used to support the robot body (1) and realize the movement of the robot. The rear wheel hub (4) is connected to a wheel hub motor, and the wheel hub motor performs an emergency braking operation according to a control signal received from a robot controller. A forward-looking camera (6) is installed in front of the robot body (1) and is used to collect image data of the environment in front of the robot, assist the robot body (1) in navigation and positioning during movement, and provide data support for the construction of a global map.

10. The monitoring system of the method for monitoring the detachment of the suction scraper of a cleaning robot according to claim 1, characterized in that: include: Image acquisition and processing module, model building module, judgment and decision-making module, exception handling module, robot controller, host computer and cloud platform; An image acquisition and processing module is used to acquire image data from the rear-view camera (2) in real time, pre-process the acquired image, including: image size adjustment and pixel value normalization operations to meet the input requirements of the subsequent target detection algorithm; and input the pre-processed image data into the target detection model pre-trained by the model building module; A model building module is used to adopt the target detection network structure of YOLOv8, use the labeled image data containing the normal state and the detached state of the suction pick (3) during the training process, and continuously adjust the model parameters so that the accuracy and recall rate of the model when detecting the suction pick (3) reach the set standards; A judgment and decision module is used to receive the suction pick (3) detection result output by the target detection model, and compare the confidence level in the detection result with a preset threshold value; when the confidence level of the suction pick (3) is detected to be lower than the preset threshold value, it is determined that the suction pick (3) has fallen off, and a corresponding abnormal processing flow is triggered; The abnormality handling module is used for generating an abnormality information instruction of the suction pick falling off by the upper computer when the judgment and decision module determines that the suction pick (3) falls off, and sending the instruction to the robot controller through a specific communication protocol; The robot controller is used to parse the instruction content after receiving the abnormal information instruction, generate a control signal according to the pre-programmed logic, and send it to the hub motors connected to the rear wheel hub (4) and the front wheel hub (5), so as to control the hub motors to brake urgently and stop the robot from moving; The host computer is used to simultaneously obtain the position coordinate information of the robot in the global map, package the abnormal information of the suction pick falling off with the position coordinate information, and send the data to the cloud platform according to the communication protocol agreed with the cloud platform; The cloud platform is used to receive and store the above data, and after parsing the data, sends the abnormal result and location information of the suction pick (3) falling off to the operation and maintenance personnel through the SMS notification interface and the instant messaging software interface.

Images