A cleaning method, device, cleaning robot and readable storage medium

Abstract

The application discloses a cleaning method and device, a cleaning robot and a readable storage medium. The application discloses obtaining a single-frame image, identifying the single-frame image based on a preset deep learning algorithm, and obtaining an initial behavior recognition result. Based on a multi-frame image fusion judgment strategy, the initial behavior recognition result corresponding to each single-frame image is fused and judged to obtain a first behavior recognition result. If the first behavior recognition result is an abnormal recognition, the single-frame images are accumulated, and the accumulated multi-frame images are subjected to time sequence behavior recognition according to a time sequence behavior recognition algorithm to determine a second behavior recognition result. Based on the first behavior recognition result and the second behavior recognition result, a target behavior recognition result is determined. The target behavior recognition result is used to determine a target cleaning strategy for cleaning processing. The application improves the cleaning efficiency.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image recognition, and particularly relates to a cleaning method and device, a cleaning robot and a readable storage medium. BACKGROUND

[0002] With the continuous increase of the rate of keeping pets at home, the demand for cleaning robots by pet-keeping families is increasingly prominent.

[0003] In the traditional technology, the cleaning robot has developed a variety of pet-related functions, including pet video monitoring through a front-end camera, cat and dog recognition for obstacle avoidance, pet highlight snapshot, pet activity hot zone recognition, and pet tool recognition.

[0004] However, in the traditional technology, only video monitoring is used as a basic monitoring tool for pet monitoring, which not only needs human control or relies on synchronous monitoring during work, but also only regards pets as ordinary obstacles, has low accuracy of pet behavior recognition, and cannot intelligently analyze the behavior of pets to adjust the cleaning logic. SUMMARY

[0005] Therefore, the technical problem to be solved by the present application is that the accuracy of pet behavior recognition is low, and the behavior of pets cannot be intelligently analyzed to adjust the cleaning logic.

[0006] To solve the above technical problems, in a first aspect, the present application provides a cleaning method, which is applied to a cleaning robot, and the method comprises:

[0007] obtaining a single-frame image and identifying the single-frame image based on a preset deep learning algorithm to obtain an initial behavior recognition result;

[0008] based on a multi-frame image fusion judgment strategy, fusing and judging the initial behavior recognition result corresponding to each single-frame image to obtain a first behavior recognition result;

[0009] if the first behavior recognition result is abnormal, accumulating each single-frame image and performing time sequence behavior recognition on the accumulated multi-frame image according to a time sequence behavior recognition algorithm to determine a second behavior recognition result;

[0010] based on the first behavior recognition result and the second behavior recognition result, determining a target behavior recognition result; the target behavior recognition result is used to determine a target cleaning strategy for cleaning processing.

[0011] In one of the embodiments, before the single-frame image is obtained, the method further comprises:

[0012] obtaining an image of a single-frame pet activity scene through an image acquisition unit;

[0013] inputting the single-frame image into a pre-trained deep learning model, and performing feature extraction and pet behavior category prediction on the single-frame image based on the deep learning model to output probability values of various pet behaviors;

[0014] obtaining an initial behavior recognition result according to the probability values of the various pet behaviors.

[0015] In one of the embodiments, the initial behavior recognition result corresponding to each single-frame image is fused and judged based on a multi-frame image fusion judgment strategy to obtain a first behavior recognition result, including:

[0016] If the initial behavior recognition results corresponding to the continuous multiple single-frame images are all the same, the initial behavior recognition result is determined as the first behavior recognition result;

[0017] If the initial behavior recognition results corresponding to the continuous multiple single-frame images are not all the same, the first behavior recognition result is determined as a default value.

[0018] In one of the embodiments, the first behavior recognition result being an identification exception includes that the first behavior recognition result is the default value, and / or the first behavior recognition result is that the pet has an abnormal behavior.

[0019] In one of the embodiments, the multiple single-frame images are accumulated, and a second behavior recognition result is determined by performing time sequence behavior recognition on the accumulated multiple-frame images according to a time sequence behavior recognition algorithm, including:

[0020] obtaining a multiple-frame image sequence based on the single-frame images within a target time length according to a preset time sequence;

[0021] performing spatial feature extraction on the multiple-frame image sequence by a time sequence behavior recognition model to obtain a feature vector;

[0022] fusing the feature vectors based on time sequence correlation to obtain a fused feature vector;

[0023] performing pet behavior recognition on the fused feature vector to determine the second behavior recognition result.

[0024] In one of the embodiments, the single-frame images within the target time length according to the preset time sequence are accumulated to obtain the multiple-frame image sequence, including:

[0025] obtaining a video segment based on the single-frame images within the target time length according to the preset time sequence;

[0026] The video segment is input into the fast-slow network, double-path frame sampling is performed through the fast-slow network, and a first key frame image sequence corresponding to a slow path and a second key frame image sequence corresponding to a fast path are obtained.

[0027] In one of the embodiments, the spatial feature extraction on the multi-frame image sequence through the time sequence behavior identification model obtains a feature vector, including:

[0028] Double-path feature extraction is performed on the first key frame image sequence and the second key frame image sequence through the fast-slow network, and a static feature vector corresponding to the slow path and a dynamic feature vector corresponding to the fast path are obtained.

[0029] In one of the embodiments, the fusion of each feature vector based on the time sequence correlation obtains a fused feature vector, including:

[0030] The number of channels of the dynamic feature vector corresponding to the fast path is adjusted, and the adjusted dynamic feature vector is aligned with the static feature vector corresponding to the slow path in terms of channels;

[0031] Dimension reduction sampling is performed on the fast path through a time average pooling layer or a convolution layer in the fast-slow network, so that the dynamic feature vector of the fast path after the dimension reduction sampling is time-aligned with the static feature vector corresponding to the slow path;

[0032] The dynamic feature vector and the static feature vector that are aligned in terms of channels and time are fused to obtain a fused feature vector.

[0033] In one of the embodiments, the pet behavior identification on the fused feature vector determines a second behavior identification result, including:

[0034] The pet behavior identification on the fused feature vector through a fully connected layer in the fast-slow network outputs probability values of various pet behaviors;

[0035] The second behavior identification result is determined according to the probability values of the various pet behaviors.

[0036] In one of the embodiments, the method further includes:

[0037] Based on the target behavior identification result, a target cleaning strategy is determined, and the target cleaning strategy includes one or more of cleaning, avoidance, and warning.

[0038] In one of the embodiments, the determination of the target cleaning strategy based on the target behavior identification result includes:

[0039] When the target behavior recognition result is a diet behavior, a virtual diet area is constructed based on a pet position or a position of a diet item, and the virtual diet area is labeled as a first cleaning area;

[0040] A cleaning area other than the first cleaning area is cleaned, and a cleaning progress and an activity state of a pet in the first cleaning area are continuously monitored;

[0041] When the other cleaning area is cleaned and / or the pet has left the first cleaning area, cleaning of the first cleaning area is performed.

[0042] In one of the embodiments, the cleaning of the first cleaning area includes:

[0043] A cleaning area and / or a dirt degree of the first cleaning area are identified, and the first cleaning area is cleaned based on the cleaning area and / or the dirt degree.

[0044] In one of the embodiments, the identification of the cleaning area and / or the dirt degree of the first cleaning area and the cleaning of the first cleaning area based on the cleaning area and / or the dirt degree include:

[0045] When the cleaning area of the first cleaning area is greater than an area threshold or the dirt degree is greater than a dirt threshold, the cleaning robot is instructed to return to a base station to clean a cleaning part of the cleaning robot;

[0046] The first cleaning area is cleaned based on the cleaning part after the cleaning.

[0047] In one of the embodiments, the cleaning of the cleaning part of the cleaning robot includes drying processing of the cleaning part of the cleaning robot and washing processing of the cleaning part of the cleaning robot.

[0048] In one of the embodiments, the instruction of the cleaning robot to return to the base station to clean the cleaning part of the cleaning robot includes:

[0049] Before the cleaning of the first cleaning area, the cleaning robot is instructed to return to the base station, and the base station is used for drying processing of the cleaning part of the cleaning robot.

[0050] In one of the embodiments, the cleaning part includes an edge brush, a roller brush, and a wiping cloth, and the cleaning of the first cleaning area based on the cleaning part after the cleaning includes:

[0051] controlling the edge brush to be in the non-cleaning position and controlling the roller brush in the cleaning position to clean the first cleaning area at a first cleaning speed, and the wiping cloth in the cleaning position to clean the first cleaning area at a second cleaning speed.

[0052] In one of the embodiments, when the first cleaning area has solid particles, the first cleaning speed of the roller brush is determined based on a particle density of the solid particles; or the first cleaning speed of the roller brush is inversely proportional to the second cleaning speed of the edge brush.

[0053] In one of the embodiments, after the first cleaning area is cleaned based on the cleaning treatment of the cleaning component, the method further comprises:

[0054] Based on the dirtiness of the first cleaning area after the first cleaning, it is determined whether to clean the cleaning component and to clean the first cleaning area repeatedly.

[0055] In one of the embodiments, the repeated cleaning of the first cleaning area comprises:

[0056] controlling the edge brush to be in the non-cleaning position and controlling the roller brush in the cleaning position to clean the first cleaning area at a first cleaning speed, and the wiping cloth in the cleaning position to clean the first cleaning area at a second cleaning speed.

[0057] In one of the embodiments, the identifying the cleaning area and / or the dirtiness of the first cleaning area, based on the cleaning area and / or the dirtiness, to clean the first cleaning area comprises:

[0058] When the cleaning area of the first cleaning area is less than an area threshold and / or the dirtiness is less than a dirtiness threshold, the first cleaning area is cleaned based on the cleaning component of the cleaning robot;

[0059] Based on the dirtiness of the first cleaning area after the first cleaning, it is determined whether to clean the cleaning component and to clean the first cleaning area repeatedly.

[0060] In one of the embodiments, the cleaning component comprises an edge brush, a roller brush and a wiping cloth, and the cleaning treatment of the first cleaning area based on the cleaning component of the cleaning robot comprises:

[0061] controlling the edge brush to be in the non-cleaning position and controlling the roller brush in the cleaning position to clean the first cleaning area at a first cleaning speed, and the wiping cloth in the cleaning position to clean the first cleaning area at a second cleaning speed.

[0062] In one of the embodiments, the determining the target cleaning strategy based on the target behavior recognition result comprises:

[0063] when the target behavior recognition result is a sleep behavior, a licking behavior, or a staying behavior, determining a second cleaning area based on a pet position;

[0064] continuously monitoring pet cleaning progress and / or activity state in the second cleaning area until cleaning of other cleaning areas except the second cleaning area is completed and / or the pet has left the second cleaning area, performing cleaning of the second cleaning area.

[0065] In one embodiment, the determining of the target cleaning strategy based on the target behavior recognition result comprises:

[0066] when the target behavior recognition result is an abnormal behavior, sending a warning notification message to a user based on a number of occurrences and / or a duration of the abnormal behavior; and / or,

[0067] when the target behavior recognition result is an abnormal behavior, constructing a virtual obstacle avoidance area based on a pet position, and performing an obstacle avoidance cleaning mode to avoid the virtual obstacle avoidance area.

[0068] In one embodiment, the sending of the warning notification message to the user based on the number of occurrences and the duration of the abnormal behavior comprises:

[0069] when the number of occurrences and / or the duration of the same abnormal behavior is greater than a preset threshold, sending a warning notification message to a user.

[0070] In one embodiment, the abnormal behavior corresponds to a reporting priority, and the method further comprises:

[0071] when the target behavior recognition result comprises a plurality of abnormal behaviors, sending a warning notification message to a user based on the abnormal behavior with the highest reporting priority.

[0072] In a second aspect, the present application further provides a cleaning device applied to a cleaning robot, comprising: a first identification module, a second identification module, a third identification module, and a first determination module, wherein:

[0073] the first identification module is configured to acquire a single frame of image, and identify the single frame of image based on a preset deep learning algorithm to obtain an initial behavior recognition result;

[0074] the second identification module is configured to perform fusion judgment on the initial behavior recognition result corresponding to each single frame of image based on a multi-frame image fusion judgment strategy to obtain a first behavior recognition result;

[0075] a third identification module, configured to, if the first behavior identification result is an identification exception, accumulate each single-frame image, and perform time-series behavior identification on the accumulated multiple-frame images according to a time-series behavior identification algorithm to determine a second behavior identification result;

[0076] a first determination module, configured to determine a target behavior identification result based on the first behavior identification result and the second behavior identification result; the target behavior identification result is used to determine a target cleaning strategy to perform cleaning processing.

[0077] In a third aspect, the present application further provides a cleaning robot, comprising a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method in any one of the first aspect when executing the computer program.

[0078] In a fourth aspect, the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the following steps when executed by a processor:

[0079] acquiring single-frame images and identifying the single-frame images based on a preset deep learning algorithm to obtain initial behavior identification results;

[0080] performing fusion judgment on the initial behavior identification results corresponding to each single-frame image based on a multi-frame image fusion judgment strategy to obtain a first behavior identification result;

[0081] if the first behavior identification result is an identification exception, accumulating each single-frame image and performing time-series behavior identification on the accumulated multiple-frame images according to a time-series behavior identification algorithm to determine a second behavior identification result;

[0082] determining a target behavior identification result based on the first behavior identification result and the second behavior identification result; the target behavior identification result is used to determine a target cleaning strategy to perform cleaning processing.

[0083] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, and the computer program implements the following steps when executed by a processor:

[0084] acquiring single-frame images and identifying the single-frame images based on a preset deep learning algorithm to obtain initial behavior identification results;

[0085] performing fusion judgment on the initial behavior identification results corresponding to each single-frame image based on a multi-frame image fusion judgment strategy to obtain a first behavior identification result;

[0086] if the first behavior identification result is an identification exception, accumulating each single-frame image and performing time-series behavior identification on the accumulated multiple-frame images according to a time-series behavior identification algorithm to determine a second behavior identification result;

[0087] Based on the first behavior recognition result and the second behavior recognition result, a target behavior recognition result is determined; the target behavior recognition result is used to determine a target cleaning strategy for cleaning processing.

[0088] The technical solution provided by the application has the following advantages:

[0089] The initial behavior recognition result is obtained through single-frame image recognition, and the first behavior recognition result is obtained by combining multi-frame image fusion discrimination strategy, so that single-frame misjudgment is effectively filtered, and the stability and reliability of conventional behavior recognition are improved. If there is no abnormal behavior in the initial behavior recognition, the first behavior recognition result is directly used as the final target behavior recognition result, which can save computing resources and improve computing efficiency. If there is a possible abnormal behavior in the initial behavior recognition result, the second behavior recognition result is determined by accumulating multiple frames of images and using a time sequence behavior recognition algorithm, which greatly improves the recognition accuracy of complex abnormal behaviors. The target behavior recognition result obtained by finally fusing the two types of results can accurately reflect the real behavior state of the pet and provide a reliable basis for the cleaning robot to develop a targeted cleaning strategy. BRIEF DESCRIPTION OF DRAWINGS

[0090] In order to more clearly illustrate the specific embodiments of the application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0091] Figure 1 It is a flowchart of the cleaning method in an embodiment;

[0092] Figure 2 It is a flowchart of the step of obtaining the initial behavior recognition result in an embodiment;

[0093] Figure 3 It is a schematic diagram of the internal structure of the cleaning robot in an embodiment;

[0094] Figure 4 It is a flowchart of the step of determining the first behavior recognition result in an embodiment;

[0095] Figure 5 It is a flowchart of the step of determining the second behavior recognition result in an embodiment;

[0096] Figure 6 It is a flowchart of the step of performing double-path sampling in the fast and slow networks in an embodiment;

[0097] Figure 7 This is a flowchart illustrating the dual-path feature extraction steps in a fast and slow network in one embodiment.

[0098] Figure 8 This is a flowchart illustrating the steps for obtaining the fused feature vector in one embodiment;

[0099] Figure 9 This is a schematic diagram of the fast and slow network processing flow in one embodiment;

[0100] Figure 10 This is a flowchart illustrating the steps for determining the second behavior recognition result based on a fast and slow network in one embodiment.

[0101] Figure 11 This is a flowchart illustrating the process of determining a target cleaning strategy in one embodiment;

[0102] Figure 12 This is a flowchart illustrating the steps of marking a first cleaning area and cleaning the first cleaning area in one embodiment.

[0103] Figure 13 This is a flowchart illustrating the steps of cleaning a first cleaning area by identifying the cleaning area and / or the degree of dirt in one embodiment.

[0104] Figure 14 This is a flowchart illustrating the cleaning steps for the first cleaning area when the cleaning area or degree of dirt exceeds a threshold, as shown in one embodiment.

[0105] Figure 15 This is a flowchart illustrating the cleaning steps performed by each cleaning component on the first cleaning area in one embodiment.

[0106] Figure 16 This is a flowchart illustrating the process of determining repeated cleaning steps for a first cleaning area in one embodiment.

[0107] Figure 17 This is a flowchart illustrating the repeated cleaning process of each cleaning component on the first cleaning area in one embodiment.

[0108] Figure 18 This is a flowchart illustrating the cleaning steps for the first cleaning area when the cleaning area or degree of dirt is less than a threshold, as shown in one embodiment.

[0109] Figure 19 This is a flowchart illustrating the process of cleaning a first cleaning area whose cleaning area or degree of dirt is less than a threshold, as shown in one embodiment.

[0110] Figure 20 This is a flowchart illustrating the cleaning steps for the second cleaning area in one embodiment;

[0111] Figure 21Fig. 1 is a flowchart of a target cleaning processing strategy step of executing pre-warning and avoidance in an embodiment;

[0112] Figure 22 Fig. 2 is a flowchart of a pre-warning notification message sending step to a user in an embodiment;

[0113] Figure 23 Fig. 3 is a flowchart of a pre-warning notification message sending step to a user based on an abnormal behavior priority in an embodiment;

[0114] Figure 24 Fig. 4 is a structural diagram of a cleaning device in an embodiment;

[0115] Figure 25 Fig. 5 is a structural diagram of a cleaning robot in an embodiment;

[0116] Figure 26 Fig. 6 is a structural diagram of a cleaning element of a cleaning robot in an embodiment. DETAILED DESCRIPTION

[0117] The technical solutions of the present application will be described clearly and completely below in conjunction with the drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The present application will be described in detail below with reference to the drawings and in conjunction with the embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0118] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0119] In the present application, the orientation words such as "up", "down", "top", "bottom" used without the opposite description are generally directed to the direction shown in the drawings, or to the vertical, perpendicular or gravity direction of the components themselves. Similarly, for the convenience of understanding and description, "inner" and "outer" refer to the inner and outer relative to the contour of the components themselves, but the above-mentioned orientation words are not used to limit the present application.

[0120] In the daily life of a pet-keeping family, cleaning robots have developed a variety of pet-related functions, including pet video monitoring through a front-end camera, cat and dog recognition for obstacle avoidance, pet highlight snapshot, pet activity hot zone recognition, and pet tool recognition, etc. However, in the conventional technology, pet monitoring is only carried out by video monitoring as a basic monitoring tool, which not only needs to be controlled by humans or relies on synchronous monitoring, but also only regards pets as ordinary obstacles, has low accuracy of pet behavior recognition, and cannot intelligently analyze the behavior of pets to adjust the cleaning logic, and thus it is difficult to dynamically optimize the cleaning strategy and respond to abnormal pet conditions in a timely manner.

[0121] Based on this, in one exemplary embodiment, as shown in Figure 1 A cleaning method is provided, which is applied to a cleaning robot, and the method comprises:

[0122] In step 101, a single-frame image is obtained, and the single-frame image is identified based on a preset deep learning algorithm to obtain an initial behavior recognition result.

[0123] The deep learning algorithm can be implemented based on a deep learning model.

[0124] In implementation, for pet-keeping families, the cleaning strategy of the cleaning robot needs to be determined based on the behavior of the pet, therefore, the cleaning robot will collect information of the cleaning scene based on image acquisition units, sensors and other devices during the cleaning process, realize the identification of the pet behavior, set a targeted target cleaning strategy based on the pet behavior, and ensure accurate supervision of the pet and deep cleaning of the cleaning area. The cleaning robot also contains a processing unit, so that the cleaning robot can acquire each single-frame image in the cleaning environment where the cleaning robot is located through the image acquisition unit. Then, the image acquisition unit transmits the acquired single-frame image to the processing unit of the cleaning robot, and the processing unit performs image recognition and screening on each single-frame image through the deep learning model, and screens to obtain an image frame containing a pet. Further, the cleaning robot identifies the behavior of the pet based on the single-frame image of the pet and the preset deep learning algorithm, and obtains the initial behavior recognition result of the pet corresponding to the single-frame image.

[0125] Specifically, the image acquisition unit can be, but is not limited to, a monocular camera, a multi-view camera, a depth camera, and a panoramic camera, and the type and number of the image acquisition unit are not limited in the embodiments of the present disclosure. Taking the image acquisition unit as a monocular camera as an example, the cleaning robot acquires a single-frame image of the pet in real time through the monocular camera carried by the cleaning robot. To ensure the image quality of the single-frame image and improve the accuracy of the pet behavior recognition, the processing unit can pre-process each image. For example, the image is cropped to an effective field of view of 800x600, and then the image contrast is enhanced through Gaussian filtering denoising and adaptive histogram equalization, to obtain a pre-processed image. Then, the processing unit inputs the pre-processed single-frame image into a pre-trained deep learning model, identifies and processes the single-frame image through the deep learning model, and finally outputs the initial behavior recognition result for the single-frame image.

[0126] The deep learning model adopts a two-stage architecture of "target detection" and "behavior classification", is pre-trained by labeled training samples, and the trained deep learning model can recognize and judge one or more pet behaviors. Optionally, the deep learning model can be a YOLO model. Through the YOLOv8 algorithm of the YOLO model, the pet target in a single frame image can be located, and the pet in the image can be calibrated, that is, the bounding box coordinates (x1, y1, x2, y2) of the calibrated pet are output. Further, the YOLO model can identify the pet species through the YOLOv8 algorithm, and recognize the pet behavior in the region framed by the bounding box coordinates. For example, the model training process labels 9 kinds of pet behaviors for recognition training. The fully connected layer of the trained YOLO model can output the probability distribution of 9 kinds of pet behaviors, that is, it contains the probability values corresponding to normal (pet staying behavior), drinking water, eating, sleeping, licking hair, scratching, hen squatting, vomiting, and excretion respectively. A behavior recognition probability threshold is set for a single frame image, and then the initial behavior recognition result is determined through the behavior recognition probability threshold. For example, the behavior category with the highest probability and greater than or equal to 0.7 is taken as the initial behavior recognition result.

[0127] Optionally, if the highest probability value in the probability value corresponding to each behavior is less than the behavior recognition probability threshold, that is, the highest probability value < 0.7, the processing unit of the cleaning robot will mark the single frame image as "unrecognized".

[0128] Step 102, based on a multi-frame image fusion judgment strategy, the initial behavior recognition results corresponding to each single frame image are fused and judged to obtain a first behavior recognition result;

[0129] In implementation, in order to improve the accuracy of pet behavior recognition result, a multi-frame image fusion judgment strategy is pre-set in the cleaning robot. When recognizing pet behavior, the cleaning robot performs fusion judgment on the initial behavior recognition results corresponding to the accumulated continuous multiple single frame images based on the multi-frame image fusion judgment strategy to obtain a first behavior recognition result. Specifically, a sliding window mechanism can be used in the processing unit of the cleaning robot to perform fusion judgment on the initial recognition results corresponding to three continuous image frames. The sliding window mechanism takes the current frame as the center, selects the previous 1 frame and the next 1 frame to form a window containing 3 frames of images, and performs fusion judgment with a time span of 0.1 seconds. The essence of the fusion judgment is consistency judgment of the initial behavior recognition results. If the initial behavior recognition results corresponding to the three frames are completely consistent and are all valid behaviors, that is, there is no unrecognized behavior, the initial behavior recognition result of the behavior is output as the first behavior recognition result. If the initial behavior recognition results corresponding to at least one frame or multiple frames are inconsistent, or there is no recognized valid behavior, a "default value" is output as the first behavior recognition result.

[0130] Optionally, for various behaviors of the pet, such as "excretion", "sleeping", and the like, a time decay mechanism is set for a sustained behavior with a duration of more than 5 seconds, that is, if the previous 3 frames output a certain behavior, and the subsequent 2 frames are not detected, then the behavior label is still retained for 2 seconds, so as to avoid behavior misjudgment caused by short interruption.

[0131] In an optional embodiment, the initial behavior recognition result recognized by the single frame image represents a regular behavior of the pet, for example, pet staying, drinking water, eating, sleeping, licking, excretion, and the like, and the cleaning robot performs fusion judgment on the plurality of initial behavior recognition results based on a continuous multi-frame image fusion judgment strategy, and outputs a consistent first behavior recognition result, which can be used as the target behavior recognition result of the pet behavior recognition, without triggering the time sequence behavior recognition process, saving the processing flow of the pet behavior recognition, and improving the pet behavior recognition efficiency.

[0132] In step 103, if the first behavior recognition result is an abnormal recognition, the single frame images are accumulated, and the time sequence behavior recognition is performed on the accumulated multi-frame images according to the time sequence behavior recognition algorithm, to determine a second behavior recognition result.

[0133] In implementation, after the initial behavior recognition results corresponding to the continuous multiple single-frame images are fused and judged to obtain the first behavior recognition result, if the pet behavior representation output by the first behavior recognition result is a simple behavior such as eating, drinking, licking, etc., in order to improve the calculation efficiency and save the calculation resources, the first behavior recognition result is taken as the final target behavior recognition result of the pet. If the first behavior recognition result is an identification exception, it is indicated that the pet behavior recognition may not be accurate only by relying on static single-frame images or multiple single-frame images, and therefore, a time sequence behavior recognition algorithm needs to be introduced to comprehensively judge the pet behavior. The identification exception of the first behavior recognition result includes two cases: 1. the first behavior recognition result is a default value; 2. the first behavior recognition result is an abnormal behavior of the pet. For the two cases, the detection of the time sequence behavior recognition needs to be involved. For case 1, the first behavior recognition result is a default value, which proves that the pet behaviors corresponding to the fused and judged continuous three frames of images are inconsistent, and it is possible that the same behavior cannot be recognized for three frames in succession, thereby leading to an inaccurate pet behavior. For case 2, it is determined that the initial recognition results of the continuous three frames of images of the first behavior recognition result appear abnormal behaviors such as scratching and vomiting. The abnormal behavior of the pet often involves complex actions of the pet, and the actions also have continuity, which indicates that the complex pet behavior is difficult to recognize, and it is possible that the pet behavior of this type cannot be recognized only by relying on single-frame images. Therefore, the time sequence behavior recognition judgment process is started, the cleaning robot accumulates each single-frame image, and the multiple frames of images accumulated are subjected to time sequence behavior recognition according to the time sequence behavior recognition algorithm to determine the second behavior recognition result. Specifically, the processing unit starts to accumulate 64 frames of images from the image frame in which the abnormal behavior is first detected, the time span of the 64 frames is about 2.1 seconds, and the 64 frames are uniformly scaled to 256*340 resolution and subjected to data enhancement; then, the optimized time sequence behavior recognition algorithm is used to perform time sequence behavior recognition on the accumulated multiple frames of images to determine the second behavior recognition result. The time sequence behavior recognition algorithm can be implemented by a time sequence behavior recognition model, which can include but is not limited to a slow-fast network (SlowFast network), a time segment network (TSN, Temporal segment networks: Towards good practices for deep action recognition), etc., and the embodiments of the present disclosure do not limit this. The specific processing process of the time sequence behavior recognition model will be described in detail in the embodiments described below, and will not be described here.

[0134] Step 104, determining a target behavior recognition result based on the first behavior recognition result and the second behavior recognition result.

[0135] In implementation, for the case that the pet abnormal behavior is identified in the initial behavior recognition result, the first behavior recognition result obtained through the multi-frame image fusion judgment is comprehensively judged with the second behavior recognition result determined by the time sequence behavior recognition algorithm to determine the final target behavior recognition result. Specifically, when the first behavior recognition result labels the single-frame image with an abnormal behavior tag such as “scratch”, “vomit”, “hen squat”, or when the fusion judgment process of multiple initial behavior recognition results detects abnormal behavior but does not pass 3-frame fusion, the time sequence recognition process is triggered, so as to determine the second behavior recognition result. In this way, when the cleaning robot comprehensively determines the final target behavior recognition result according to the first and second behavior recognition results, it follows the priority rule for processing: if the second behavior recognition result is “scratch”, “vomit” or “hen squat” and the probability is ≥0.75, the abnormal behavior is given priority, and the second behavior recognition result is directly taken as the target result; if the second behavior recognition result is “unidentified” or normal behavior, the first behavior recognition result that meets the 3-frame fusion is taken as the reference; if the first behavior recognition result is a regular behavior and the second behavior recognition result is an abnormal behavior, the probability of the abnormal behavior is compared with the probability of the regular behavior, and when the probability of the abnormal behavior is ≥ the probability of the regular behavior, the second behavior recognition result is taken as the reference, otherwise the first behavior recognition result is taken. The final output result can include but is not limited to behavior category, 0-1 range confidence, behavior occurrence timestamp, and cleaning robot positioning coordinates based on SLAM (Simultaneous Localization and Mapping) map, providing data support for subsequent motion planning.

[0136] In the embodiment, the initial behavior recognition result is obtained through single-frame image recognition, the first behavior recognition result is obtained by combining the multi-frame image fusion judgment strategy, the single-frame misjudgment is effectively filtered, and the stability and reliability of regular behavior recognition are improved; at the same time, for the abnormal behavior that may exist in the initial behavior recognition result, the second behavior recognition result is determined by accumulating multiple frames of images and using the time sequence behavior recognition algorithm, which greatly improves the recognition accuracy of complex abnormal behaviors; the target behavior recognition result obtained by finally fusing the two types of results can accurately reflect the real behavior state of the pet, and can provide reliable basis for the cleaning robot to formulate targeted cleaning strategies, so that the cleaning process is more intelligent and more in line with actual needs.

[0137] In an example embodiment, as shown in Figure 2 Before step 101, the method further includes:

[0138] Step 201: acquiring an image of a single-frame pet activity scene through an image acquisition unit.

[0139] In implementation, as shown in Figure 3 Figure 3 ​As a schematic view of the internal structure of the cleaning robot, the image acquisition unit is included in the cleaning robot, through which real-time image acquisition of the cleaning environment can be performed. For pet-keeping families, pet activity scenes exist in the cleaning environment, so that the image of the pet activity scene can be further acquired during the cleaning process of the cleaning robot. Specifically, the image acquisition unit of the cleaning robot can be selected from a monocular camera, a depth camera, a low-illumination camera, and the like. The device type of the image acquisition unit in the cleaning robot is not limited in the embodiments of the present disclosure, and is selected and set according to the light conditions, field of view requirements, and the like of the actual application scene. During the acquisition process, as the cleaning robot moves, the image acquisition unit continuously captures the picture of the pet activity scene at a preset frame rate, and a frame of image is intercepted as a single-frame pet activity scene image every fixed time interval. At the same time, in order to ensure the accuracy of subsequent identification, the acquired image needs to contain clear pet subject and surrounding environment information, so as to avoid loss of pet features due to excessive occlusion, blur or too dark light. After the acquisition is completed, the image is temporarily stored in the local cache of the cleaning robot, waiting for subsequent processing.

[0140] In step 202, the single-frame image is input into the pre-trained deep learning model, and feature extraction and pet behavior category prediction are performed on the single-frame image based on the deep learning model, and probability values of various pet behaviors are output.

[0141] In implementation, the cleaning robot has a trained deep learning model integrated in advance. The cleaning robot can input the acquired single-frame image into the pre-trained deep learning model, and perform feature extraction and pet behavior category prediction on the single-frame image based on the deep learning model, and finally output probability values of various pet behaviors.

[0142] Specifically, the cleaning robot first pre-processes the input single-frame image, including adjusting the size to the required specification of the model, for example, 224x224 pixels, performing normalization processing, and mapping the pixel value to the range of 0-1, and the like, to adapt to the input requirements of the model. Subsequently, the pre-processed image is input into the trained deep learning model, and the pre-processed image is processed. First, the target detection network is used to locate the pet position and crop the pet region, and then the behavior classification network is used to extract features of the pet region, capture key features such as the limb posture, action contour, and hair texture of the pet, and finally perform pet behavior category prediction based on these features, and output probability values of preset pet behaviors, such as normal (stay), drinking water, eating, scratching, vomiting, and the like, and the sum of the probability values of all behavior categories is 1.

[0143] The pre-trained deep learning model can adopt a two-stage architecture of "YOLOv8 target detection + EfficientNet-B4 behavior classification" to process the preprocessed single-frame image, or can adopt a Faster RCNN network (Faster Region-based Convolutional Neural Networks) to identify the pet behavior, and the disclosure does not limit this.

[0144] In step 203, an initial behavior recognition result is obtained according to the probability values of various pet behaviors.

[0145] In implementation, for the probability values of various pet behaviors output by the deep learning model, the cleaning robot determines the pet behavior corresponding to the highest probability value among the probability values of various pet behaviors as the initial behavior recognition result.

[0146] Specifically, after obtaining the probability values of various pet behaviors, the processing unit of the cleaning robot analyzes and judges the probability values, and selects the behavior category with the highest probability value as the candidate result. If the highest probability value is greater than or equal to a preset behavior recognition probability threshold, the behavior category is determined as the initial behavior recognition result; if the highest probability value is less than the preset threshold, it is considered that the current image cannot accurately recognize the pet behavior, and the initial behavior recognition result is marked as "unrecognized". For example, if the probability of "drinking water" output by the deep learning model is 0.85, and the value is the highest among all behavior categories and greater than the preset behavior recognition probability threshold 0.7, the initial behavior recognition result is "drinking water"; if the highest probability value is 0.65, which is less than the preset behavior recognition probability threshold 0.7, the initial behavior recognition result is "unrecognized".

[0147] In this embodiment, the single-frame pet activity scene image is accurately acquired by the image acquisition unit, and then the pre-trained deep learning model is used for feature extraction and behavior category prediction of the image. With the powerful learning ability of the model, the key features such as the body posture and action details of the pet can be comprehensively captured, and the probability values of various behaviors can be accurately output. Finally, the initial behavior recognition result is determined according to the probability values, which improves the real-time performance and accuracy of pet behavior recognition.

[0148] In an exemplary embodiment, as shown in Figure 4 the specific processing process of step 102 includes:

[0149] In step 401, if the initial behavior recognition results corresponding to the plurality of continuous single-frame images are all the same, the initial behavior recognition result is determined as the first behavior recognition result.

[0150] In implementation, to improve the accuracy of pet behavior recognition, after initial behavior recognition is performed on a single frame of image, the cleaning robot can further obtain initial behavior recognition results corresponding to continuous multiple single frames of image based on a multi-frame image fusion judgment strategy, perform fusion judgment, i.e. consistency judgment, on the multiple initial behavior recognition results, and if the continuous multiple initial behavior recognition results are all the same, the cleaning robot determines the corresponding pet behavior in the initial behavior recognition result as the first behavior recognition result.

[0151] Specifically, if the initial behavior recognition results corresponding to continuous 3 single frames of image with a time span of about 0.1 second are all the same and are all valid behaviors, the cleaning robot directly determines the initial behavior recognition result as the first behavior recognition result. For example, when the initial recognition results of the continuous 3 frames are all pet "sleeping" behaviors, and the behavior probability values of each frame all satisfy a preset behavior recognition probability threshold (such as ≥ 0.7), it is indicated that the behavior has high stability and credibility, and therefore "sleeping" is taken as the current first behavior recognition result.

[0152] In step 402, if the initial behavior recognition results corresponding to the continuous multiple single frames of image are not all the same, the first behavior recognition result is determined as a default value.

[0153] In implementation, if the initial behavior recognition results corresponding to the continuous multiple single frames of image are not all the same, there is inconsistency, the cleaning robot determines the first behavior recognition result as a default value or as a null value. The fusion judgment of the multi-frame image does not output an effective result. For example, the initial behavior recognition results corresponding to continuous 3 frames of image with a time span of 0.1 second are fused and judged, for example, the initial behavior recognition results corresponding to 2 frames of image are "drinking water" and the initial behavior recognition result corresponding to 1 frame of image is "running" behavior, the initial behavior recognition results of the 3 frames are not all the same, and therefore the first behavior recognition result is determined as a default value.

[0154] In this embodiment, the first behavior recognition result is determined by performing consistency judgment on the initial behavior recognition results of the continuous multiple single frames of image, the consistent initial behavior recognition result is determined as the first behavior recognition result when the results are all the same, the behavior continuity of the continuous frames can be effectively utilized to ensure the stability and credibility of the recognition result, and when the initial behavior recognition results are not all the same, the first behavior recognition result is set as a default value, which can avoid the error result caused by single frame misjudgment, instantaneous motion interference, etc. from being adopted, and reduce the recognition error. This processing manner not only fully utilizes the advantages of multi-frame image fusion, filters unstable recognition information, but also lays a reliable foundation for determining the final target behavior by combining the time sequence recognition result subsequently, and improves the accuracy and rigor of the overall pet behavior recognition.

[0155] In an exemplary embodiment, if there is an abnormal behavior of the pet in the initial behavior recognition result, in order to avoid the complexity of the abnormal behavior, only relying on a single frame image cannot accurately identify, therefore, when the abnormal behavior occurs, as shown in Figure 5 The specific processing process of step 103 includes:

[0156] Step 501, based on the preset time sequence, accumulate each single frame image in the target time length, and obtain a plurality of image sequences.

[0157] In implementation, the cleaning robot accumulates a plurality of single frame images in the target time length based on the preset time sequence, so that the cleaning robot can obtain a video segment of the target time length based on the plurality of single frame images. Further, for the video segment of the target time length, image preprocessing is performed, each frame image in the video segment is spatially adjusted and scaled to a fixed resolution, for example, 224*224 resolution, and a data enhancement method is applied to each frame image in the video segment through random cropping, horizontal flipping and other means to improve the generalization ability of the model. In this way, for the preprocessed video segment, data sampling is performed on the video segment in combination with the sampling requirements of the time sequence behavior recognition model, so that the time sequence behavior recognition model can process a plurality of image sequences.

[0158] Specifically, the target time length of the accumulated plurality of frame images can be pre-set based on the regular duration of the abnormal behavior, which is not limited in the embodiments of the present disclosure. For behaviors such as "scratching" and "vomiting", the duration is set to about 2.1 seconds, which corresponds to 64 frame images collected at a frame rate of 30fps. During the accumulation process, the single frame images are strictly arranged in the time sequence of image acquisition to ensure that the plurality of image sequences can fully reflect the time development process of the pet behavior. At the same time, a timestamp is added to each frame image to provide a time dimension reference for subsequent time sequence correlation analysis. After the accumulation is completed, the plurality of image sequences are stored in a video segment format for easy input into the time sequence behavior recognition model for processing.

[0159] Step 502, spatial feature extraction of the plurality of image sequences is performed by the time sequence behavior recognition model to obtain a feature vector.

[0160] In implementation, the cleaning robot performs spatial feature extraction on the plurality of image sequences by the time sequence behavior recognition model to obtain a feature vector. The time sequence behavior recognition model can be, but is not limited to, an optimized SlowFast network, a TSN time period network, etc. In this way, the feature extraction network in the time sequence behavior recognition model in the cleaning robot will perform deep feature mining on the pet area contained in each frame image, and extract high-dimensional spatial features containing pet limb posture, action contour, hair texture, etc. After encoding, the features form corresponding feature vectors, and each feature vector can accurately represent the spatial feature attributes of the pet in the single frame image.

[0161] At step 503, the feature vectors are fused based on the time sequence correlation to obtain a fused feature vector.

[0162] In implementation, the time sequence behavior recognition model in the cleaning robot fuses the obtained feature vectors based on the time sequence correlation between multiple frames of images to obtain a fused feature vector. In the process of feature fusion, the time sequence behavior recognition model analyzes the time correlation of the feature vectors in the multiple frames of image sequences by using a time sequence modeling mechanism to capture the dynamic change rule of the pet behavior in the time dimension. For example, for the “vomiting” behavior, the time sequence behavior recognition model focuses on the correlation between the “lowering the head and arching the back” feature vector in the previous frames and the “body twitching” feature vector in the subsequent frames, and integrates these feature vectors with time sequence correlation into a fused feature vector by one or more fusion methods such as weighting, splicing, or direct addition. The fused feature vector not only contains the spatial feature information of each single frame of image, but also contains the dynamic change feature of the pet behavior over time, thereby more comprehensively representing the complete behavior process of the pet.

[0163] At step 504, the pet behavior is recognized based on the fused feature vector to determine a second behavior recognition result.

[0164] In implementation, the time sequence behavior recognition model in the cleaning robot recognizes the pet behavior based on the fused feature vector to determine a second behavior recognition result. Specifically, the cleaning robot inputs the fused feature vector into the classification layer (e.g., a fully connected layer) of the time sequence behavior recognition model, and the classification layer predicts the pet behavior class based on the spatial features and time sequence features contained in the fused feature vector, and outputs the probability values of each class of pet behavior. Then, the time sequence behavior recognition model filters out the behavior class with the highest probability and meeting the behavior recognition probability threshold requirement according to a preset behavior recognition probability threshold (e.g., the current behavior recognition probability threshold is 0.75), and determines it as the second behavior recognition result.

[0165] In this embodiment, the single frames of images within the target time duration are accumulated in a preset time sequence to form a sequence of multiple frames of images, and the feature vectors are extracted by the time sequence behavior recognition model, and then the fused feature vectors obtained by fusing the feature vectors based on the time sequence correlation contain not only spatial information but also dynamic changes in the time dimension, and comprehensively represent the complete behavior process of the pet; finally, the fused feature vectors are recognized to determine the second behavior recognition result, which greatly improves the recognition accuracy of the pet abnormal behavior and effectively makes up for the limitations of single frame recognition and simple multi-frame fusion.

[0166] In an exemplary embodiment, the process of determining the second behavior recognition result is described taking the time sequence behavior recognition model as a fast-slow network, as shown in Figure 6 the specific processing process of step 501 includes:

[0167] At step 601, each single-frame image within a target time duration is accumulated based on a preset time sequence to obtain a video segment.

[0168] In implementation, the cleaning robot accumulates each single-frame image within a target time duration based on a preset time sequence to obtain a video segment. The target time duration is set according to the typical duration characteristics of the abnormal behavior of the pet, and the process has been described in step 401 of the above embodiment, which will not be repeated here.

[0169] Optionally, during the image frame accumulation process, the single-frame images are strictly arranged in the time stamp sequence of image acquisition to ensure that the video segment can fully reflect the time process of the pet behavior from occurrence to development, and each frame image is subjected to preliminary verification, for example, frames with severe blur or the pet completely moving out of the picture are removed, and finally a continuous and complete video segment is formed.

[0170] At step 602, the video segment is input into the fast-slow network to perform double-path frame sampling through the fast-slow network to obtain a first key frame image sequence corresponding to a slow path and a second key frame image sequence corresponding to a fast path.

[0171] In implementation, after obtaining the video segment, the cleaning robot inputs the video segment into the fast-slow network (e.g., SlowFast network) to perform double-path frame sampling through the fast-slow network to obtain a first key frame image sequence corresponding to a slow path and a second key frame image sequence corresponding to a fast path. Specifically, the slow path (Slow path) in the fast-slow network has the characteristics of high spatial resolution and low temporal resolution, and the fast path (Fast path) has the characteristics of low spatial resolution and high temporal resolution. In this way, the slow path (Slow path) in the fast-slow network samples the video segment with a large time step (e.g., τ = 16), i.e., one frame is selected every 16 frames, 4 key images are extracted from the 64-frame video segment to form the first key frame image sequence, which has a large time span and focuses on capturing the static spatial features of the pet behavior, such as body posture and body shape; the fast path (Fast path) in the fast-slow network samples with a small time step (e.g., β = 2), i.e., one frame is selected every 2 frames, 32 key images are extracted from the 64-frame video segment to form the second key frame image sequence, which has a high temporal resolution and focuses on capturing the dynamic change characteristics of the pet behavior, such as action amplitude and motion trajectory. In this way, through this double-path sampling method, two groups of key frame image sequences are obtained, which can not only retain the overall spatio-temporal context of the behavior, but also take into account the detailed motion information.

[0172] In this embodiment, single-frame images in the target time length are accumulated in a preset time sequence to form a video segment, which completely retains the continuous change process of the pet behavior in the time dimension; then the video segment is input into the fast-slow network for double-path frame sampling, the first key frame image sequence obtained by the slow path can capture the overall spatial features and long-time change trend of the behavior, and the second key frame image sequence obtained by the fast path can accurately capture the detailed changes of the rapid dynamics, this double-path sampling method takes into account both the static features and dynamic features of the behavior, reduces the redundant data processing amount to improve the efficiency, and comprehensively retains the key information, which lays a foundation for subsequent accurate extraction of fusion features and improvement of complex behavior recognition accuracy.

[0173] In an exemplary embodiment, as shown in FIG. 5, the specific processing process of step 502 includes: Figure 7

[0174] Step 701: Double-path feature extraction is performed on the first key frame image sequence and the second key frame image sequence by the fast-slow network to obtain a static feature vector corresponding to the slow path and a dynamic feature vector corresponding to the fast path.

[0175] ​In implementation, the first key frame image sequence and the second key frame image sequence are subjected to double-path feature extraction through the fast and slow networks to obtain a static feature vector corresponding to the slow path and a dynamic feature vector corresponding to the fast path. Specifically, the initial convolutional layer of the slow path uses 1x7x7 3D convolution (time x height x width) with a step size of (1, 2, 2) to process the first key frame image sequence. For example, 4 frames sampled from a continuous 64-frame video segment with τ = 16, a deep 3D convolutional neural network such as 3D ResNet is used to gradually extract spatial detail features in the image, including static information such as the limb structure, torso posture, and facial expression of the pet, and finally encode these features into a fixed-dimension static feature vector to represent the stable spatial attributes of the behavior. The 3D ResNet (three-dimensional residual network) composed of the residual block structure of the slow path is divided into multiple consecutive stages according to function. Taking ResNet-50 (a network containing 50 trainable parameters) as an example, it is usually divided into conv1 (initial convolutional layer), conv2_x to conv5_x (four residual stages), each stage is stacked with several residual blocks, and the output feature map size of each stage is gradually reduced and the channel number is gradually increased, for example, the channel number changes as 64→256→512→1024, and the spatial resolution is reduced. The initial convolutional layer of the fast path uses 5x1x1 3D convolution with a step size of (1, 2, 2) and a channel number that is only 1 / 8 of that of the slow path, and a lightweight residual block is used. In this way, the second key frame image sequence, for example, 32 frames sampled with β = 2, is processed using a lightweight 3D convolution structure to capture the dynamic changes of pet behavior at a higher temporal resolution, such as the rapid swinging of the paw, the amplitude of the body twitch, and the continuous trajectory of the action, and after feature compression, a dynamic feature vector is formed, which usually has a channel number of 1 / 8 of that of the slow path to balance the amount of calculation. Through this double-path parallel extraction method, both the static basic features of the behavior and the dynamic change rules can be accurately captured, providing comprehensive feature support for subsequent feature fusion.

[0176] In this embodiment, the first key frame image sequence and the second key frame image sequence are subjected to double-path feature extraction through the fast and slow networks. The slow path focuses on extracting static features such as the limb structure and torso posture of the pet from key frames with a large time span and forming a static feature vector, which can accurately capture the stable spatial attributes of the behavior. The fast path extracts dynamic features such as paw swinging and body twitching from key frames with high temporal resolution and forms a dynamic feature vector, which can accurately capture the instantaneous motion changes of the behavior. This double-path parallel extraction method not only retains the basic static information of the behavior but also captures the dynamic details, achieving multi-dimensional and comprehensive feature representation of pet behavior and effectively improving the recognition ability of complex behaviors such as abnormal behaviors.

[0177] In an exemplary embodiment, as shown in FIG. 5, the specific processing procedure of step 503 includes: Figure 8

[0178] Step 801, adjust the channel number of the dynamic feature vector corresponding to the fast path, and perform channel alignment on the adjusted dynamic feature vector and the static feature vector corresponding to the slow path.

[0179] In implementation, the cleaning robot adjusts the channel number of the dynamic feature vector corresponding to the fast path through the fast-slow network, and performs channel alignment on the adjusted static feature vector corresponding to the slow path. Specifically, due to the different network structures of the fast and slow paths, the extracted dynamic feature vector and static feature vector usually have different channel numbers, for example, the dynamic feature vector is 64 channels, and the static feature vector is 1024 channels, therefore, it is necessary to adjust the channel dimension of the dynamic feature vector through a 1x1 convolution layer, and map the channel number to be consistent with the static feature vector, for example, gradually expand the channel number from 64 to 256, and further expand it to 512 until it reaches 1028. Furthermore, based on the same channel number, the channel alignment is performed, so that after the channel alignment, the two types of feature vectors have the same dimension scale in the channel dimension, which lays a structural foundation for subsequent feature fusion, and ensures that the dynamic feature and the static feature can be effectively integrated in the same dimension space.

[0180] Step 802, through the time average pooling layer or the convolution layer in the fast-slow network, dimension reduction sampling is performed on the fast path, so that the dynamic feature vector of the dimension reduction sampled fast path and the static feature vector corresponding to the slow path are time aligned.

[0181] In implementation, through the time average pooling layer or the convolution layer in the fast-slow network, dimension reduction sampling is performed on the fast path, so that the dynamic feature vector of the dimension reduction sampled fast path and the static feature vector corresponding to the slow path are time aligned. Since the fast path samples more key frames, as shown in FIG. 8, the number of key frames sampled by the fast path is 32 frames, and the time dimension of the dynamic feature vector is usually greater than that of the slow path (for example, 4 frames), therefore, it is necessary to perform time average pooling or time convolution operation with a step of 4 to reduce the time dimension of the dynamic feature vector from 32 to 4, so as to keep the time dimension consistent with that of the static feature vector, thereby realizing the time alignment between the dynamic feature vector and the static feature vector. The time alignment ensures the positional correspondence of the two types of feature vectors on the time axis, avoids the feature misalignment caused by the difference in time scale, and makes the fused feature accurately reflect the association between the static and dynamic attributes at the same time node. Figure 9 Step 803, perform feature fusion on the dynamic feature vector and the static feature vector after channel alignment and time alignment, to obtain a fused feature vector.

[0182] ​​

[0183] In implementation, the cleaning robot performs feature fusion on the dynamic feature vector and the static feature vector that are aligned in channel and time, to obtain a fused feature vector. The specific fusion position can be after the end of each residual block, to realize multi-level feature interaction. After the double alignment in channel and time dimensions, element-level addition or channel splicing is used for feature fusion.

[0184] The feature fusion by element-level addition mainly strengthens the consistent key information in the two types of features through weighted fusion, for example, the "arched posture" static feature involved in the "vomiting" behavior in pet abnormal behavior is superimposed with the "body twitching" dynamic feature. The feature fusion by channel splicing retains the complete information of the two types of features and forms a higher-dimensional feature vector, for example, 1024+1024=2048 channels. The fused feature vector contains both the static spatial features and the dynamic time features of the pet behavior, and can comprehensively represent the overall attributes and detailed changes of the behavior, providing more abundant and robust feature input for subsequent behavior recognition.

[0185] In this embodiment, the number of channels of the fast-path dynamic feature vector is adjusted to realize channel alignment; then the fast path is dimensionally sampled by a time average pooling layer or a convolution layer to complete time alignment; finally, the feature vectors after double alignment are fused, which not only retains the stable spatial information such as pet limb posture in static features, but also integrates the instantaneous motion information such as action change in dynamic features, so that the fused feature vector can comprehensively and accurately represent the space-time attributes of pet behavior. Further, the accuracy of pet behavior recognition is improved.

[0186] In an exemplary embodiment, as shown in FIG. 5, the specific processing process of step 504 includes: Figure 10

[0187] Step 1001: performing pet behavior recognition on the fused feature vector through the full connection layer in the fast-slow network, and outputting the probability value of each type of pet behavior.

[0188] ​In implementation, after completing the fusion of the feature vectors, the cleaning robot performs pet behavior recognition on the fused feature vectors through the full connection layer and the activation function layer (Softmax) in the fast-slow network, and outputs probability values of each pet behavior. Specifically, after obtaining the fused feature vectors, the fused feature vectors are input into the full connection layer of the fast-slow network, which performs nonlinear transformation and dimension compression on the fused feature vectors, and maps the high-dimensional fused feature to a preset pet behavior category space. The number of neurons of the full connection layer is consistent with the number of behavior categories, and each neuron corresponds to a score of a behavior category. After processing by the Softmax activation function, the scores are converted into probability values corresponding to each category, and the sum of the probability values of all categories is 1. For example, if the fused feature vectors contain typical spatiotemporal features of the “vomiting” behavior, the full connection layer will output a high probability value of the “vomiting” category, while reducing the probability values of other irrelevant categories, thereby quantifying the matching degree between the fused feature and each behavior.

[0189] At step 1002, a second behavior recognition result is determined according to the probability values of each pet behavior.

[0190] In implementation, after the fast-slow network of the cleaning robot outputs the probability values of each pet behavior, the classification layer in the fast-slow network determines the second behavior recognition result according to the probability values of each pet behavior. Specifically, after obtaining the probability values of each pet behavior, the behavior category with the highest probability is first screened out, and then it is determined whether the highest probability value meets a preset behavior recognition probability threshold (usually set to ≥0.75 for abnormal behaviors to reduce false positives). If the highest probability value meets the requirement of the behavior recognition probability threshold, the corresponding behavior category is determined as the second behavior recognition result; if the highest probability value is lower than the behavior recognition probability threshold, or the probability distribution of all categories is relatively average (without a dominant category), the second behavior recognition result is marked as “unidentified”. For example, when the probability value of “hen squatting” is 0.82, which is higher than the behavior recognition probability threshold 0.75 and is the highest value, the second behavior recognition result is “hen squatting”.

[0191] In this embodiment, the fused feature vectors are recognized by the full connection layer of the fast-slow network to output probability values of each behavior, and the second behavior recognition result is determined according to the probability values. By setting a reasonable threshold, the most likely behavior category is screened out, which not only ensures the accuracy of recognition, but also effectively reduces false positives. The overall process fully utilizes the spatiotemporal information contained in the fused feature vectors, and improves the recognition accuracy of complex pet behaviors.

[0192] In an optional embodiment, the second behavior recognition result can also be determined through the TSN time period network. Specifically, the cleaning robot is pre-trained with a time period network, the accumulated multiple frames of images are input into the pre-trained time period network, the accumulated multiple frames of images are segmented into a plurality of segments according to a preset time period through the time period network, and key frame features are extracted from each segment. Then, the spatial branch of the time period network extracts static features of each key frame, and the time sequence branch captures the time evolution law of the behavior by fusing the inter-frame dynamic information of different segments. Next, the pooling layer of the time period network is used for pooling operation to aggregate the feature vectors of all segments to obtain the comprehensive feature representation of the entire behavior sequence, that is, to obtain the fusion feature vector. Finally, the second behavior recognition result is determined through the classifier combined with the multi-segment voting strategy, that is, the behavior category with the second highest confidence and meeting the time sequence logical coherence is selected from all possible behavior categories. The specific processing manner of the embodiment of the present disclosure for determining the second behavior recognition result is not limited.

[0193] In an exemplary embodiment, for the recognition of pet behaviors, the cleaning robot gives accurate cleaning strategies, thereby realizing deep cleaning of the cleaning environment and improving the cleaning coverage and cleaning efficiency of pet-keeping families. As shown in FIG. 11, the method further includes: Figure 11

[0194] Step 1101, determining a target cleaning strategy based on the target behavior recognition result.

[0195] The target cleaning strategy includes one or more of cleaning, avoidance and warning.

[0196] ​In implementation, the cleaning robot determines a target cleaning strategy based on the target behavior recognition result. Specifically, the cleaning robot dynamically matches the corresponding processing strategy according to the target behavior recognition result and the scene features corresponding to the behavior. For example, when the target behavior recognition result is "pet drinking water and leaving the water area", the cleaning robot triggers the targeted cleaning strategy for the pet drinking water behavior, calls the refined sweeping scheme for the sewage area, and cleans the drinking area of the pet; if it is identified that "the pet is eating", the "avoidance" strategy is preferentially executed, the eating area is marked as a to-be-swept area and is swept around to avoid disturbing the pet eating. For another example, the target cleaning strategy also includes a function of warning the user, so when it is identified that "the pet frequently scratches and reaches the threshold", the "warning" strategy is started, and an abnormal pet reminder is sent to the user through the APP (application program); when it is identified that "the pet has a vomiting behavior", a manual cleaning reminder is sent to the user through the APP, and the automatic cleaning of the area is temporarily suspended to prevent secondary pollution. For a complex scene, for example, the pet scratches abnormally while drinking water, the cleaning robot can also integrate multiple cleaning strategies, first executes avoidance to not disturb the pet, and then executes the cleaning and warning strategies after the pet leaves to handle the sewage and remind the user to pay attention to the pet's health, ensuring the comprehensiveness and adaptability of the strategy. The execution examples of the specific cleaning strategies for different target behavior recognition results will be given in the following embodiments of the application, which will not be described here.

[0197] In the embodiment, the target cleaning strategy determined based on the target behavior recognition result includes one or more of cleaning, avoidance and warning, which can realize the accurate adaptation of the cleaning robot to the pet behavior. It not only guarantees the cleaning efficiency, but also embodies the friendly care for the pet. At the same time, through the flexible combination of multiple strategies, it can meet the diversified needs in complex scenes, reduce invalid cleaning and misoperation, and improve the overall cleaning effect.

[0198] In an exemplary embodiment, as shown in Figure 12 The specific processing process of step 1101 includes:

[0199] Step 1201, when the target behavior recognition result is a feeding behavior, a virtual feeding area is constructed based on the position of the pet or the position of the food, and the virtual feeding area is marked as a first cleaning area.

[0200] In implementation, when the target behavior recognition result is the pet's eating behavior, that is, the pet is drinking water or eating in the area, in order not to disturb the normal diet of the pet, the cleaning robot constructs a virtual eating area based on the position of the pet or the position of the eating article (for example, a water basin or a food basin), and labels the virtual eating area as a first cleaning area to be cleaned. Specifically, the cleaning robot accurately locates the current position of the pet or the placement position of the eating article such as a water basin or a food basin through a body sensor such as an RGB camera or an infrared sensor, takes the position as the center, and automatically draws a virtual rectangular or circular area as a virtual eating area according to the body size of the pet, for example, a radius of 50 cm for a small dog or a radius of 80 cm for a large dog, or the regular activity range of the eating article (water basin or food basin), and labels the virtual eating area as the first cleaning area. At the same time, the cleaning robot stores the boundary information, center coordinates and other parameters of the first cleaning area to the map module of the cleaning robot, labels it as a "delayed cleaning area" in the internal map, and sets the cleaning priority of the cleaning area to ensure that the area range can be accurately recognized in subsequent cleaning path planning.

[0201] Step 1202, clean other cleaning areas except the first cleaning area, and continuously monitor the cleaning progress and the activity state of the pet in the first cleaning area.

[0202] In implementation, since the pet is drinking water or eating in the first cleaning area, the cleaning robot selects to clean other cleaning areas except the first cleaning area, that is, the peripheral area of the first cleaning area, based on the detailed sweeping scheme planned according to the target cleaning strategy, to avoid interference with the pet's eating. At the same time, the cleaning robot continuously monitors the cleaning progress of the peripheral cleaning area of the first cleaning area and the activity state of the pet in the first cleaning area to issue motion planning instructions for the next cleaning work.

[0203] Specifically, the cleaning robot first sweeps the space outside the first cleaning area according to the preset global cleaning path, at this time the side brush, the roller brush and the cloth are in the normal working state, to efficiently complete the basic cleaning of other cleaning areas. At the same time, the body sensor of the cleaning robot collects images and infrared data of the first cleaning area in real time, judges whether the pet is still active in the area through a behavior recognition algorithm, for example, whether the pet's head is close to the food basin or the body is within the area boundary, and synchronously records the cleaning completion ratio of other areas, for example, the percentage of the cleaned area to the total area, when the cleaning progress reaches 90% or the pet is detected to leave the first cleaning area, triggers the next cleaning preparation instruction.

[0204] Step 1203, when the cleaning of other cleaning areas is completed and / or the pet has left the first cleaning area, the cleaning of the first cleaning area is performed.

[0205] In implementation, when the other cleaning areas are cleaned and / or the pet has left the first cleaning area, the cleaning robot performs cleaning of the first cleaning area.

[0206] Specifically, if all other areas have been completely cleaned, i.e., the cleaning robot determines that the cleaning progress is 100%, and / or the sensor of the cleaning robot detects that the pet has completely left the first cleaning area, i.e., the pet profile is not captured in the collected images for 5 consecutive seconds in the first cleaning area, the cleaning robot adjusts the cleaning path and drives to the first cleaning area. Cleaning of the first cleaning area is performed. In this process, the cleaning robot automatically switches to the corresponding cleaning mode according to the previously identified diet type, i.e., the pet's drinking or eating behavior: for the drinking area, the side brush is lifted and not working, the roller brush and the cloth are operated according to the sewage cleaning parameters; for the feeding area, the side brush is not working, and the roller brush is raised to increase the speed to sweep the particles. During the cleaning process, the sensor continuously monitors whether the pet returns to the area. If the pet reenters, the cleaning is paused and the area is exited. After the pet leaves again, the work is resumed until the first cleaning area is cleaned.

[0207] The cleaning mode corresponding to the different diet types of the pet will be described in detail in the following embodiments, which will not be described here.

[0208] In this embodiment, by constructing a virtual diet area and marking it as the first cleaning area, the scope that needs special treatment in the cleaning process of the cleaning robot is accurately circled. The other areas outside the first cleaning area are preferentially cleaned to avoid interference of the cleaning behavior on the pet's diet. The cleaning progress and the pet's activity state are continuously monitored to efficiently promote the overall cleaning work and reasonably plan the cleaning rhythm without disturbing the pet. At the same time, it also avoids the possible surprise or secondary pollution of the pet in the presence, greatly improving the accuracy and cleaning efficiency of intelligent cleaning.

[0209] In an exemplary embodiment, as shown in FIG. 12, the specific processing process of step 1203 of performing the cleaning step of the first area includes: Figure 13

[0210] Step 1301: Identify the cleaning area and / or the dirt level of the first cleaning area, and clean the first cleaning area based on the cleaning area and / or the dirt level.

[0211] ​In implementation, after determining to clean the first cleaning area, the cleaning robot identifies the cleaning area and / or the dirtiness degree of the first cleaning area, and formulates a corresponding cleaning mode and cleaning scheme based on the cleaning area and / or the dirtiness degree to clean the first cleaning area. Specifically, the cleaning robot performs a comprehensive scan on the first cleaning area through the RGB sensor and the infrared sensor carried by the robot body, determines the actual area within the area boundary through image segmentation technology, and compares the actual area with a preset area threshold (for example, 0.5 square meters) to determine the size of the cleaning area. The identification of the dirtiness degree of the first cleaning area is performed by analyzing the gray value and texture features of stains in the image, such as the reflective area of sewage and the distribution density of food particles, converting the gray value and the texture features into a quantitative dirtiness index, and comparing the dirtiness index with a dirtiness threshold to determine the dirtiness degree of the area. Then, based on the identification result, the cleaning robot automatically matches the cleaning parameters to determine the specific cleaning mode and cleaning scheme to clean the first cleaning area.

[0212] Optionally, the process of identifying the dirtiness degree of the first cleaning area: by processing the preset dirtiness index, a quantitative dirtiness score can be obtained, for example, the dirtiness score is 1-10 points, so that the dirtiness threshold is preset as 6 points, and thus the size relationship between the quantitative dirtiness index and the dirtiness threshold can determine the dirtiness degree of the first cleaning area.

[0213] In this embodiment, the cleaning range is determined according to the size of the cleaning area of the cleaning area, and the cleaning intensity is adjusted according to the dirtiness degree, so that the cleaning effect of the first cleaning area is ensured, the increase of energy consumption and the wear of components caused by excessive cleaning are avoided, and the efficiency and rationality of cleaning are further improved.

[0214] In an exemplary embodiment, since the first cleaning area is the eating area of the pet, examples of identifying the pet drinking water or eating behavior in the first cleaning area are taken to illustrate the target cleaning strategy of the cleaning robot in detail, as shown in Figure 14 The specific processing process of step 1301 includes:

[0215] Step 1401: when the cleaning area of the first cleaning area is greater than the area threshold or the dirtiness degree is greater than the dirtiness threshold, instructing the cleaning robot to return to the base station to clean the cleaning part of the cleaning robot.

[0216] In implementation, because the cleaning robot is relatively dirty after cleaning other areas, if it directly cleans the first cleaning area, it is easy to cause the dirt to expand, causing secondary pollution to the first cleaning area. Therefore, in order to ensure that the cleaning task can be successfully completed, before the cleaning robot starts cleaning the first cleaning area, the state of the cleaning element can be evaluated based on the cleaning degree required by the cleaning task of the first cleaning area, to determine whether the cleaning element needs to be cleaned. Specifically, for the pet behavior type of the first cleaning area being pet drinking behavior, the cleaning robot determines through sensor data that the cleaning area of the first cleaning area exceeds a preset area threshold, or the dirt degree exceeds a dirt threshold. In this way, when the cleaning area of the first cleaning area is greater than the area threshold or the dirt degree is greater than the dirt threshold, it means that there may be more water in the first cleaning area. The cleaning robot can first be instructed to return to the base station to clean the cleaning element of the cleaning robot, for example, to perform cleaning and drying treatment, to prevent the area of dirt from expanding, and to avoid secondary pollution to the cleaning area, and then to clean the first cleaning area. That is, the processing unit of the cleaning robot sends an instruction to the cleaning robot to return to the base station. After receiving the instruction, the cleaning robot suspends the current cleaning preparation work, and autonomously returns to the base station according to the planned optimal path, and the base station starts cleaning the cleaning element.

[0217] For the pet behavior type of the first cleaning area being pet eating behavior, the cleaning robot determines through sensor data that the cleaning area of the first cleaning area exceeds a preset area threshold, or the dirt degree exceeds a dirt threshold. In this way, before formally starting to clean the area, the cleaning robot will actively return to the base station to clean the cloth comprehensively to remove residual stains, and perform moderate drying treatment. Considering that the cleaning of the first cleaning area is usually dominated by particulate matter, the core purpose of cleaning the cleaning element is to avoid secondary pollution of residual dirt on the cloth to subsequent cleaning, and to ensure the cleaning effect. Therefore, the cleaning element is mainly cleaned, for example, to remove entangled fibers and embedded particles in the gap, and the like. In this way, after completing the preparation work, the cleaning robot goes to the to-be-cleaned area to perform the first complete cleaning.

[0218] In an optional embodiment, the cleaning process on the cleaning component of the cleaning robot includes drying treatment and cleaning treatment on the cleaning component of the cleaning robot. Specifically, the cleaning component includes a side brush, a roller brush and a cloth, and the cleaning treatment is performed on the cloth and the roller brush which are prone to dirt adhesion. The base station first flushes the surface of the cloth with high-pressure water flow to remove dirt, food residue or hair, and simultaneously starts the built-in brush to clean the roller brush in depth to remove the entangled fibers and the embedded particles. For stubborn stains (such as dried oil stains), a neutral cleaning agent is added to assist in dissolving, ensuring that there is no residual dirt on the surface of the cleaning component. After the cleaning is completed, the drying treatment is performed: the base station heats and dries the cleaning component through a hot air circulation system, and the drying temperature is set according to the material of the cleaning component, while the air flow disturbance is used to ensure uniform drying and avoid local moisture breeding bacteria. In addition, the drying time is dynamically adjusted according to the dirt level of the cleaning component: if there is still a small amount of water stain left after cleaning, the drying time is extended until it is completely dry. Through the combination of cleaning and drying, the dirt on the cleaning component can be completely removed to prevent secondary pollution, and the cleaning component can be kept dry and clean to provide reliable guarantee for subsequent cleaning of the first cleaning area, improving the overall cleaning efficiency and hygiene standard.

[0219] In step 1402, the first cleaning area is cleaned based on the cleaning component after the cleaning treatment.

[0220] In implementation, after the cleaning component of the cleaning robot is cleaned and dried by the base station, the cleaning robot re-plans the path to return to the first cleaning area and starts the targeted cleaning process. For the cloth, due to the deep cleaning and intensive drying, it can effectively avoid bringing residual dirt to the cleaning area. During the cleaning process, the cleaning robot adjusts the coverage density of the cleaning path according to the cleaning area and the dirt level of the first cleaning area, for example, a large area of water stain identified in the early stage. For the areas with serious dirt, the reciprocating cleaning mode is used to increase the contact time between the cloth and the ground. For the edges and corners, the side brush is used to help gather dirt, and then the roller brush and the cloth are used for cooperative processing. At the same time, the cleaning robot monitors the state of the cleaning component in real time, and if it finds that the cloth again absorbs a lot of dirt or the roller brush is slightly entangled, it will automatically shorten the interval with the base station to clean the cleaning component again, ensuring that the cleaning component is always clean to perform the cleaning task, and finally realizing the thorough cleaning of the first cleaning area and avoiding secondary pollution.

[0221] In this embodiment, for the case that the cleaning area of the first cleaning area is too large or the dirt level is high, the cleaning robot is first instructed to return to the base station for cleaning treatment of the cleaning component, and then the cleaned cleaning component is used for cleaning, avoiding secondary pollution caused by the dirty cleaning component in the large area or high dirt area, and ensuring that the cleaning component is in the best state for cleaning work. The cleaning effect is significantly improved.

[0222] In an exemplary embodiment, as shown in Figure 15 When the cleaning member includes the side brush, the roller brush and the cloth, the specific process of cleaning the first cleaning area in step 1402 includes:

[0223] In step 1501, the side brush is controlled to be in the non-cleaning position, and the roller brush in the cleaning position is controlled to rotate at a first cleaning speed, and the cloth in the cleaning position is controlled to rotate at a second cleaning speed, to clean the first cleaning area.

[0224] In implementation, when the cleaning robot cleans the first cleaning area with a cleaning area greater than the area threshold or a dirt level greater than the dirt threshold, the side brush can be controlled to be in the non-cleaning position, i.e. the side brush is lifted and not working, and at the same time, the roller brush in the cleaning position is controlled to rotate at a first cleaning speed, and the cloth in the cleaning position is controlled to rotate at a second cleaning speed, to clean the first cleaning area.

[0225] Specifically, the cleaning strategy of the cleaning robot is different for different types of dirt in the first cleaning area. For example, for the first cleaning area identified by the pet drinking behavior standard, the main cleaning problem is the presence of dirty water. At this time, if the cleaning area of the first cleaning area is greater than the area threshold, or the dirt level is greater than the dirt threshold, the cleaning robot issues a control instruction to lift or store the side brush to the non-working position, for example, the side brush is lifted upward to the target position of the chassis, to avoid stirring the dirty water and other dirt in the first cleaning area during the cleaning process, to prevent the spread of stains causing secondary pollution. At the same time, ensure that the roller brush and the cloth are in the cleaning position in contact with the ground: the roller brush rotates at a preset first cleaning speed, which is the normal cleaning speed of the roller brush. The cloth of the cleaning robot rotates at a second cleaning speed to clean the dirty water stains and other dirt in the first cleaning area. The second cleaning speed is the normal cleaning speed of the cloth, which can effectively wipe the ground stains while cleaning the ground at the normal cleaning speed, reducing the splashing of dirty water caused by high-speed friction, and the normal cleaning speed can make the cloth contact the ground more fully, improving the adsorption effect on local stubborn stains. At this time, the second cleaning speed can be the same as the first cleaning speed.

[0226] For example, in the first cleaning area identified based on pet eating behavior, the main issue is cleaning solid particles such as cat food crumbs and dog food pellets. If the cleaning area of ​​the first cleaning area exceeds a certain threshold, the cleaning robot issues a control command to raise or retract the side brush to a non-working position. For instance, the side brush is raised to a target position on the chassis to prevent it from agitating food particles and other dirt during cleaning, thus preventing secondary pollution. Simultaneously, the cleaning robot ensures the roller brush and mop are in contact with the ground: the roller brush rotates at a preset first cleaning speed, which is its high-speed cleaning capability. This first cleaning speed allows it to quickly gather and suck up dispersed dirt, such as cat food crumbs and dog food pellets, into the dust collection box. Then, the cleaning robot's mop rotates at a second cleaning speed, lower than the first cleaning speed of the roller brush, to clean the ground in the first cleaning area. In this case, the first cleaning speed is greater than the second cleaning speed.

[0227] In this embodiment, placing the side brush in a non-cleaning position can prevent food particles, sewage, and other dirt from spreading outwards from the first cleaning area when it rotates, thus reducing the risk of secondary pollution. Furthermore, the roller brush and the cloth in the cleaning position are controlled to perform targeted cleaning treatment on the first cleaning area at a first cleaning speed and a second cleaning speed, respectively, making the cleaning of the first cleaning area more thorough and precise.

[0228] In one exemplary embodiment, such as Figure 16 As shown, after step 1402, the method further includes:

[0229] Step 1601: Based on the degree of dirt in the first cleaned area after the initial cleaning, determine whether to clean the cleaning parts and repeat the cleaning of the first cleaned area.

[0230] In practice, after the initial cleaning of the first cleaning area, the cleaning robot performs a second scan using its body sensors to re-evaluate the degree of residual dirt within the area. For example, image analysis can be used to calculate indicators such as the percentage of dirt area and particle density to detect the degree of dirt in the first cleaning area. If the degree of residual dirt is still higher than a preset threshold, the first cleaning area is determined to require further processing. At this point, the cleaning robot will assess the current state of the cleaning components to determine whether to perform further cleaning. If further cleaning is required, the robot will be instructed to return to the base station to perform targeted cleaning of the roller brush and cloth, and then replan its path to return to the first cleaning area to repeat the cleaning process. This process is repeated at least once or multiple times until the cleanliness of the first cleaning area meets the preset cleaning standard, at which point the cleaning of the first cleaning area is complete.

[0231] In this embodiment, by performing dirt assessment on the first cleaned area, if the dirt is still not up to standard, timely cleaning of the cleaning element can prevent the residual dirt carried by the cleaning element from causing secondary pollution, and the first cleaning area is repeatedly cleaned again. In this way, the cleaning element is put into work in a clean state, and the cleaning parameters adjusted in a targeted manner can more efficiently handle the remaining stains. If the dirt has reached the standard, no additional operation is needed, reducing unnecessary energy consumption and cleaning element wear. Both the cleaning quality of the first cleaning area and efficiency and resource conservation are taken into account, making the cleaning process more intelligent and reasonable.

[0232] In an optional embodiment, when the first cleaning area is repeatedly cleaned, for example, when the first cleaning area is repeatedly mopped, the drying degree of the mop gradually increases with the increase of the number of repeated mopping, forming a dynamic adjustment mechanism that the number of repeated mopping is proportional to the drying degree. Specifically, when the mop is used for the first time, the drying degree of the mop is low to enhance the adsorption capacity of the mop for liquid dirt, so as to ensure that the residual water marks can be effectively wiped off. When the mop is used for the second time, the dirt on the ground has been greatly reduced, and only a small amount of water stains remain. At this time, the base station appropriately increases the drying degree of the mop to reduce the moisture carried by the mop, so as to avoid secondary wetting of the ground. As the number of repeated mopping continues to increase (such as the third time and above), the ground tends to be clean, and only the surface water stains need to be handled. The drying degree of the mop is further increased, and even close to completely dry. Through the dry mop, the residual moisture is quickly absorbed, and the ground is quickly dried. This design of dynamically adjusting the drying degree according to the repeated mopping stage not only ensures efficient removal of stains in the early stage, but also avoids the ground from being re-humidified due to the mop being too wet in the later stage. Finally, the balance between cleaning effect and drying efficiency is achieved, so that the first cleaning area can quickly recover to a dry state while being thoroughly clean.

[0233] In an exemplary embodiment, as shown in Figure 17 the specific processing process of step 1601 of repeatedly cleaning the first cleaning area includes:

[0234] In step 1701, the edge brush is controlled to be in the non-cleaning position, and the roller brush in the cleaning position is controlled to have a third cleaning speed, and the mop in the cleaning position is controlled to have a first cleaning speed, to clean the first cleaning area.

[0235] In implementation, after the first cleaning area is cleaned for the first time, when it is determined by the cleaning robot that repeated cleaning of the first cleaning area is needed, to ensure the cleaning effect of the first cleaning area, the cleaning element can be cleaned repeatedly first to ensure that the cleaning element does not cause secondary pollution to the first cleaning area, and then, after the cleaning element is cleaned, the cleaning robot adjusts the parameters according to the type of residual dirt to determine the cleaning mode of the cleaning element, thereby realizing repeated cleaning of the first cleaning area. Specifically, when the first cleaning area is mainly composed of solid particles, whether the cleaning area of the first cleaning area is greater than the area threshold or the cleaning area of the first cleaning area is less than the area threshold, the cleaning robot controls the side brush to be in a non-cleaning position, i.e., lifted and not working, to avoid the side brush disturbing the particulate matter and causing secondary pollution, and at the same time, the roller brush is operated at a high speed (third cleaning speed) and the rotation speed of the cloth (first cleaning speed) is appropriately increased to clean the first cleaning area one or more times. At this time, the third cleaning speed is equal to the first cleaning speed.

[0236] When the first cleaning area is mainly composed of sewage, if the cleaning area of the first cleaning area is greater than the area threshold or the degree of dirt is greater than the dirt threshold, the cleaning robot controls the side brush to be in a non-cleaning position, i.e., lifted and not working, to avoid the side brush disturbing the sewage area and causing secondary pollution, and at the same time, the roller brush is operated at a reduced speed (third cleaning speed) and the rotation speed of the cloth (first cleaning speed) is appropriately increased to clean the first cleaning area one or more times until the first cleaning area is cleaned. At this time, the third cleaning speed is less than the first cleaning speed. If the cleaning area of the first cleaning area is less than the area threshold or the degree of dirt is less than the dirt threshold, the cleaning robot adopts the same cleaning mode as the greater case to clean the first cleaning area one or more times until the first cleaning area is cleaned. In addition, if the cleaning area of the first cleaning area is greater than the area threshold and the degree of dirt is less than the dirt threshold, the cleaning robot controls the side brush and the roller brush to be in a non-cleaning position, i.e., lifted and not working, to avoid the side brush disturbing the sewage area and causing secondary pollution, and at the same time, the cloth is operated at a normal speed to clean the first cleaning area one or more times until the first cleaning area is cleaned.

[0237] In an exemplary embodiment, the first cleaning area demarcated for the identified pet eating behavior can have solid particles, such as cat food crumbs, dog food particles, and the like. When the first cleaning area has solid particles, the first cleaning speed of the roller brush during the cleaning of the first cleaning area can be determined based on the particle density of the solid particles, or the first cleaning speed of the roller brush is inversely proportional to the second cleaning speed of the side brush. Specifically, when the first cleaning area has solid particles, such as pet food crumbs and granular residues, the first cleaning speed of the roller brush of the cleaning robot can be dynamically adjusted according to the actual scene. For example, the cleaning robot identifies the particle density of the solid particles through a sensor. When it is detected that the solid particles are densely distributed in the area, for example, more than 5 particles per square centimeter, the first cleaning speed of the roller brush is automatically increased to a higher gear to quickly gather and remove a large amount of dirt by using stronger centrifugal force and rolling-in capacity. If the number of solid particles is small, for example, less than 2 particles per square centimeter, the rotation speed is appropriately reduced, so that the cleaning effect is ensured while reducing energy consumption and component wear. In another optional implementation, the first cleaning speed of the roller brush can be inversely proportional to the second cleaning speed of the cloth. When solid particles need to be cleaned first, the roller brush runs at high speed (for example, 1800 rpm), and the cloth is reduced to low speed (for example, 400 rpm) to avoid the cloth from raising particles when it rotates quickly. When the number of solid particles decreases and the remaining stains need to be wiped, the rotation speed of the roller brush is reduced (for example, 1000 rpm), and the rotation speed of the cloth is increased accordingly (for example, 800 rpm) to balance the cleaning and wiping functions through dynamic allocation of rotation speed, so that solid particles are efficiently removed while ground stains are also fully treated. At the same time, by controlling the first cleaning speed of the roller brush and the second cleaning speed of the cloth to be inversely proportional, the power is balanced, which can increase the endurance of the cleaning robot, and thus improve the overall cleaning quality and cleaning efficiency.

[0238] In an exemplary embodiment, since the first cleaning area is the pet eating area, the target cleaning strategy of the cleaning robot is described by taking the identified pet drinking behavior or eating behavior in the first cleaning area as an example, as shown in FIG. 13. Figure 18 The specific processing process of step 1301 includes:

[0239] Step 1801, when the cleaning area of the first cleaning area is less than the area threshold and / or the dirt level is less than the dirt threshold, the first cleaning area is cleaned based on the cleaning element of the cleaning robot.

[0240] In implementation, when the cleaning area of the first cleaning area is less than the area threshold or the dirtiness is less than the dirtiness threshold, the cleaning robot does not need to return to the base station due to light dirtiness, and can directly use the current cleaning element to clean the first cleaning area with softer parameters. Specifically, the cleaning robot determines that the first cleaning area belongs to a small range or light dirtiness, and does not need to start the intensive cleaning process, but directly calls the regular cleaning mode to perform the first cleaning on the first cleaning area.

[0241] In step 1802, whether the cleaning element needs to be cleaned is determined based on the dirtiness of the first cleaning area after the first cleaning, and the first cleaning area is repeatedly cleaned.

[0242] In implementation, after the first cleaning of the first cleaning area, the cleaning robot re-detects the dirtiness of the first cleaning area. If the dirtiness of the first cleaning area still does not meet the cleaning standard, the cleaning robot determines whether the current cleaning element needs to be cleaned again based on the dirtiness of the first cleaning area. If the cleaning element needs to be cleaned, the first cleaning area is repeatedly cleaned after the cleaning element is processed. If the current cleaning element remains relatively clean, the first cleaning area can be directly repeatedly cleaned based on the current cleaning element.

[0243] Specifically, after the first cleaning of the first cleaning area, the sensor quickly scans the residual dirt of the first cleaning area. If the dirtiness is below the preset secondary cleaning threshold, the cleaning element does not need to be processed, and the first cleaning area is not repeatedly cleaned. If there is still a small amount of residual dirt, such as local water stains that are not dry, it is determined that the first cleaning area needs to be repeatedly cleaned. However, due to light dirtiness, the cleaning robot does not need to return to the base station, and can directly use the current cleaning element to perform one-time supplementary cleaning with softer parameters, which ensures the cleanliness of the area and avoids the efficiency loss caused by frequent return to the base station. If it is detected that the cleaning element needs to be cleaned, the first cleaning area is repeatedly cleaned after the cleaning element is processed.

[0244] In this embodiment, for the case that the first cleaning area is small and / or the dirtiness is low, the cleaning element of the cleaning robot is directly used for cleaning, which can quickly and efficiently complete the basic cleaning and avoid wasting time and energy due to starting a complex process. If the dirtiness meets the standard, no additional operation is needed, which reduces unnecessary resource consumption. If there is still a small amount of residual dirt, targeted processing is performed, which ensures the cleaning effect and does not increase the wear and tear of the cleaning element and energy consumption due to excessive cleaning.

[0245] In an exemplary embodiment, the cleaning element of the cleaning robot includes a side brush, a roller brush, a mop, and the like. Figure 19As shown, in step 1801, the first cleaning area is cleaned based on the cleaning element of the cleaning robot, specifically including:

[0246] In step 1901, the side brush is controlled to be in the non-cleaning position, and the roller brush in the cleaning position and the cloth in the cleaning position are both controlled to clean the first cleaning area at the second cleaning speed.

[0247] In implementation, when the cleaning area of the first cleaning area is smaller than the area threshold and / or the dirtiness is smaller than the dirtiness threshold, the cleaning robot controls the side brush to be in the non-cleaning position, i.e., to be lifted and not to work, and controls the roller brush in the cleaning position and the cloth in the cleaning position to clean the first cleaning area at the second cleaning speed.

[0248] Specifically, the cleaning robot first issues an instruction to lift or store the side brush to the non-working position on the side of the body, so as to avoid spreading a small amount of dirt to the cleaned area when the side brush rotates in the first cleaning area with a small area or light dirt. At the same time, the cleaning state of the roller brush and the cloth in contact with the ground is ensured, and they are uniformly operated at the second cleaning speed, i.e., the normal cleaning speed of the roller brush and the cloth. In this way, the roller brush sweeps a small amount of food particles or debris in the area at the speed, which not only avoids energy waste caused by high-speed operation, but also avoids excessive friction on the ground. The cloth is matched at the same speed to wipe a small amount of water stains or oil stains on the ground by moderate friction frequency, without the need to increase the speed to achieve the cleaning effect.

[0249] In the embodiment, by controlling the side brush to be in the non-cleaning position, the spread of a small amount of dirt in the first cleaning area by the rotation of the side brush is avoided, and secondary pollution is prevented. By allowing the roller brush and the cloth in the cleaning position to work at the second cleaning speed, the cleaning demand for a small range and light dirt is met, and unnecessary energy consumption and component wear caused by high-speed operation are avoided. Energy saving and equipment protection are considered, and the cleaning efficiency of a small area and light dirt area is improved.

[0250] In an exemplary embodiment, as shown in Figure 20 In step 1101, the target cleaning strategy is determined based on the target behavior recognition result, specifically including:

[0251] In step 2001, when the target behavior recognition result is the sleep behavior, the licking behavior or the staying behavior, the second cleaning area is determined based on the pet position.

[0252] In implementation, when the target behavior recognition result of the pet is sleeping behavior, licking behavior or staying behavior, the cleaning robot marks the second cleaning area based on the current pet location of the pet. Specifically, the cleaning robot captures the behavior state of the pet in real time through a camera, an infrared sensor or the like, and once it is identified that the pet is in a state of sleeping, licking or staying at a place for a long time, the second cleaning area is marked with the current location of the pet as the core, combined with the pet size and the activity range. For example, for a small pet, a circular area with a radius of 60 cm is marked with the center of the body as the origin, and for a large pet, the area is expanded to a radius of 100 cm, and the area is marked as a "delayed cleaning area" in the internal map to avoid the cleaning robot entering when the pet is active.

[0253] In step 2002, the activity state of the pet in the second cleaning area is continuously monitored until the cleaning of the other cleaning areas except the second cleaning area is completed and / or the pet has left the second cleaning area, and the cleaning of the second cleaning area is performed.

[0254] In implementation, the cleaning robot cleans the other areas (peripheral areas) except the second cleaning area, and at the same time, continuously monitors the cleaning progress of the other areas and / or the activity state of the pet in the second cleaning area until the cleaning of the other cleaning areas except the second cleaning area is completed and / or the pet has left the second cleaning area, and the cleaning of the second cleaning area is performed.

[0255] Specifically, the cleaning robot preferentially performs regular cleaning on the space outside the second cleaning area to avoid disturbing the pet that is sleeping, licking or staying, and at the same time, tracks the dynamic of the pet in the second cleaning area in real time through a sensor: if the pet is still in the state of sleeping, licking or staying, the cleaning robot will bypass the area for cleaning; if it is detected that the pet has left or the other areas have completed the entire cleaning, the cleaning robot can adjust the path to go to the second cleaning area for cleaning. During the cleaning of the second cleaning area, the parameters are adjusted according to the type of dirt that may exist in the second cleaning area, for example, the winding processing capability of the roller brush on hair is improved, the mop is operated at low humidity to avoid disturbing the pet that may return, and the thorough cleaning of the area is ensured without disturbing the pet in rest or activity.

[0256] In this embodiment, for the behaviors of the pet such as sleeping, licking or staying, the second cleaning area is marked and the cleaning is delayed to avoid the cleaning robot intruding when the pet is resting or quietly active, and to reduce the disturbance to the pet; the pet state is continuously monitored and the second area is cleaned after the cleaning of the other areas is completed or the pet leaves, which can not only ensure the orderly progress of the overall cleaning work, but also allow the pet to be in a comfortable environment without disturbance, and at the same time, ensure that the dirt in the second area is cleaned in time, and the cleaning of the second cleaning area is intelligent and accurate.

[0257] In an exemplary embodiment, as shown in Figure 21 The step 1101 of determining a target cleaning strategy based on the target behavior recognition result includes the following steps:

[0258] In step 2101, when the target behavior recognition result is an abnormal behavior, an early warning notification message is sent to the user based on the number of occurrences and / or the duration of the abnormal behavior.

[0259] In implementation, when the target behavior recognition result is an abnormal behavior, the cleaning robot can send an early warning notification message to the user based on the number of occurrences and / or the duration of the abnormal behavior. Specifically, the cleaning robot can identify the abnormal behavior of the pet through a camera, a sound sensor, or other devices, such as persistent pacing, frequent scratching at the same location, etc. In addition, to avoid false positives and false negatives, the cleaning robot can also remind the user when the number of occurrences and / or the duration of the abnormal behavior reaches a preset threshold. For example, if the number of occurrences of the abnormal behavior within 1 hour exceeds 3 times, or the duration of a single occurrence exceeds 5 minutes, the cleaning robot will automatically trigger the early warning mechanism and push a notification message containing the type of abnormal behavior, the time period of occurrence, and the real-time snapshot to the user through the mobile APP, reminding the user to pay attention to the pet's state in time and check whether there are health risks or environmental problems.

[0260] In step 2102, when the target behavior recognition result is an abnormal behavior, a virtual obstacle avoidance area is constructed based on the pet's location, and an obstacle avoidance cleaning mode is executed to avoid the virtual obstacle avoidance area.

[0261] In implementation, when the target behavior recognition result is an abnormal behavior, the cleaning robot can also construct a virtual obstacle avoidance area based on the pet's location and execute an obstacle avoidance cleaning mode to avoid the virtual obstacle avoidance area. Specifically, once the cleaning robot identifies that the pet is in an abnormal state, it will construct a virtual obstacle avoidance area centered on the pet's current location and combined with the pet's activity trajectory. For example, if the pet is persistently pacing around the sofa, the sofa and the surrounding 1-meter range will be designated as an obstacle avoidance area, and marked as a prohibited entry area in the cleaning map. In this way, the cleaning robot will automatically plan a path to avoid the virtual obstacle avoidance area when executing the cleaning task, so as to avoid approaching the pet and causing further emotional fluctuations. At the same time, the cleaning robot will reduce its operating noise to reduce additional stimulation to the pet, ensuring that the cleaning work in other areas is completed without disturbing the pet, until the pet's abnormal behavior is resolved or the user manually adjusts the cleaning strategy.

[0262] In this embodiment, when the abnormal behavior of the pet is identified, a warning is sent according to the number of occurrences and the duration of the abnormal behavior, so that the user can know the abnormal state of the pet in time and take corresponding measures quickly. The virtual obstacle avoidance area is constructed based on the position of the pet, and the obstacle avoidance cleaning is performed, so that the cleaning robot can avoid disturbing the pet and prevent the pet from being in an abnormal state. At the same time, the cleaning work in other areas can be carried out normally, so that the comfort of the pet is ensured and the overall cleaning effect is not affected, and the balance between the care of the pet and the cleaning demand is achieved.

[0263] In an exemplary embodiment, as shown in FIG. 21, the specific processing process of sending a warning notification message to the user based on the number of occurrences and / or the duration of the abnormal behavior in step 2101 includes: Figure 22

[0264] In step 2201, when the number of occurrences and / or the duration of the same abnormal behavior is greater than a preset threshold, a warning notification message is sent to the user.

[0265] In implementation, in order to avoid the situation that the pet is touched by mistake or the abnormal behavior is misreported, the cleaning robot can also set a threshold for the number of occurrences and the duration of the abnormal behavior, so that the cleaning robot sends a warning notification message to the user only when the number of occurrences and / or the duration of the same abnormal behavior is greater than the preset threshold.

[0266] Specifically, the cleaning robot can record and count the abnormal behavior of the pet, such as frequently hitting objects, not eating or drinking for a long time, and hiding for a long time. At the same time, a corresponding judgment threshold is preset, for example, the threshold for the number of occurrences of the same abnormal behavior within 24 hours is 5 times, and the threshold for the duration of a single occurrence is 30 minutes. In this way, if it is monitored that the pet hits the furniture frequently for 6 times within a day, or the duration of not eating or drinking continuously is more than 40 minutes, both of which meet the condition of being greater than the preset threshold, the cleaning robot can integrate the specific information of the abnormal behavior, including the behavior type, the cumulative number of times, the total duration, the occurrence time period, and the associated environment picture, and push a warning notification through the mobile phone application bound by the user. Optionally, the notification content of the warning notification can explicitly prompt the key data of the abnormal behavior, such as "the pet's hitting behavior has reached 6 times today, which exceeds the threshold of 5 times", and suggest the user to check whether the pet has injury, stress or health problem, so as to help the user discover potential risks in time and provide timely attention and care for the pet.

[0267] ​In this embodiment, the occurrence frequency and duration of the same abnormal behavior are determined by setting a preset threshold value, and when the threshold value is exceeded, a warning notification is sent to the user, which can effectively avoid misjudgment of occasional and short-term abnormal behaviors of the pet and ensure the accuracy of the warning. This mechanism prevents the user from being disturbed by irrelevant information, and at the same time, when the pet has a persistent or frequent abnormal behavior, the user can receive a timely reminder, so as to quickly intervene to understand the pet's condition, investigate whether there is a health problem, an unsuitable environment, etc., provide timely care and help for the pet, and ensure the safety and health of the pet.

[0268] In an exemplary embodiment, as shown in Figure 23 the abnormal behavior corresponds to a reporting priority, and the method further comprises:

[0269] Step 2301, when the target behavior recognition result contains multiple abnormal behaviors, a warning notification message is sent to the user based on the abnormal behavior with the highest reporting priority.

[0270] In implementation, in the process of identifying the behavior of the pet, when the target behavior recognition result contains multiple abnormal behaviors, the cleaning robot will send a warning notification message to the user based on the abnormal behavior with the highest reporting priority.

[0271] Specifically, the cleaning robot will pre-set a clear reporting priority level for different types of pet abnormal behaviors, for example, behaviors that may endanger the life safety of the pet are listed as the highest priority, such as hen squatting behavior, vomiting behavior, etc. Behaviors that affect the health of the pet are listed as medium priority, such as scratching, etc. And behaviors that only reflect emotional abnormalities are listed as low priority. In this way, when the cleaning robot identifies multiple abnormal behaviors at the same time, it will automatically compare the priority levels of each abnormal behavior, filter out the behavior with the highest priority, and generate a warning notification message around the abnormal behavior. The content of the warning notification message will specify the type, occurrence time, duration and on-site screenshot of the abnormal behavior with the highest priority, and briefly mention the existence of other low-priority abnormal behaviors, which ensures that the user can first pay attention to the most urgent situation, and also enables the user to fully understand the overall state of the pet in order to take targeted measures.

[0272] In this embodiment, when the target behavior recognition result contains multiple abnormal behaviors, a warning notification message is sent based on the abnormal behavior with the highest reporting priority, which avoids the user being disturbed by a large amount of information and ignoring the core situation, ensures that the user can know the problem that threatens the safety and health of the pet at the first time and handle it in time, and also enables the user to pay attention to other abnormalities after handling the high-priority problem, which improves the response efficiency and better safeguards the safety and health of the pet.

[0273] It should be understood that although the steps in the flowcharts involved in the above embodiments are shown in sequence according to the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the above embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0274] Based on the same inventive concept, the embodiments of the present application also provide a cleaning device for implementing the above-mentioned cleaning method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more obstacle cleaning device embodiments provided below can refer to the limitations of the obstacle cleaning method in the above text, which will not be repeated here.

[0275] In one exemplary embodiment, as shown in Figure 24 A cleaning device 2400 is provided for cleaning a robot, comprising: a first identification module 2401, a second identification module 2402, a third identification module 2403, and a first determination module 2404, wherein:

[0276] The first identification module 2401 is configured to obtain a single frame image and identify the single frame image based on a preset deep learning algorithm to obtain an initial behavior identification result;

[0277] The second identification module 2402 is configured to perform fusion judgment on the initial behavior identification result corresponding to each single frame image based on a multi-frame image fusion judgment strategy to obtain a first behavior identification result;

[0278] The third identification module 2403 is configured to accumulate each single frame image if the first behavior identification result is an identification anomaly, and perform time sequence behavior identification on the accumulated multi-frame images according to a time sequence behavior identification algorithm to determine a second behavior identification result;

[0279] The first determination module 2404 is configured to determine a target behavior identification result based on the first behavior identification result and the second behavior identification result; the target behavior identification result is used to determine a target cleaning strategy for cleaning processing.

[0280] In one embodiment, the first identification module 2401 is specifically configured to:

[0281] An image of a single frame of a pet activity scene is acquired by the image acquisition unit;

[0282] The single frame image is input into a pre-trained deep learning model, and feature extraction and pet behavior category prediction are performed on the single frame image based on the deep learning model, and probability values of various pet behaviors are output.

[0283] According to the probability values of various pet behaviors, an initial behavior recognition result is obtained.

[0284] In one of the embodiments, the second recognition module 2402 is specifically configured to:

[0285] If the initial behavior recognition results corresponding to the continuous multiple single frame images are all the same, the initial behavior recognition result is determined as the first behavior recognition result.

[0286] If the initial behavior recognition results corresponding to the continuous multiple single frame images are not all the same, the first behavior recognition result is determined as a default value.

[0287] In one of the embodiments, the first behavior recognition result is an identification exception, including that the first behavior recognition result is a default value, and / or the first behavior recognition result is that the pet has an abnormal behavior.

[0288] In one of the embodiments, the third recognition module 2403 is specifically configured to:

[0289] Based on the preset time sequence, each single frame image within the target time length is accumulated to obtain a multi-frame image sequence.

[0290] The spatial feature extraction is performed on the multi-frame image sequence by the time sequence behavior recognition model to obtain a feature vector.

[0291] Based on the time sequence correlation, the feature vectors are fused to obtain a fused feature vector.

[0292] The pet behavior recognition is performed on the fused feature vector to determine a second behavior recognition result.

[0293] In one of the embodiments, the third recognition module 2403 is specifically configured to:

[0294] Based on the preset time sequence, each single frame image within the target time length is accumulated to obtain a video segment.

[0295] The video segment is input into the fast-slow network, and the double-path frame sampling is performed by the fast-slow network to obtain a first key frame image sequence corresponding to a slow path and a second key frame image sequence corresponding to a fast path.

[0296] In one of the embodiments, the third recognition module 2403 is specifically configured to:

[0297] The first key frame image sequence and the second key frame image sequence are subjected to double-path feature extraction through the fast and slow networks, to obtain a static feature vector corresponding to a slow path and a dynamic feature vector corresponding to a fast path.

[0298] In one of the embodiments, the third identification module 2403 is specifically configured to:

[0299] The number of channels of the dynamic feature vector corresponding to the fast path is adjusted, and the adjusted dynamic feature vector is aligned with the static feature vector corresponding to the slow path in terms of channels;

[0300] The fast path is subjected to dimension reduction sampling through a time average pooling layer or a convolution layer in the fast and slow networks, so that the dynamic feature vector of the fast path after the dimension reduction sampling is time-aligned with the static feature vector corresponding to the slow path;

[0301] The dynamic feature vector and the static feature vector that are aligned in terms of channels and time are subjected to feature fusion, to obtain a fused feature vector.

[0302] In one of the embodiments, the third identification module 2403 is configured to:

[0303] The fused feature vector is subjected to pet behavior identification through a full connection layer in the fast and slow networks, to output probability values of various pet behaviors;

[0304] The second behavior identification result is determined according to the probability values of the various pet behaviors.

[0305] In one of the embodiments, the cleaning device 2400 further includes:

[0306] The second determination module is configured to determine a target cleaning strategy based on the target behavior identification result, the target cleaning strategy including one or more of cleaning, avoiding and warning.

[0307] In one of the embodiments, the second determination module is specifically configured to:

[0308] When the target behavior identification result is a feeding behavior, a virtual feeding area is constructed based on a pet position or a position of a feeding article, and the virtual feeding area is labeled as a first cleaning area;

[0309] Other cleaning areas except the first cleaning area are cleaned, and a cleaning progress and an activity state of the pet in the first cleaning area are continuously monitored;

[0310] When the other cleaning areas are cleaned and / or the pet has left the first cleaning area, cleaning of the first cleaning area is performed.

[0311] In one of the embodiments, the second determination module is specifically configured to:

[0312] identify a cleaning area and / or a dirt level of the first cleaning area, and clean the first cleaning area based on the cleaning area and / or the dirt level.

[0313] In one of the embodiments, the second determining module is specifically configured to:

[0314] when the cleaning area of the first cleaning area is greater than the area threshold or the dirt level is greater than the dirt threshold, instruct the cleaning robot to return to the base station to clean the cleaning part of the cleaning robot;

[0315] clean the first cleaning area based on the cleaning part after the cleaning process.

[0316] In one of the embodiments, the second determining module is specifically configured to: dry the cleaning part of the cleaning robot and clean the cleaning part of the cleaning robot.

[0317] In one of the embodiments, the second determining module is specifically configured to:

[0318] before cleaning the first cleaning area, instruct the cleaning robot to return to the base station; the base station is configured to dry the cleaning part of the cleaning robot.

[0319] In one of the embodiments, the cleaning part includes a side brush, a roller brush and a cloth, and the second determining module is specifically configured to:

[0320] control the side brush to be in a non-cleaning position, and control the roller brush in the cleaning position to clean the first cleaning area at a first cleaning speed and the cloth in the cleaning position to clean the first cleaning area at a second cleaning speed.

[0321] In one of the embodiments, when the first cleaning area has solid particles, the first cleaning speed of the roller brush is determined based on the particle density of the solid particles; or the first cleaning speed of the roller brush is inversely proportional to the second cleaning speed of the side brush.

[0322] In one of the embodiments, the cleaning device 2400 further includes:

[0323] The third determining module is configured to determine whether to clean the cleaning part based on the dirt level of the first cleaning area after the first cleaning, and to clean the first cleaning area repeatedly.

[0324] In one of the embodiments, the third determining module is specifically configured to:

[0325] control the side brush to be in a non-cleaning position, and control the roller brush in the cleaning position to clean the first cleaning area at a third cleaning speed and the cloth in the cleaning position to clean the first cleaning area at a first cleaning speed.

[0326] In one of the embodiments, the second determining module is specifically configured to:

[0327] When the cleaning area of the first cleaning area is less than the area threshold and / or the dirtiness is less than the dirtiness threshold, the first cleaning area is cleaned based on the cleaning element of the cleaning robot.

[0328] Based on the dirtiness of the first cleaning area after the first cleaning, it is determined whether to clean the cleaning element and repeat the cleaning of the first cleaning area.

[0329] In one of the embodiments, the cleaning element includes an edge brush, a roller brush, and a cloth, and the second determining module is specifically configured to:

[0330] The edge brush is controlled to be in the non-cleaning position, and the roller brush in the cleaning position and the cloth in the cleaning position are both controlled to clean the first cleaning area at the second cleaning speed.

[0331] In one of the embodiments, the second determining module is specifically configured to:

[0332] When the target behavior recognition result is a sleep behavior, a licking behavior, or a stay behavior, the second cleaning area is calibrated based on the pet position.

[0333] The pet cleaning progress and / or the activity state in the second cleaning area are continuously monitored until the cleaning of the other cleaning areas except the second cleaning area is completed and / or the pet has left the second cleaning area, and the cleaning of the second cleaning area is performed.

[0334] In one of the embodiments, the second determining module is specifically configured to:

[0335] When the target behavior recognition result is an abnormal behavior, an early warning notification message is sent to the user based on the number of occurrences and / or the duration of the abnormal behavior; and / or,

[0336] When the target behavior recognition result is an abnormal behavior, a virtual obstacle avoidance area is constructed based on the pet position, and an obstacle avoidance cleaning mode is executed to avoid the virtual obstacle avoidance area.

[0337] In one of the embodiments, the second determining module is specifically configured to:

[0338] When the number of occurrences and / or the duration of the same abnormal behavior is greater than a preset threshold, an early warning notification message is sent to the user.

[0339] In one of the embodiments, the abnormal behavior corresponds to a reporting priority, and the cleaning device 2400 further includes:

[0340] The sending module is configured to, when the target behavior recognition result includes multiple abnormal behaviors, send an early warning notification message to the user based on the abnormal behavior with the highest reporting priority.

[0341] Obviously, the above-described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments of the present application, those skilled in the art can make other different forms of changes or modifications without creative labor, and all these changes and modifications should belong to the protection scope of the present application.

[0342] The above-described various modules can be implemented by software, hardware, and combinations thereof, in whole or in part. The above-described various modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above-described various modules.

[0343] In an example embodiment, a cleaning robot 1000 is provided, as shown in Figure 25 The cleaning robot can be a sweeping robot, a mopping robot, a sweeping and mopping robot, a window cleaning robot, etc., and can include a body, a walking system, a sensing system, a cleaning element, a processing unit, etc. As shown in Figure 26 The cleaning element of the cleaning robot 1000 includes a side brush 210, a roller brush 220, and a cloth 230. For example, the cloth 230 can be a disc cloth, a vibrating disc cloth, a track-type cloth, a rotary-type cloth, or a roller-type cloth. For example, the rotary-type cloth has a long rectangular projection in the height direction. For example, the track-type cloth has a long hole-shaped longitudinal section in the height direction, and the rotary-type cloth has a circular hole-shaped longitudinal section. The rotary-type cloth has a cavity inside, and an inner support bracket is arranged in the cavity to tension the rotary-type cloth. One side of the inner support bracket in the length direction is rotationally connected to a driving shaft, and the other side of the inner support bracket in the length direction is rotationally connected to a driven shaft. The driving shaft and the driven shaft are parallel to each other and parallel to the width direction of the body, and the rotary-type wet cleaning element is tensioned outside the driving shaft and the driven shaft.

[0344] In an example embodiment, a cleaning robot is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the above-described method embodiments.

[0345] In an example embodiment, a computer readable storage medium is provided, storing a computer program, and the computer program is executed by a processor to implement the steps in the above-described method embodiments.

[0346] In an example embodiment, a computer program product is provided, including a computer program, and the computer program is executed by a processor to implement the steps in the above-described method embodiments.

[0347] The above-described embodiments are merely illustrative for the present application and are described in a relatively specific and detailed manner, but should not be construed as limiting the scope of the present application. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

[0348] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments, wherein the storage medium can be a disc, an optical disc, a read-only memory (Rom), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc. The storage medium can also include a combination of the above-mentioned types of memories.

[0349] In some embodiments, any reference to storage, a database or other medium can include at least one of a non-volatile memory and a volatile memory. The non-volatile memory can include a Read-Only Memory (ROM), a tape, a floppy disk, a flash memory, an optical storage, a high-density embedded non-volatile memory, a Resistive Random Access Memory (ReRAM), a Magnetoresistive Random Access Memory (MRAM), a Ferroelectric Random Access Memory (FRAM), a Phase Change Memory (PCM), a graphene memory, and the like. The volatile memory can include a Random Access Memory (RAM), an external cache memory, and the like. As an illustration and not a limitation, the RAM can be in a variety of forms, such as a Static Random Access Memory (SRAM) or a Dynamic Random Access Memory (DRAM), and the like. The database involved in the embodiments provided herein can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, and the like, without being limited thereto. The processor involved in the embodiments provided herein can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an Artificial Intelligence (AI) processor, and the like, without being limited thereto.

[0350] Any combination of the technical features of the above embodiments can be made, and in order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the range disclosed in the present application.

[0351] The above-described embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be noted that, for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A cleaning method characterized by, The method is applied to a cleaning robot, and the method comprises: acquiring single-frame images and identifying the single-frame images based on a preset deep learning algorithm to obtain initial behavior recognition results; fusing and judging the initial behavior recognition results corresponding to each single-frame image based on a multi-frame image fusion judgment strategy to obtain first behavior recognition results; if the first behavior recognition results are identification abnormities, accumulating the single-frame images and performing time sequence behavior recognition on the accumulated multi-frame images according to a time sequence behavior recognition algorithm to determine second behavior recognition results; determining target behavior recognition results based on the first behavior recognition results and the second behavior recognition results; the target behavior recognition results are used to determine a target cleaning strategy for cleaning processing; the time sequence behavior recognition algorithm comprises: accumulating each single-frame image within a target time length based on a preset time sequence to obtain a multi-frame image sequence; performing spatial feature extraction on the multi-frame image sequence through a time sequence behavior recognition model to obtain feature vectors; fusing each feature vector based on time sequence correlation to obtain a fused feature vector; performing pet behavior recognition on the fused feature vector to determine second behavior recognition results.

2. The method of claim 1, wherein, the acquiring single-frame images and identifying the single-frame images based on a preset deep learning algorithm to obtain initial behavior recognition results comprises: acquiring single-frame pet activity scene images through an image acquisition unit; inputting the single-frame images into a pre-trained deep learning model and performing feature extraction and pet behavior category prediction on the single-frame images based on the deep learning model to output probability values of each pet behavior category; obtaining initial behavior recognition results according to the probability values of each pet behavior category.

3. The method of claim 1, wherein, the fusing and judging the initial behavior recognition results corresponding to each single-frame image based on a multi-frame image fusion judgment strategy to obtain first behavior recognition results comprises: if the initial behavior recognition results corresponding to a plurality of continuous single-frame images are all the same, the initial behavior recognition results are determined as the first behavior recognition results; if the initial behavior recognition results corresponding to a plurality of continuous single-frame images are not all the same, the first behavior recognition results are determined as default values.

4. The method of claim 1, wherein, the first behavior recognition results being identification abnormities comprises: the first behavior recognition results being default values, and / or the first behavior recognition results being abnormal pet behaviors.

5. The method of claim 1, wherein, the accumulating each single-frame image within a target time length based on a preset time sequence to obtain a multi-frame image sequence comprises: accumulating each single-frame image within a target time length based on a preset time sequence to obtain a video segment; inputting the video segment into a fast-slow network, performing double-path frame sampling through the fast-slow network to obtain a first key frame image sequence corresponding to a slow path and a second key frame image sequence corresponding to a fast path.

6. The method of claim 5, wherein, the performing spatial feature extraction on the multi-frame image sequence through a time sequence behavior recognition model to obtain feature vectors comprises: The slow-fast network is used for double-path feature extraction on the first key frame image sequence and the second key frame image sequence, to obtain a static feature vector corresponding to the slow path and a dynamic feature vector corresponding to the fast path.

7. The method of claim 5, wherein, The feature vectors are fused based on time sequence correlation to obtain a fused feature vector, including: The number of channels of the dynamic feature vector corresponding to the fast path is adjusted, and the adjusted dynamic feature vector is aligned with the static feature vector corresponding to the slow path in terms of channels; The fast path is dimensionally sampled through a time average pooling layer or a convolution layer in the slow-fast network, so that the dynamic feature vector of the dimensionally sampled fast path is time-aligned with the static feature vector corresponding to the slow path; The dynamic feature vector and the static feature vector that are aligned in terms of channels and time are fused to obtain a fused feature vector.

8. The method of claim 5, wherein, The fused feature vector is used for pet behavior recognition to determine a second behavior recognition result, including: The fused feature vector is used for pet behavior recognition through a fully connected layer in the slow-fast network, to output probability values of various pet behaviors; The second behavior recognition result is determined according to the probability values of the various pet behaviors.

9. The method of claim 1, wherein, The method further includes: A target cleaning strategy is determined based on the target behavior recognition result, and the target cleaning strategy includes one or more of cleaning, avoiding, and warning.

10. The method of claim 9, wherein, The target cleaning strategy is determined based on the target behavior recognition result, including: When the target behavior recognition result is a feeding behavior, a virtual feeding area is constructed based on a pet position or a position of a feeding item, and the virtual feeding area is labeled as a first cleaning area; Cleaning is performed on other cleaning areas except the first cleaning area, and cleaning progress and an activity state of the pet in the first cleaning area are continuously monitored; When the other cleaning areas are cleaned and / or the pet has left the first cleaning area, cleaning of the first cleaning area is performed.

11. The method of claim 10, wherein, The cleaning of the first cleaning area is performed, including: A cleaning area and / or a dirtiness degree of the first cleaning area are identified, and the first cleaning area is cleaned based on the cleaning area and / or the dirtiness degree.

12. The method of claim 11, wherein, The cleaning area and / or the dirtiness degree of the first cleaning area are identified, and the first cleaning area is cleaned based on the cleaning area and / or the dirtiness degree, including: When the cleaning area of the first cleaning area is greater than an area threshold or the dirtiness degree is greater than a dirtiness threshold, the cleaning robot is instructed to return to a base station to clean cleaning components of the cleaning robot; The first cleaning area is cleaned based on the cleaning components after cleaning processing.

13. The method of claim 12, wherein, The cleaning processing of the cleaning components of the cleaning robot includes drying processing of the cleaning components of the cleaning robot and cleaning processing of the cleaning components of the cleaning robot.

14. The method of claim 13, wherein, The cleaning robot is instructed to return to the base station to clean the cleaning components of the cleaning robot, including: Before the first cleaning area is cleaned, the cleaning robot is instructed to return to a base station; the base station is used to dry the cleaning components of the cleaning robot.

15. The method of claim 12, wherein, The cleaning components include an edge brush, a roller brush, and a cloth, and the cleaning of the first cleaning area based on the cleaning treatment of the cleaning components includes: The edge brush is controlled to be in a non-cleaning position, and the roller brush in a cleaning position is controlled to clean the first cleaning area at a first cleaning speed, and the cloth in a cleaning position is controlled to clean the first cleaning area at a second cleaning speed.

16. The method of claim 15, wherein, When the first cleaning area has solid particles, the first cleaning speed of the roller brush is determined based on the particle density of the solid particles; or the first cleaning speed of the roller brush is inversely proportional to the second cleaning speed of the edge brush.

17. The method of claim 12, wherein, After the cleaning of the first cleaning area based on the cleaning treatment of the cleaning components, the method further includes: Based on the dirtiness of the first cleaning area after the first cleaning, it is determined whether to clean the cleaning components and repeatedly clean the first cleaning area.

18. The method of claim 17, wherein, The repeatedly cleaning of the first cleaning area includes: The edge brush is controlled to be in a non-cleaning position, and the roller brush in a cleaning position is controlled to clean the first cleaning area at a third cleaning speed, and the cloth in a cleaning position is controlled to clean the first cleaning area at a first cleaning speed.

19. The method of claim 17, wherein, The number of times of repeatedly cleaning the first cleaning area is proportional to the drying degree of the cloth.

20. The method of claim 11, wherein, The identification of the cleaning area and / or the dirtiness of the first cleaning area, based on the cleaning area and / or the dirtiness, includes: When the cleaning area of the first cleaning area is less than an area threshold and / or the dirtiness is less than a dirtiness threshold, the cleaning treatment of the first cleaning area based on the cleaning components of the cleaning robot is performed; Based on the dirtiness of the first cleaning area after the first cleaning, it is determined whether to clean the cleaning components and repeatedly clean the first cleaning area.

21. The method of claim 20, wherein, The cleaning components include an edge brush, a roller brush, and a cloth, and the cleaning of the first cleaning area based on the cleaning treatment of the cleaning components includes: The edge brush is controlled to be in a non-cleaning position, and the roller brush in a cleaning position is controlled to clean the first cleaning area at a third cleaning speed, and the cloth in a cleaning position is controlled to clean the first cleaning area at a first cleaning speed.

22. The method of claim 9, wherein, The determination of the target cleaning strategy based on the target behavior recognition result includes: When the target behavior recognition result is a sleep behavior, a licking behavior, or a stay behavior, a second cleaning area is calibrated based on the pet position; The pet cleaning progress and / or the activity state in the second cleaning area are continuously monitored until the cleaning of other cleaning areas except the second cleaning area is completed and / or the pet has left the second cleaning area, and the cleaning of the second cleaning area is performed.

23. The method of claim 9, wherein, The determination of the target cleaning strategy based on the target behavior recognition result includes: When the target behavior recognition result is an abnormal behavior, a warning notification message is sent to the user based on the number of occurrences and / or the duration of the abnormal behavior; and / or, When the target behavior recognition result is an abnormal behavior, a virtual obstacle avoidance area is constructed based on a pet position, an obstacle avoidance cleaning mode is executed, and the virtual obstacle avoidance area is avoided.

24. The method of claim 23, wherein, The method further includes: When the number of occurrences and / or the duration of the same abnormal behavior is greater than a preset threshold, a warning notification message is sent to the user.

25. The method of claim 23, wherein, The abnormal behavior corresponds to a reporting priority, and the method further includes: When the target behavior recognition result includes multiple abnormal behaviors, a warning notification message is sent to the user based on the abnormal behavior with the highest reporting priority.

26. A cleaning robot comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1-25.

27. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1-25.

Images