Target following cleaning method for sweeping robot based on visual tracking and radar ranging

Abstract

The application relates to the technical field of sweeping robots, in particular to a sweeping robot target following cleaning method based on visual tracking and radar ranging. The method acquires image data and ranging data of a cleaning area in real time through a camera unit and a ranging unit, and obtains target object and to-be-cleaned object information after fusion. The moving track of the target object is simulated according to the motion characteristics and a trajectory prediction model of the target object. The target sweeping robot and the cleaning mode are determined in combination of the to-be-cleaned object information, the cleaning capacity and the position information of the sweeping robot. The path information and the driving information are acquired based on the moving track of the target object and the position information of the sweeping robot, so as to guide the sweeping robot to reach the to-be-cleaned area for cleaning. The application solves the problems of inaccurate target object and to-be-cleaned object identification and low cleaning efficiency in the prior art, and improves the identification precision and the cleaning efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sweeping robots, in particular to a sweeping robot target following cleaning method based on visual tracking and radar ranging. BACKGROUND

[0002] With the popularization of intelligent cleaning equipment, sweeping robots are increasingly widely used in home and commercial scenarios. However, the existing technology still faces many challenges in actual application. Similar existing technologies include Chinese Patent No. CN111240309B, which proposes a method and device for sweeping robot to perform cleaning work and electronic equipment. The method includes: determining a reference block for performing cleaning work based on current environmental information, and then performing cleaning work based on the reference block to control the sweeping robot to perform cleaning operation. In addition, similar existing technologies include European Patent No. EP2801313B1, which provides a cleaning robot and a control method thereof. The cleaning robot can have a main body, a moving unit arranged on the main body to move the main body in a cleaning space, a cleaning unit arranged on the main body to clean the floor of the cleaning space, an image acquisition unit configured to acquire a floor image of the floor, an image of the cleaning space, and a control unit configured to determine whether there is a foreign object on the floor of the cleaning space based on the floor image, and control the moving unit to move the main body to the position of the foreign object. The cleaning robot detects foreign objects not located on the moving track of the cleaning robot by acquiring an image of the floor to be cleaned, and moves to the foreign object position for detection and cleaning when a foreign object is detected. The above two patent documents solve the problem of cleaning by sweeping robots, but the recognition accuracy of the cleaning object and the dynamic target object is not enough, and when multiple sweeping robots are needed to clean together, the full use of cleaning force and efficiency is not enough. SUMMARY

[0003] The present application provides a sweeping robot target following cleaning method based on visual tracking and radar ranging. The patent significantly improves the recognition accuracy and cleaning efficiency of the target object of the sweeping robot in complex environments by fusing visual tracking and radar ranging technology, combining dynamic target recognition and trajectory prediction. The method comprises:

[0004] The first data and the second data of the cleaning area are acquired by the camera unit and the ranging unit of each sweeping robot in real time, and are sent to the detection unit;

[0005] The detection unit receives and acquires a plurality of target objects and cleaning objects according to the first data and the second data, fuses the first data and the second data, and acquires a plurality of target object information and cleaning information in the cleaning area;

[0006] According to each target object information, a motion feature and a correlation between the target object are extracted, and a moving track of each target object is obtained based on the motion feature, the correlation and a track prediction model;

[0007] According to the to-be-cleaned information, a cleaning ability and position information of each robot, a target robot and an optimal cleaning mode are determined;

[0008] Based on the moving track of each target object and the position information of the target robot, path information and corresponding travel information of each target robot are obtained;

[0009] Each target robot reaches the to-be-cleaned area and performs cleaning according to the corresponding path information and travel information.

[0010] As a preferred technical solution of the present application, a plurality of target objects and to-be-cleaned objects are obtained, comprising:

[0011] The detection unit inputs the first data into a first recognition model to obtain a first category and a first category probability of a first object in the first data, and inputs the second data into a second recognition model to obtain a second category and a second category probability of a second object in the second data;

[0012] The distance between the first object and the second object is calculated, and when the distance is less than or equal to a set distance, the first object and the second object are the same object, and the same object is taken as the target object, otherwise, they are not the same object;

[0013] When the first category and the second category corresponding to the target object are the same and the first category probability and the second category probability are both greater than or equal to a first threshold, the first category is taken as the category of the target object; when one of the first category probability and the second category probability is greater than the first threshold and the other is less than or equal to a second threshold, the first data and the second data are re-obtained, and the step is repeated; when the first category probability and the second category probability are both less than or equal to the second threshold, the target object is a to-be-cleaned object;

[0014] When they are not the same object, when the first category probability or the second category is greater than or equal to the first threshold, the first object or the second object is taken as the target object, otherwise, the first object or the second object is taken as the to-be-cleaned object.

[0015] As a preferred technical solution of the present application, a moving track of each target object is obtained, comprising:

[0016] The motion feature of the target object is acquired through each target object information, the correlation between the first motion feature of the target object and the second motion feature of other target objects in a preset range around the target object is analyzed, the motion feature of the target object is input into the trajectory prediction model to acquire the moving trajectory of the target object, and the moving trajectory of the other target objects is acquired based on the moving trajectory of the target object and the correlation, wherein the motion feature includes shape, motion speed, motion posture and motion trajectory.

[0017] As a preferred technical solution of the present application, the target sweeping robot and the optimal cleaning mode are determined, comprising:

[0018] According to the position information of each sweeping robot and the position information of the object to be cleaned, the cleaning information is sent to each idle sweeping robot within a preset range from the object to be cleaned, the first cleaning information and the optimal cleaning mode predicted by the sweeping robot are received, and the sweeping robot corresponding to the greatest difference between the first cleaning information and the cleaning information is taken as the first target sweeping robot, wherein the difference is the cleaning ability of the sweeping robot.

[0019] The first cleaning information corresponding to the first target robot is sent to other sweeping robots within the preset range, the second cleaning information and the optimal cleaning mode predicted by each other sweeping robot are received, and the other sweeping robot corresponding to the second cleaning information satisfying the cleaning standard and also satisfying the smallest difference between the first cleaning information and the cleaning information is taken as the second target sweeping robot, and the target sweeping robot includes the first target sweeping robot and the second target sweeping robot.

[0020] As a preferred technical solution of the present application, the path information and the travel information of each target sweeping robot are acquired, comprising:

[0021] The connection between the position information of the target sweeping robot and the position information of the object to be cleaned is acquired, the path segment and the time segment corresponding to any target object and having a distance greater than a set value between the moving trajectory are acquired one by one within a preset width range on both sides of the connection, and the path information and the travel information of the target sweeping robot are acquired based on the speed range, the position information of the target sweeping robot and the path segment and the time segment corresponding to the path segment.

[0022] As a preferred technical solution of the present application, when the target sweeping robot changes the association relationship between the target object and other target objects in the process of reaching the to-be-cleaned position or in the cleaning process, the moving track of the target object and the moving track of the other target objects are updated in real time, and new path information of the target sweeping robot is obtained according to the updated moving track.

[0023] As a preferred technical solution of the present application, when the number of target sweeping robots is greater than or equal to 2, the target sweeping robot with stronger cleaning ability has a higher cleaning sequence.

[0024] As a preferred technical solution of the present application, the target object information includes the category, image and position information of the target object, and the to-be-cleaned information at least includes the category, shape, image, area, volume and position information of the to-be-cleaned object.

[0025] The present application also provides a sweeping robot target following cleaning system based on visual tracking and radar ranging, which is used to execute the above method, and comprises:

[0026] The sweeping robot comprises a camera unit, a ranging unit and a communication unit, wherein the camera unit is used to acquire first data of a cleaning area in real time, the ranging unit is used to acquire second data of the cleaning area in real time, and the communication unit is used to send the first data and the second data to a detection unit.

[0027] The detection unit is used to receive and acquire a plurality of target objects and to-be-cleaned objects according to the first data and the second data, fuse the first data and the second data, and acquire target object information and to-be-cleaned information in the cleaning area.

[0028] The prediction unit is used to extract motion features according to the target object information, and predict the moving track of each target object based on the motion features and a track prediction model.

[0029] The determination unit is used to determine a target sweeping robot and a cleaning mode according to the to-be-cleaned information, the cleaning ability and the position information of each sweeping robot.

[0030] The calculation unit is used to acquire path information and travel information of each target sweeping robot based on the moving track of the object and the position information of the target sweeping robot.

[0031] The sweeping robot is also used to reach the to-be-cleaned area according to the corresponding path information and travel information, and clean based on the optimal cleaning mode.

[0032] The application further provides a computer readable storage medium, wherein instructions are stored on the computer readable storage medium, and the instructions are executed by a processor to implement the method.

[0033] Effects

[0034] The application significantly improves the cleaning efficiency and intelligent level of the sweeping robot in a complex environment by fusing visual tracking and radar ranging technology, combining dynamic target identification and trajectory prediction, through synchronous acquisition and fusion of visual and radar data, combining a first identification model (based on image data) and a second identification model (based on radar data), a double verification mechanism is realized, when the identification results of the two are consistent within a distance threshold, the category of the target object is confirmed, if there is a probability difference or data conflict, it is reacquired or classified as a to-be-cleaned object, this multi-modal data fusion method effectively reduces the misjudgment rate, ensures the accurate distinction between dynamic targets (such as pedestrians, pets) and to-be-cleaned objects (such as garbage, stains), and lays a reliable foundation for subsequent path planning.In a dynamic environment, the movement of target objects such as pedestrians walking or pets running directly affects the safety of the cleaning path. By extracting the motion characteristics (speed, direction, posture) of the target objects and analyzing their association with surrounding objects (such as the interaction between people and carried objects), a trajectory prediction model is used to generate future movement trajectories. At the same time, the system monitors the dynamic changes between target objects in real time (such as the trajectory deviation caused by the change in the length of the pet leash), and adjusts the path of the robot accordingly. For example, when planning the path, the system will reserve a safety distance around the moving trajectory of the target object, and select a collision-free path segment based on the time window. This dynamic obstacle avoidance mechanism not only reduces the risk of collision, but also improves the continuity and efficiency of the cleaning process. For large areas or high dynamic scenes, a multi-robot cooperative working strategy is proposed. The system evaluates the best cleaning mode (such as strong mode for stubborn stains) based on the position, area and shape of the object to be cleaned, combined with the real-time position and cleaning ability (such as suction strength, wiping function) of each robot. Robots with stronger cleaning ability are preferentially assigned to high difficulty areas, while other robots are assigned as needed. In addition, the system adjusts the working range of the robot in real time through a dynamic task allocation algorithm to avoid repeated cleaning or coverage blind spots. This intelligent collaboration mechanism significantly improves the overall cleaning efficiency, especially for complex public scenes such as shopping malls and airports. The system can automatically match the cleaning mode according to the specific characteristics of the object to be cleaned (such as stain type, distribution density), for example, for oil-stained areas, a strong scrubbing mode is started, while ordinary dust areas use an energy-saving mode. At the same time, the path planning module selects the optimal travel route and time period based on the speed range and energy consumption characteristics of the robot, reducing unnecessary movement and energy waste. This adaptive strategy not only prolongs the battery life of the robot, but also reduces hardware wear and tear through precise cleaning. During the cleaning process, the system continuously monitors environmental changes (such as the addition of obstacles or sudden changes in the movement trajectory of target objects) and updates path information in real time. For example, when detecting that a certain area is blocked due to the gathering of pedestrians, the system immediately re-plans a detour route to ensure uninterrupted cleaning tasks. In addition, through a redundant data verification mechanism (such as multiple collection and verification of low probability recognition results), the system can still operate stably in the face of sensor noise or temporary interference, improving the overall robustness. Through the cooperation of the above technical solutions, the robot achieves efficient, safe and intelligent cleaning in a dynamic environment, solving the problems of inaccurate target recognition, rigid path planning and low efficiency of multi-machine cooperation in traditional methods. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0036] Figure 1 The flow chart of the target following cleaning method of the sweeping robot based on visual tracking and radar ranging of the present application;

[0037] Figure 2 The flow chart of the target object and object to be cleaned detection method in the embodiment of the present application;

[0038] Figure 3 The flow chart of the target sweeping robot and optimal cleaning mode determination method in the embodiment of the present application;

[0039] Figure 4 The structural diagram of the target following cleaning system of the sweeping robot based on visual tracking and radar ranging of the present application. DETAILED DESCRIPTION

[0040] The terms "first", "second", "third", "fourth" and the like (if any) in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the term "comprise" or "have" and any variation thereof is intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0041] For the convenience of understanding, the specific process of the embodiments of the present application will be described as follows, as shown in the figure, the target following cleaning method of the sweeping robot based on visual tracking and radar ranging in the embodiments of the present application comprises: Figure 1

[0042] Step S1: Real-time acquisition of first data and second data of the cleaning area by the camera unit and the ranging unit of each sweeping robot respectively, and sending to the detection unit;

[0043] ​Specifically, in a large area public area, multiple sweeping robots are needed to jointly complete the cleaning work. In order to improve the efficiency of the cleaning work and not to affect the walking of pedestrians, each sweeping robot respectively acquires first data, i.e. image data, and second data, i.e. radar data, of the cleaning area in real time through its camera unit and ranging unit. These data are sent to the detection unit through the wireless communication module. The detection unit processes and fuses the first data and the second data. The above technical solution provides a basis for accurately classifying and detecting target objects and objects to be cleaned in the cleaning area.

[0044] Step S2: The detection unit receives and acquires multiple target objects and objects to be cleaned according to the first data and the second data, fuses the first data and the second data, and acquires multiple target object information and cleaning information in the cleaning area;

[0045] Specifically, after the detection unit receives the image data and radar data sent by each sweeping robot, the image data is input into an image recognition model, i.e. a first recognition model, to identify multiple objects and give a first category of each object, i.e. a first category of pedestrians, carried objects, pets, etc. and a first category probability. The radar data can not only acquire the distance between the corresponding sweeping robot and other objects, but also detect the change in the reflection signal frequency of the object through the Doppler effect. By analyzing this change, the motion characteristics and geometric characteristics of the object can be acquired. Then, a radar data recognition model, i.e. a second recognition model, is used to identify a second category of the second object in the radar data and a second category probability. Through data fusion, it is determined whether the second object and the first object are the same object according to the position information of the first object and the second object. Then, according to the first category and the first category probability corresponding to the first object and the second category and the second category probability corresponding to the second object, the multiple target objects and objects to be cleaned in the cleaning area are determined. Through a data fusion algorithm, the two groups of data are fused to acquire each target object information, including the category, image information and position information of the corresponding target object, and to acquire cleaning information, including the shape, distribution area, volume and position information of the object to be cleaned. Through the above technical solution and the cooperation of step S1, accurate target object information and cleaning object information can be acquired, which lays a foundation for accurately predicting the moving track of the target object and acquiring an accurate target sweeping robot.

[0046] Step S3: According to each target object information, the motion characteristics and the correlation between the target objects are extracted, and based on the motion characteristics, the correlation and a trajectory prediction model, the moving track of each target object is acquired.

[0047] Specifically, after obtaining the target object information, the motion characteristics of the target object are extracted, including at least motion speed, motion direction, motion posture, and motion trajectory. These motion characteristics are calculated through the position changes of the target object at consecutive time points. For example, by analyzing the position changes of the target object at multiple time points, the motion speed and motion direction of the target object can be calculated. The motion posture of the target object can be obtained through the image information of the target object. These motion characteristics are input into a trajectory prediction model. Based on the historical motion data and current motion characteristics of the target object, the trajectory prediction model predicts the moving trajectory of the target object in the future. The trajectory prediction model can be a prediction model based on machine learning, or a model based on physical motion laws. In addition, the moving trajectory of the target object can be used to obtain the moving trajectory of other target objects that have a correlation with it. Through the above technical solutions, the moving trajectory of each target object can be obtained, providing an important basis for the path information of the sweeping robot.

[0048] Step S4: determining the target sweeping robot and the best cleaning mode according to the to-be-cleaned information, the cleaning ability and position information of each sweeping robot;

[0049] Specifically, according to the to-be-cleaned information and the cleaning ability and position information of each sweeping robot, including at least the category, shape, image, area, volume, and position information of the to-be-cleaned object, the most suitable sweeping robot and cleaning mode are determined. The to-be-cleaned information is sent to the idle sweeping robots within a preset range from the to-be-cleaned object. Each sweeping robot predicts its cleaning effect and the best cleaning mode according to its own cleaning ability and position information. The determination unit receives these prediction information and selects the sweeping robot combination with the best cleaning effect as the target sweeping robot. At the same time, the best cleaning mode is determined according to the characteristics and cleaning requirements of the to-be-cleaned object, such as the powerful cleaning mode and the ordinary cleaning mode. Through the above technical solutions, the efficient completion of the cleaning task is ensured, and the waste of cleaning resources is avoided.

[0050] Step S5: obtaining the path information and corresponding driving information of each target sweeping robot based on the moving trajectory of each target object and the position information of the target sweeping robot;

[0051] Specifically, the shortest path of the target sweeping robot from the current position to the position of the object to be cleaned is calculated based on the position information of the target sweeping robot, and within the preset width range on both sides of the path, the path segments and time segments with a distance greater than a set value from the moving track of the target object are obtained, which ensures that the sweeping robot will not collide with the target object during movement. At the same time, the travel information of the target sweeping robot is generated based on the speed range and position information of the target sweeping robot, including the travel speed, travel direction, etc. These path information and travel information provide an important basis for the navigation and control of the sweeping robot, ensuring that it can smoothly reach the cleaning area and clean it.

[0052] Step S6: Each of the target sweeping robots reaches the cleaning area according to the corresponding path information and travel information and cleans based on the optimal cleaning mode.

[0053] Specifically, after receiving the path information and travel information, the target sweeping robot starts navigation and control according to these information. The sweeping robot adjusts its travel direction and speed according to the path information to ensure that it can smoothly reach the cleaning area along the predetermined path. During travel, the sweeping robot monitors the surrounding environment in real time through its sensors to ensure the accuracy and safety of the path information. When the sweeping robot reaches the cleaning area, it cleans according to the optimal cleaning mode, for example, if the cleaning mode is the powerful cleaning mode, the sweeping robot will start powerful suction, wiping and other functions to ensure the cleaning effect of the object to be cleaned. Through the above technical solutions, not only the cleaning efficiency is improved, but also the cleaning quality is ensured.

[0054] Further, a plurality of target objects and objects to be cleaned are obtained, such as Figure 2 As shown, it includes:

[0055] The control unit inputs the first data into a first recognition model to obtain the first category and first category probability of the first object in the first data, and also inputs the second data into a second recognition model to obtain the second category and second category probability of the second object in the second data;

[0056] The distance between the first object and the second object is calculated. When the distance is less than or equal to a set distance, the first object and the second object are the same object, and the same object is taken as the target object, otherwise, they are not the same object;

[0057] In the case that the first category and the second category corresponding to the target object are the same and the first category probability and the second category probability are both greater than or equal to the first threshold, the first category is taken as the category of the target object; in the case that one of the first category probability and the second category probability is greater than the first threshold and the other is less than or equal to the second threshold, the first data and the second data are re-acquired and the step is repeated; in the case that the first category probability and the second category probability are both less than or equal to the second threshold, the target object is a to-be-cleaned object.

[0058] In the case that the first object and the second object are not the same object, in the case that the first category probability or the second category probability is greater than or equal to the first threshold, the first object or the second object is taken as the target object, otherwise, the first object or the second object is taken as the to-be-cleaned object.

[0059] Specifically, since the objects in the cleaning area are in dynamic motion, the first data, i.e., image data, generated by the camera unit can have motion blur or blurring, causing inaccurate object recognition. In addition, the ranging unit, i.e., the radar, has low resolution when identifying target objects and can also encounter multipath effects, causing inaccurate target object recognition. For the above reasons, using the camera unit or the ranging unit alone cannot accurately identify the target objects in the cleaning area, let alone obtain the motion trajectory of each target object. Therefore, the first data, i.e., image data, is input into the first recognition model to obtain the first category and first category probability of the first object. The first recognition model is a prediction model trained on sample image data corresponding to different objects. The first category of the first object includes people, pets, luggage, or other objects. The first category probability is the probability that the object in the image data is the first object. The second data, i.e., radar data, is input into the second recognition model to obtain the second object in the radar data and the second category and second category probability corresponding to the second object. The content of the second category is the same as that of the first category. The second recognition model is a prediction model trained on sample radar data corresponding to different objects. The distance between the first object and the second object is used to determine whether they are the same target object. Since the camera unit is a depth camera device, the position information of the first object can be obtained from the first data, and the distance between the first object and the second object can be calculated. The collection of the first data and the second data is synchronized. When the distance is less than or equal to the set distance, the first object and the second object are the same object, i.e., the target object. The set distance is less than or equal to 0.3m, otherwise, not the same object, further determine the category of the target object, if the first category of the first object and the second category of the second object are the same, and the first category probability and the second category probability are both greater than or equal to the first threshold (for example, 80%), the first category is taken as the category of the target object, for example, if the camera and the radar both identify the pet, and the probability is greater than 80%, it is confirmed that the object is a pet, if only one of the first category probability and the second category probability is greater than the first threshold, and the other is less than or equal to the second threshold, for example, 50%, the first data and the second data are reacquired through step S1, and the step is repeated, for example, if the camera identifies the probability of "person" as 85%, but the radar identifies the probability of "person" as 45%, the data is reacquired for judgment, if the first category probability and the second category probability are both less than or equal to the second threshold, that is, the similarity of the objects in the first category and the second category is relatively low, whether the category of the first object and the second object is the same or not, it is considered that the object is a to-be-cleaned object, wherein the objects in the first category and the second category are preset objects, that is, non-to-be-cleaned objects, for example, if the probability identified by the camera and the radar is less than 50%, it is considered that the object is a to-be-cleaned object; when the first object and the second object are not the same object, that is, only one of the image data or the radar data shows the object, for example: transparent object, the image data cannot be monitored, and also for example: when the ranging unit cannot detect the position of the object, the camera unit can be detected, when one of the camera unit or the ranging unit detects the first object or the second object, and when the corresponding first category probability or second category probability is greater than the first threshold, that is, the category of the first object or the second object can be identified, at this time, the first object or the second object is also taken as the target object, otherwise, when the first category probability or the second category probability corresponding to the first object or the second object is less than the first threshold, it is considered that the first object or the second object is a to-be-cleaned object, through the technical solution, the target object and the to-be-cleaned object in the to-be-cleaned area can be accurately identified, which lays a foundation for further acquiring the moving track of the target object and acquiring the to-be-cleaned object information.

[0060] Further, the moving track of each target object is acquired, comprising:

[0061] The motion feature of the target object is acquired through each target object information, the correlation between the first motion feature of the target object and the second motion feature of other target objects within a preset range is analyzed, the motion feature of the target object is input into the trajectory prediction model to acquire the moving trajectory of the target object, and the moving trajectory of the other target objects is acquired based on the moving trajectory of the target object and the correlation.

[0062] Specifically, the target object information includes the category, image information and position information of the target object, the motion feature of the target object is acquired through the real-time image information and position information in the target object information, the motion feature includes the shape, motion speed, motion posture and motion trajectory of the target object, the shape and motion posture of the target object can be acquired through the category and image of the target object, the motion speed and motion trajectory can be acquired through the position information of the target object changing over time, the correlation between the motion features of the target object and other objects is analyzed, for example, the target object is a person and the other object is a pet or a carrying object, the moving trajectory of the target object is acquired by inputting the category and motion feature of the target object into the trajectory prediction model, the trajectory prediction model is a prediction model trained by the historical moving trajectory of the target object, and the moving trajectory of the other target object is acquired based on the correlation between the target object and the corresponding other target object. Through the technical solution, the moving trajectory of each target object can be accurately predicted, and the path information of the target sweeping robot is laid as a foundation.

[0063] Further, the target sweeping robot and the best cleaning mode are determined, such as Figure 3 as shown, comprising:

[0064] According to the position information of each sweeping robot and the position information of the object to be cleaned, the object to be cleaned information is sent to each idle sweeping robot within a preset range of the object to be cleaned, the first cleaning information and the best cleaning mode predicted by the sweeping robot are received, and the sweeping robot with the largest difference between the first cleaning information and the object to be cleaned information is taken as the first target sweeping robot, wherein the difference is the cleaning ability of the sweeping robot.

[0065] The first cleaning information corresponding to the first target robot is sent to other sweeping robots within the preset range, and the second cleaning information and the optimal cleaning mode predicted by each of the other sweeping robots are received, and the other sweeping robot corresponding to the second cleaning information that meets the cleaning standard and also meets the first cleaning information and the minimum cleaning information to be cleaned is taken as a second target sweeping robot, and the target sweeping robot includes the first target sweeping robot and the second target sweeping robot.

[0066] Specifically, after detecting the object to be cleaned, the position information of the object to be cleaned is obtained according to the object to be cleaned information corresponding to the object to be cleaned, wherein the object to be cleaned information includes the shape, distribution area, volume and position information of the object to be cleaned, and the position information of the object to be cleaned is also obtained through the second data corresponding to each of the objects to be cleaned. Similarly, the position information of each sweeping robot is calculated according to the reference position of at least two reference objects in the cleaning area corresponding to the second data obtained by each sweeping robot. Since the cleaning ability of the sweeping robot is not completely the same with different working time, the cleaning information is sent to each idle sweeping robot within a preset range from the object to be cleaned. After each sweeping robot receives the cleaning information, the optimal cleaning effect for the cleaning information, i.e. the first cleaning information and the corresponding optimal cleaning mode, is obtained through the internal cleaning prediction model. The greater the difference between the first cleaning information and the cleaning information, the stronger the cleaning ability of the corresponding sweeping robot, and vice versa. The sweeping robot with the strongest cleaning ability is taken as the first target sweeping robot. The first cleaning information output by the first target sweeping robot is sent to other sweeping robots within the preset range, and the second cleaning information predicted by the other sweeping robots is received through the detection unit. The other sweeping robot that meets the cleaning standard and has the minimum difference between the first cleaning information and the cleaning information to be cleaned is taken as the second target sweeping robot. The first target sweeping robot and the second target sweeping robot are taken as the target sweeping robot. Through the technical solution, the target sweeping robot with cleaning ability meeting the requirements can be obtained, so that the cleaning ability of the robot is fully utilized and the waste of the cleaning ability of the sweeping robot is reduced.

[0067] Further, the path information and driving information of each target sweeping robot are obtained, including:

[0068] According to the position information of the target robot and the position information of the object to be cleaned, a line is obtained between the two, and within a preset width range on both sides of the line, path segments and time segments corresponding to any target object and having a distance greater than a set value between moving tracks are obtained one by one, and path information and travel information of the target robot are obtained based on a speed range of the target robot, position information, and the path segments and time segments corresponding to the path segments.

[0069] Specifically, by the position information of the target robot and the position information of the object to be cleaned, the positions of the target robot and the object to be cleaned are connected to obtain a nearest path, i.e., the line, and to ensure that the target robot avoids all target objects during reaching the position to be cleaned, path segments and time segments corresponding to any target object and having a distance greater than a set value between moving tracks are obtained one by one within a preset width range on both sides of the line, for example, the set value is 0.6 meters, and the path segments through which the target robot can pass at a set speed within a corresponding time segment are obtained according to the path segments, corresponding time segments, and a speed range of the target robot, wherein the set speed is within the speed range of the target robot, and all the path segments through which the target robot can pass from the current position of the target robot to the position of the object to be cleaned are taken as the path information, and the set speed and time segment corresponding to each path segment are taken as the travel information of the target robot. By the technical solution, not only the path information of the target robot can be obtained, but also the travel information of each path segment in the path information can be obtained, thereby improving the cleaning efficiency.

[0070] Further, when the association relationship between the target object and other target objects changes during the target robot reaching the position to be cleaned or during cleaning, the moving tracks corresponding to the target object and other target objects are updated in real time, and new path information of the target robot is obtained according to the updated moving tracks.

[0071] Specifically, since the moving track of the other target object associated with the target object is closely related to the target object, once the association between the target object and the other target object changes, even if the moving track of the target object does not change, the moving track of the other target object will change, for example, the target object is a person, and the other target object is a pet, when the length of the pet's leash changes, the moving track of the pet will also change, therefore, when the target robot reaches the to-be-cleaned position or during cleaning, it is necessary to detect the change of the association between the target object and the corresponding other target object in real time, and update the path information of the target robot in time when the association changes, so as to prevent the target robot from colliding with the target object or the other target object, thereby reducing the cleaning efficiency.

[0072] Further, when the number of target robots is greater than or equal to 2, the target robot with stronger cleaning ability is arranged in the front of the cleaning sequence.

[0073] Specifically, when multiple target robots work cooperatively, in order to efficiently complete the cleaning task according to the respective cleaning ability and position information, the robot with stronger cleaning ability is arranged in the front of the cleaning sequence to ensure the cleaning efficiency and quality, which not only improves the cleaning efficiency, but also ensures the cleaning quality, avoiding the problem of incomplete cleaning of the robot with weaker cleaning ability in the area with larger cleaning difficulty.

[0074] Further, the target object information includes the category, image and position information of the target object, and the to-be-cleaned information at least includes the category, shape, image, area, volume and position information of the to-be-cleaned object.

[0075] The application also provides a target following cleaning system of a sweeping robot based on visual tracking and radar ranging, which is used to execute the above method, as shown in the accompanying drawings, the system comprises: Figure 4 as shown, the system comprises:

[0076] The sweeping robot comprises a camera unit, a ranging unit and a communication unit, wherein the camera unit is used to acquire first data of a cleaning area in real time, the ranging unit is used to acquire second data of the cleaning area in real time, and the communication unit is used to send the first data and the second data to a detection unit.

[0077] The detection unit is used to receive and acquire multiple target objects and to-be-cleaned objects according to the first data and the second data, fuse the first data and the second data, and acquire multiple target object information and to-be-cleaned information in the cleaning area.

[0078] a prediction unit configured to extract a motion feature according to the target object information, and predict a moving track of each target object based on the motion feature and a track prediction model;

[0079] a determination unit configured to determine a target cleaning robot and a cleaning mode according to the to-be-cleaned information, a cleaning capability and position information of each cleaning robot;

[0080] a calculation unit configured to obtain path information and travel information of each target cleaning robot based on the moving track of the object and the position information of the target cleaning robot;

[0081] The cleaning robot is further configured to reach the to-be-cleaned area according to the corresponding path information and travel information, and clean based on the optimal cleaning mode.

[0082] Further, the present application also provides a computer readable storage medium, wherein instructions are stored on the computer readable storage medium, and the instructions are executed by a processor to implement the above method.

[0083] In summary, the present application significantly improves the cleaning efficiency and intelligent level of the sweeping robot in complex environment by fusing visual tracking and radar ranging technology, combining dynamic target recognition and trajectory prediction. The present application realizes a double verification mechanism by synchronous acquisition and fusion of visual and radar data, combining the first identification model and the second identification model. When the identification results of the two are consistent within the distance threshold, the category of the target object is confirmed. If there is a probability difference or data conflict, it is reacquired or classified as a to-be-cleaned object. This multi-modal data fusion method effectively reduces the misjudgment rate, ensures the accurate distinction between dynamic targets and to-be-cleaned objects, and lays a reliable foundation for subsequent path planning. In a dynamic environment, the movement of the target object will directly affect the safety of the cleaning path. By extracting the motion characteristics of the target object and analyzing its correlation with surrounding objects, a trajectory prediction model is used to generate the future moving trajectory. At the same time, the system monitors the dynamic changes between target objects in real time and adjusts the path of the sweeping robot accordingly, improving the continuity and efficiency of the cleaning process. For large areas or high dynamic scenes, a multi-robot cooperative working strategy is proposed. The system evaluates the best cleaning mode through a cleaning prediction model based on the position, area and shape of the to-be-cleaned object, the real-time position and cleaning ability of each sweeping robot. The robot with stronger cleaning ability is preferentially assigned to high-difficulty areas, while other robots are assigned as needed. In addition, the system adjusts the working range of the robot in real time through a dynamic task allocation algorithm to avoid repeated cleaning or coverage blind spots. This intelligent cooperation mechanism significantly improves the overall cleaning efficiency. At the same time, the path planning module selects the optimal driving route and time period based on the speed range and energy consumption characteristics of the robot, reducing invalid movement and energy waste. Through the cooperation of the above technical solutions, the sweeping robot realizes efficient, safe and intelligent cleaning in a dynamic environment, solving the problems of inaccurate target recognition, rigid path planning and low multi-machine cooperation efficiency in traditional methods.

[0084] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, system and unit can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0085] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0086] The above-described embodiments are merely used to illustrate the technical solutions of the present application, rather than limit the same; even though the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

[0087] The above-described embodiments are merely used to illustrate the technical solutions of the present application, rather than limit the same; even though the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for target following cleaning of a sweeping robot based on visual tracking and radar ranging, characterized in that, The method comprises: real-time acquisition of first data and second data of a cleaning area by a camera unit and a distance measuring unit of each robot, and sending to a detection unit; the detection unit receives and acquires a plurality of target objects and objects to be cleaned according to the first data and the second data, fuses the first data and the second data, and acquires a plurality of target object information and cleaning information in the cleaning area; extracting motion features and correlation between the target objects according to each target object information, and acquiring a moving track of each target object based on the motion features, the correlation and a trajectory prediction model; determining a target robot and a best cleaning mode according to the cleaning information, the cleaning ability and the position information of each robot; acquiring path information and corresponding driving information of each target robot based on the moving track of each target object and the position information of the target robot; each target robot reaches the cleaning area and performs cleaning according to the corresponding path information and driving information; wherein the determination of the target robot and the best cleaning mode comprises: sending the cleaning information to each idle robot within a preset range of the object to be cleaned according to the position information of each robot and the position information of the object to be cleaned, receiving first cleaning information and a best cleaning mode predicted by the robot after cleaning, and taking the robot with the largest difference between the first cleaning information and the cleaning information as a first target robot, wherein the difference is the cleaning ability of the robot; sending the first cleaning information of the first target robot to other robots within the preset range, receiving second cleaning information and a best cleaning mode predicted by each other robot, and taking the other robot that meets the cleaning standard corresponding to the second cleaning information and also meets the smallest difference between the first cleaning information and the cleaning information as a second target robot, wherein the target robot comprises the first target robot and the second target robot.

2. The method of claim 1, wherein, acquiring a plurality of target objects and objects to be cleaned comprises: the detection unit inputs the first data into a first recognition model to acquire a first category and a first category probability of a first object in the first data, and also inputs the second data into a second recognition model to acquire a second category and a second category probability of a second object in the second data; calculating the distance between the first object and the second object, when the distance is less than or equal to a set distance, the first object and the second object are the same object, and the same object is taken as the target object, otherwise, they are not the same object; In the case that the first category probability and the second category probability are both greater than or equal to the first threshold value, the first category is taken as the category of the target object; in the case that one of the first category probability and the second category probability is greater than the first threshold value and the other is less than or equal to the second threshold value, the first data and the second data are re-acquired, and the step is repeated; in the case that the first category probability and the second category probability are both less than or equal to the second threshold value, the target object is a to-be-cleaned object; In the case that the first object and the second object are not the same object, in the case that the first category probability or the second category is greater than or equal to the first threshold value, the first object or the second object is taken as the target object, otherwise, the first object or the second object is taken as the to-be-cleaned object.

3. The method of claim 1, wherein, The moving track of each target object is acquired, including: The motion feature of the target object is acquired through the target object information, the correlation between the first motion feature of the target object and the second motion feature of other target objects in a preset range around the target object is analyzed, the motion feature of the target object is input into the track prediction model to acquire the moving track of the target object, and the moving track of the other target objects is acquired based on the moving track of the target object and the correlation, wherein the motion feature includes shape, motion speed, motion posture and motion track.

4. The method of claim 1, wherein, The path information and the travel information of each target robot are acquired, including: The connecting line between the position information of the target robot and the position information of the to-be-cleaned object is acquired, the path segment and the time segment with a distance greater than a set value between the moving track corresponding to any target object are acquired in a preset width range on both sides of the connecting line, and the path information and the travel information of the target robot are acquired based on the speed range, the position information of the target robot, the path segment and the time segment corresponding to the path segment.

5. The method of claim 1, wherein, In the case that the correlation between the target object and other target objects changes in the process of the target robot reaching the to-be-cleaned position or in the cleaning process, the moving track corresponding to the target object and other target objects is updated in real time, and the new path information of the target robot is acquired according to the updated moving track.

6. The method of claim 1, wherein, In the case that the number of target robots is greater than or equal to 2, the target robot with a stronger cleaning ability has a higher cleaning order.

7. The method of claim 1, wherein, The target object information includes the category, image and position information of the target object, and the to-be-cleaned information at least includes the category, shape, image, area, volume and position information of the to-be-cleaned object.

8. A target following cleaning system for a robotic floor-sweeper based on visual tracking and radar ranging, the system being for performing the method of any one of claims 1-7, characterized in that, The system includes: The robot includes a camera unit, a distance measuring unit and a communication unit, wherein the camera unit is used to acquire the first data of the cleaning area in real time, the distance measuring unit is used to acquire the second data of the cleaning area in real time, and the communication unit is used to send the first data and the second data to the detection unit. The detection unit is configured to receive and acquire a plurality of target objects and objects to be cleaned according to the first data and the second data, fuse the first data and the second data, and acquire target object information and to-be-cleaned information of the plurality of target objects in the cleaning area; The prediction unit is configured to extract motion features according to the target object information, and predict a moving track of each target object based on the motion features and a track prediction model; The determination unit is configured to determine a target robot and a cleaning mode according to the to-be-cleaned information, a cleaning capability and position information of each robot; The calculation unit is configured to acquire path information and travel information of each target robot based on the moving track of the object and the position information of the target robot; The robot is further configured to reach the to-be-cleaned area according to the corresponding path information and travel information, and clean based on an optimal cleaning mode.

9. A computer-readable storage medium having stored thereon instructions, the computer-readable storage medium comprising: The instructions, when executed by a processor, implement the method of any one of claims 1-7.

Images