Sanitation sweeping decision-making method based on cleaning robot and related equipment thereof

Abstract

The invention discloses an environmental sanitation cleaning decision-making method based on a cleaning robot and related equipment thereof, and relates to the technical field of environmental sanitation. The mistaken cleaning operation of the cleaning robot on the dynamic obstacle can be effectively avoided. The method comprises the following steps: firstly, obtaining multi-modal data of an environmental sanitation area, and carrying out time-space synchronization and coordinate system unification, so that environmental perception data is comprehensive and accurate; then, by dividing the environment sensing data into dynamic objects and static objects, dynamic obstacles such as pedestrians and animals can be accurately distinguished. Thirdly, removing dynamic objects in the environment sensing data, and filling the removed area to obtain static environment data only containing the static objects; and finally, comparing the static environment data with a historical map, and identifying a real cleaning target. Therefore, the robot can avoid dynamic obstacles and only clean static garbage, pollutants and other cleaning objects, so that the risk of accidental injury to pedestrians and animals is effectively avoided, and the cleaning safety and pertinence are improved.

Description

Technical Field

[0001] The present application relates to the field of environmental sanitation technology, and in particular to an environmental sanitation cleaning decision-making method based on a cleaning robot and related equipment. Background Art

[0002] With the acceleration of urbanization, urban environmental sanitation management is facing increasing pressure. The traditional manual sanitation cleaning method is inefficient and cannot meet the growing demand for urban cleaning. In order to improve the efficiency of sanitation work and save labor costs, various cleaning robots have emerged and are widely used in the field of sanitation.

[0003] Related cleaning robots usually use a single type of sensor, such as lidar or camera, to perceive the environment. The robot processes the data collected by the sensor and constructs a two-dimensional or three-dimensional map model of the environment. Then, the robot generates a cleaning path covering the entire area through certain path planning algorithms in the map, such as coverage path planning or random tree search. Finally, according to the planned path, the robot controls the chassis motor, cleaning brush and other actuators to move in sequence, cleans and collects ground garbage or pollutants, and completes the entire cleaning process.

[0004] However, in sanitation scenarios, related cleaning robots often indiscriminately clean or collect dynamic obstacles such as pedestrians and animals together with cleaning objects such as garbage and pollutants. This not only creates the risk of accidental injury to pedestrians and animals, but also makes the robot's cleaning behavior abnormal and inefficient. Summary of the invention

[0005] The present application provides a sanitation cleaning decision-making method based on a cleaning robot and related equipment, which are used to avoid the risk of accidental injury to pedestrians and animals and make the robot's cleaning behavior efficient and normal.

[0006] In the first aspect, the present application provides a sanitation cleaning decision-making method based on a cleaning robot, including: obtaining multimodal data of a sanitation area to be cleaned, the multimodal data being data obtained by different sensors detecting the sanitation area to be cleaned and containing geometric features of the sanitation area to be cleaned; performing spatiotemporal synchronization and coordinate system integration on the multimodal data to obtain environmental perception data; dividing the environmental perception data into dynamic objects and static objects; removing all dynamic objects in the environmental perception data and filling the removed areas to obtain static environmental data; comparing the static environmental data with a historical environmental map, and identifying suspected new targets through difference analysis; determining whether the suspected new target is obtained by filling; if not obtained by filling, determining the suspected new target as a cleaning target; and outputting a cleaning decision result based on the cleaning target.

[0007] By adopting the above technical solution, the cleaning robot can effectively avoid the mistaken cleaning operation of dynamic obstacles. First, by acquiring multimodal data of the sanitation area and performing spatiotemporal synchronization and coordinate system one, the environmental perception data is comprehensive and accurate. Then, by dividing the environmental perception data into dynamic objects and static objects, dynamic obstacles such as pedestrians and animals can be accurately distinguished. Next, the dynamic objects in the environmental perception data are removed and the removed areas are filled to obtain static environmental data containing only static objects. Finally, by comparing the static environmental data with the historical map, the real cleaning target is identified. In this way, the robot can avoid dynamic obstacles and only clean static garbage, pollutants and other cleaning objects, thereby effectively avoiding the risk of accidental injury to pedestrians and animals and improving the safety and pertinence of cleaning. At the same time, by removing the ineffective cleaning of dynamic obstacles, the robot's cleaning behavior has become more efficient and normal.

[0008] In combination with some embodiments of the first aspect, in some embodiments, the step of dividing the environmental perception data into dynamic objects and static objects specifically includes: selecting two consecutive frames of image data in the environmental perception data and determining them as the first image and the second image; matching and registering the first image and the second image; cropping the unregistered areas in the registered first image and the registered second image; identifying the difference area between the cropped first registered image and the cropped second registered image, determining the difference area as a dynamic object, and defining other areas as static objects; repeating the above steps until all image data are processed and all dynamic objects and static objects are obtained.

[0009] By adopting the above technical solution, dynamic objects in environmental perception data can be accurately identified. First, by selecting two consecutive frames of image data and matching and registering them, the correspondence between the two frames of images can be found. Then, by cropping the unregistered area and identifying the difference area of ​​the registered image, the changed area can be located, which represents the dynamic object. Finally, by repeatedly processing all image data, dynamic objects and static objects in the environment can be fully identified. Among them, matching and registration can eliminate image changes caused by robot movement, cropping unregistered areas can eliminate interference from image edges, and identifying difference areas can capture changes caused by object movement, laying the foundation for subsequent removal of dynamic interference and focusing on cleaning static targets.

[0010] In combination with some embodiments of the first aspect, in some embodiments, the step of dividing the environmental perception data into dynamic objects and static objects specifically includes: selecting two consecutive frames of image data in the environmental perception data and determining them as a third image and a fourth image; matching and registering the third image and the fourth image; cropping the unregistered areas in the registered third image and the registered fourth image; identifying the difference areas between the cropped third registered image and the cropped fourth registered image; repeating the above steps until all image data are processed to obtain all difference areas; continuously tracking and selecting the position changes of the current difference area in subsequent image data to obtain a series of candidate dynamic areas, and the current difference area is any difference area; counting the frequency of each candidate dynamic area being identified as a difference area in all frame image pairs; when the frequency is greater than the frequency threshold, determining the current difference area as a dynamic object; repeating the above steps until all difference areas are processed to obtain all dynamic objects; and defining other areas as static objects.

[0011] By adopting the above technical solution, a series of candidate dynamic regions can be obtained by continuously tracking the position changes of the difference regions in the continuous image data. These candidate regions represent the regions that are continuously changing in the image sequence, and are likely to correspond to real dynamic objects. Then, by counting the frequency of the candidate dynamic regions being identified as difference regions in all image pairs, their dynamic properties can be further determined. The higher the frequency, the more likely it is a real dynamic target. Finally, the candidate regions whose frequency exceeds the threshold are determined as the final dynamic objects. By combining tracking and frequency statistics, the interference of temporary and random image changes on the recognition of dynamic objects can be eliminated, and the robustness and reliability of recognition can be improved. At the same time, defining other regions as static objects can comprehensively describe the static and dynamic distribution in the environment, providing the robot with clear and complete environmental cognition.

[0012] In combination with some embodiments of the first aspect, in some embodiments, after the step of performing spatiotemporal synchronization and coordinate system one on the multimodal data to obtain environmental perception data, the method also includes: detecting feature points of multimodal data with the same time; using the feature points to calculate the transformation matrix of the multimodal data; using the transformation matrix to perform geometric changes on the multimodal data; and fusing the multimodal data after the geometric changes.

[0013] By adopting the above technical solution, the feature points in different sensor data are detected, and the correspondence between them can be found. Feature points often represent prominent locations in the environment, are consistent in different modal data, and can be used as a benchmark for fusion. Then, by calculating the transformation matrix between the feature points, the geometric correspondence of different modal data can be determined. The transformation matrix describes the mapping method from one data to another, and reflects their registration relationship in space. Next, the multimodal data is geometrically transformed using the transformation matrix, which can unify them into a common coordinate system and eliminate geometric deviations. Finally, the transformed multimodal data is fused to obtain geometrically consistent and complementary enhanced environmental perception data.

[0014] In combination with some embodiments of the first aspect, in some embodiments, the step of outputting the cleaning decision result according to the cleaning target specifically includes: calculating a cleaning path with the shortest path according to the cleaning target; and outputting the cleaning decision result according to the cleaning path.

[0015] By adopting the above technical solution, the working efficiency of the cleaning robot can be improved. After determining the cleaning target, by planning the shortest cleaning path, the robot can cover all cleaning targets with the shortest driving distance, reducing unnecessary repeated movements and detours, thereby shortening the time of the entire cleaning process and improving cleaning efficiency. At the same time, the shortest cleaning path also means that the robot can complete the established cleaning tasks with the least energy and extend the battery life. In addition, the shortest path can also reduce the wear and tear of the robot and increase its service life. By outputting the cleaning decision results according to the planned optimal cleaning path, the robot can quickly, efficiently and reliably complete the cleaning work in the sanitation area, greatly improving the level of intelligence of sanitation operations.

[0016] In combination with some embodiments of the first aspect, in some embodiments, after the step of determining the suspected new target as a cleaning target if it is not filled, the method further includes: if it is filled, determining the suspected new target as a pending target.

[0017] By adopting the above technical solution, the suspected targets obtained by filling in are temporarily marked as pending targets instead of being directly cleaned. In this way, the robot can give priority to cleaning more certain targets, thereby improving the efficiency and accuracy of cleaning.

[0018] In combination with some embodiments of the first aspect, in some embodiments, after the step of outputting the cleaning decision result according to the cleaning target, the method also includes: after receiving the new historical environment map, identifying the changed parts of the new historical environment map; updating the historical environment map according to the changed parts so that the updated historical environment map is the same as the new historical environment map.

[0019] By adopting the above technical solution, the robot can continuously learn and update its environmental cognition by continuously receiving new historical environmental maps after the cleaning decision. Once the new map is found to be different from the old map, the robot's historical map will be automatically adjusted by identifying the changed part to keep it synchronized with the latest environmental status. This incremental learning method allows the robot to adapt to dynamic changes in the environment in a timely manner, such as new fixed facilities and removed old obstacles.

[0020] In the second aspect, the present application provides a cleaning robot, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the cleaning robot to execute the method described in the first aspect and any possible implementation of the first aspect.

[0021] In a third aspect, the present application provides a computer program product comprising instructions, which, when executed on a cleaning robot, enables the cleaning robot to execute the method described in the first aspect and any possible implementation of the first aspect.

[0022] In a fourth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on a cleaning robot, causes the cleaning robot to execute the method described in the first aspect and any possible implementation of the first aspect.

[0023] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. It can effectively avoid the cleaning robot's mistaken cleaning operation on dynamic obstacles. First, by acquiring multimodal data of the sanitation area and performing spatiotemporal synchronization and coordinate system unity, the environmental perception data is comprehensive and accurate. Then, by dividing the environmental perception data into dynamic objects and static objects, dynamic obstacles such as pedestrians and animals can be accurately distinguished. Next, the dynamic objects in the environmental perception data are removed and the removed areas are filled to obtain static environmental data containing only static objects. Finally, by comparing the static environmental data with the historical map, the real cleaning target is identified. In this way, the robot can avoid dynamic obstacles and only clean static garbage, pollutants and other cleaning objects, thereby effectively avoiding the risk of accidental injury to pedestrians and animals and improving the safety and pertinence of cleaning. At the same time, by removing the ineffective cleaning of dynamic obstacles, the robot's cleaning behavior has become more efficient and normal.

[0024] 2. Dynamic objects in environmental perception data can be accurately identified. First, by selecting two consecutive frames of image data and matching and registering them, the correspondence between the two frames of images can be found. Then, by cropping the unregistered area and identifying the difference area of ​​the registered image, the area that has changed can be located, which represents the dynamic object. Finally, by repeatedly processing all image data, dynamic objects and static objects in the environment can be fully identified. Among them, matching and registration can eliminate image changes caused by robot movement, cropping unregistered areas can eliminate interference from image edges, and identifying difference areas can capture changes caused by object movement, laying the foundation for subsequent removal of dynamic interference and focusing on cleaning static targets.

[0025] 3. By continuously tracking the position changes of the difference areas in the continuous image data, a series of candidate dynamic areas can be obtained. These candidate areas represent areas that continue to change in the image sequence and are likely to correspond to real dynamic objects. Then, by counting the frequency with which the candidate dynamic areas are identified as difference areas in all image pairs, their dynamic properties can be further determined. The higher the frequency, the more likely it is a real dynamic target. Finally, the candidate areas whose frequency exceeds the threshold are determined as the final dynamic objects. By combining tracking and frequency statistics, the interference of temporary and random image changes on dynamic object recognition can be eliminated, and the robustness and reliability of recognition can be improved. At the same time, defining other areas as static objects can comprehensively describe the static and dynamic distribution in the environment, providing the robot with clear and complete environmental cognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flowchart of a sanitation cleaning decision-making method based on a cleaning robot in an embodiment of the present application; Figure 2 is another flowchart of the sanitation cleaning decision-making method based on the cleaning robot in an embodiment of the present application; Figure 3 It is a schematic diagram of an exemplary hardware structure of the cleaning robot in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0028] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0029] See also Figure 1 , which is a flow chart of the sanitation cleaning decision-making method based on the cleaning robot in an embodiment of the present application.

[0030] S101, obtaining multimodal data of the sanitation area to be cleaned, where the multimodal data is data including geometric features of the sanitation area to be cleaned obtained by detecting the sanitation area to be cleaned by different sensors; Among them, multimodal data refers to multiple types of data from different sensors, which are used to represent different characteristics of the environment. The sanitation area to be cleaned refers to the target area that needs to be cleaned, usually referring to public places such as streets and squares in the city. Geometric features refer to the spatial attributes of objects such as shape, size, and position, reflecting the three-dimensional structure of the environment.

[0031] In some embodiments, before step S101, each sensor is controlled to collect data on the cleaning area. The sensors may include cameras, laser radars, ultrasonic waves, etc., which respectively obtain data in different modes such as images, point clouds, and distances. These sensors perceive the environment from different angles, and the data contains geometric features of objects in the environment, such as object contours, occupancy, etc. The collection process can be one-time or continuous until sufficient data covering the entire cleaning area is obtained.

[0032] S102, performing time-space synchronization and coordinate system 1 on the multimodal data to obtain environmental perception data; Among them, time-space synchronization refers to aligning the data obtained by different sensors in time and space, eliminating time and space deviations, and expressing the data in the same time-space framework. Coordinate system 1 refers to converting data from different sensors into the same reference coordinate system to eliminate coordinate offsets caused by differences in the installation positions and angles of different sensors.

[0033] Specifically, based on the timestamp and acquisition frequency of each sensor data, time alignment is performed through interpolation, averaging and other methods to synchronize different sensor data in time. Then, using the external parameters (relative position and angle) and internal parameters (focal length, distortion, etc.) between sensors, coordinate transformation and image correction are performed to unify the different sensor data into the robot body coordinate system and align them in space. Finally, the images, point clouds and other data after time and space alignment are fused to generate complete environmental perception data.

[0034] In some embodiments, the sampling points of different sensors may be completely synchronized or close to each other, so that the multimodal data may be considered to be time synchronized.

[0035] S103, dividing the environmental perception data into dynamic objects and static objects; Among them, dynamic objects refer to objects whose position or shape changes over multiple time steps, usually moving objects, such as pedestrians, vehicles, etc. Static objects refer to objects whose position and shape remain unchanged over multiple time steps, usually fixed objects, such as buildings, trees, etc.

[0036] In some embodiments, S103 specifically includes: S1031, selecting two consecutive frames of image data from the environment perception data and determining them as a first image and a second image; In some embodiments, there is usually a certain time interval between two selected frames of images to reflect the movement changes of the object, but the interval should not be too large to avoid the object moving too fast and being difficult to match.

[0037] S1032, matching and registering the first image and the second image; It should be noted that since the first image and the second image are acquired continuously and the time interval between them is short, it can be assumed that the shooting angles of the two images are basically the same. In this case, a large part of the first image and the second image will overlap, and only a small part of the area will be different due to scene changes or the presence of moving objects. The purpose of registration is to find the overlapping areas in the two images and align them as accurately as possible so that the positions of the same object or scene in the two images correspond to each other.

[0038] In some embodiments, a region-based registration method is used. First, a region of interest in the first image is selected, such as a square region in the center of the image. Then, the region most similar to the region of interest is searched in the second image, and similarity metrics such as normalized cross-correlation and mutual information can be used. Next, by maximizing the similarity metric, such as exhaustive search, gradient descent and other optimization algorithms, the region in the second image that best matches the region of interest in the first image is found. Finally, based on the coordinate difference of the matching area, the translation transformation matrix between the images is estimated, and the coordinate transformation of the second image is performed to align it with the first image to complete the registration.

[0039] S1033, cropping the unregistered areas in the registered first image and the registered second image; The unregistered area refers to the area where the correspondence cannot be found due to the movement of the robot after the two images are registered, and is usually located at the edge of the image.

[0040] S1034, identifying a difference region between the cropped first registered image and the cropped second registered image, determining the difference region as a dynamic object, and defining other regions as static objects; In some embodiments, the identification of difference areas and the classification of dynamic and static objects can be achieved in a variety of ways: Optionally, a frame difference-based method is used. The cropped registered images are subtracted pixel by pixel to obtain a difference map. The difference map is thresholded to extract significant difference areas. The difference areas are marked as dynamic objects, and the non-difference areas are marked as static objects. Optionally, a background modeling-based method is used. Using the cropped registered image sequence and algorithms such as Gaussian mixture models, a background model of the scene is constructed. The current frame is compared with the background model to identify significantly different foreground areas as dynamic objects. The background areas are marked as static objects. It can be understood that other dynamic and static object recognition methods can also be used, which are not limited here.

[0041] S1035, repeat the above steps until all image data are processed and all dynamic objects and static objects are obtained.

[0042] S104, removing all dynamic objects in the environmental perception data and filling the removed areas to obtain static environmental data; Among them, removing dynamic objects refers to removing dynamic objects identified in the environmental perception data from the data, so that the environmental representation only contains static objects. Filling refers to repairing the vacant areas in the environmental representation after removing dynamic objects, so that the environmental representation remains complete and continuous.

[0043] In some embodiments, the dynamic object pixels are set to empty (eg, 0 value) or removed, which is not limited here.

[0044] In some embodiments, interpolation is performed based on the texture and structure of surrounding pixels; timing information may also be used to fill in the background before and after the object disappears, which is not limited here.

[0045] S105, comparing the static environment data with the historical environment map, and identifying suspected new targets through difference analysis; In some embodiments, suspected new targets can be realized in a variety of ways: Optionally, a frame difference-based method is used. The cropped registered images are subtracted pixel by pixel to obtain a difference map. The difference map is thresholded to extract areas of significant difference. Optionally, a background modeling-based method is used. Using the cropped registered image sequence, a background model of the scene is constructed using algorithms such as a Gaussian mixture model. The current frame is compared with the background model to identify significantly different foreground areas as suspected new targets. It is understandable that other dynamic and static object recognition methods can also be used, which are not limited here.

[0046] S106, determining whether the suspected newly added target is obtained by filling; In some embodiments, the judgment of suspected new targets and filling results can be achieved in a variety of ways: Optionally, a method based on overlapping area is used. The intersection area of ​​the suspected new target area and the original dynamic object area is calculated. If the ratio of the intersection area to the target area exceeds a certain threshold (such as 50%), it is determined that the target is obtained by filling, which is not limited here.

[0047] If the target is filled, the suspected new target will be identified as a pending target. It can be seen that the suspected targets obtained by filling in are temporarily marked as pending targets instead of being directly cleaned. In this way, the robot can give priority to cleaning more certain targets, thereby improving the efficiency and accuracy of cleaning.

[0048] S107, if the target is not filled, the suspected new target is determined as a cleaning target; In some embodiments, the suspected new targets that are not obtained by filling are further analyzed to extract their visual features and geometric features, such as appearance texture, shape outline, volume size, etc., and then use the pre-established garbage classification model to determine whether the target belongs to a certain type of garbage or debris. If the target meets the characteristics of garbage, it is determined as a cleaning target and the target type, location, volume and other attribute information are marked. If the target does not meet the characteristics of garbage, it is discarded and not included in the subsequent cleaning range. Through this step, the real garbage target can be accurately identified from the environmental changes and cleaning can be carried out in a targeted manner.

[0049] S108. Output a cleaning decision result according to the cleaning target.

[0050] The cleaning decision result refers to the next cleaning action plan for the determined cleaning target, including cleaning sequence, path planning, cleaning tool selection, etc., which is used to guide the actual work of the cleaning robot. The output refers to converting the decision result into a control instruction, which is passed to the robot's execution module to drive it to perform the cleaning operation.

[0051] In some embodiments, S108 specifically includes: S1081, calculating the shortest cleaning path according to the cleaning target; In some embodiments, the shortest cleaning path planning based on the cleaning target can be achieved in a variety of ways: Optionally, an A search algorithm based on a grid map is used. The environmental map is divided into grids, and the cost value of the grid is set according to the cleaning target and the obstacle position. Starting from the starting point, a heuristic search is performed with the grid as a node to evaluate the lengths of different paths until the shortest path covering the cleaning target is found. Optionally, a sampling-based RRT algorithm is used. Random sampling is performed in the state space, and a tree structure is constructed based on the distance between the sampling point and the cleaning target and the obstacle. By continuously expanding and optimizing the tree, the shortest collision-free path connecting the starting point and the cleaning target is searched. It can be understood that other path planning methods, such as genetic algorithms, Dijkstra algorithms, etc., can also be used, which are not limited here.

[0052] S1082. Output the cleaning decision result according to the cleaning path.

[0053] It can be seen that the working efficiency of the cleaning robot can be improved. After determining the cleaning target, by planning the shortest cleaning path, the robot can cover all cleaning targets with the shortest driving distance, reducing unnecessary repeated movements and detours, thereby shortening the time of the entire cleaning process and improving cleaning efficiency. At the same time, the shortest cleaning path also means that the robot can complete the established cleaning tasks with the least energy and extend the battery life. In addition, the shortest path can also reduce the wear and tear of the robot and increase its service life. By outputting the cleaning decision results according to the planned optimal cleaning path, the robot can quickly, efficiently and reliably complete the cleaning work in the sanitation area, greatly improving the level of intelligence of sanitation operations.

[0054] It can be seen that the cleaning robot can effectively avoid the accidental cleaning operation of dynamic obstacles. First, by acquiring multimodal data of the sanitation area and performing spatiotemporal synchronization and coordinate system unity, the environmental perception data is comprehensive and accurate. Then, by dividing the environmental perception data into dynamic objects and static objects, dynamic obstacles such as pedestrians and animals can be accurately distinguished. Next, the dynamic objects in the environmental perception data are removed and the removed areas are filled to obtain static environmental data containing only static objects. Finally, the real cleaning target is identified by comparing the static environmental data with the historical map. In this way, the robot can avoid dynamic obstacles and only clean static garbage, pollutants and other cleaning objects, thereby effectively avoiding the risk of accidental injury to pedestrians and animals and improving the safety and pertinence of cleaning. At the same time, by removing the ineffective cleaning of dynamic obstacles, the robot's cleaning behavior has become more efficient and normal.

[0055] See also Figure 2 , which is another flow chart of the sanitation cleaning decision-making method based on the cleaning robot in an embodiment of the present application.

[0056] S201, selecting two consecutive frames of image data from the environment perception data and determining them as a third image and a fourth image; It should be noted that the principle and process of this step are similar to those of step S1031. For relevant implementation details, reference may be made to step S1031 and no limitation is given here.

[0057] S202, matching and registering the third image and the fourth image; It should be noted that the principle and process of this step are similar to those of step S1032. For relevant implementation details, reference may be made to step S1032 and no limitation is given here.

[0058] S203, cropping the unregistered areas in the registered third image and the registered fourth image; It should be noted that the principle and process of this step are similar to those of step S1033. For relevant implementation details, reference may be made to step S1033 and no limitation is given here.

[0059] S204, identifying a difference area between the cropped third registered image and the cropped fourth registered image; It should be noted that the principle and process of this step are similar to those of step S1034. For relevant implementation details, reference may be made to step S1034 and no limitation is given here.

[0060] S205: Whether all image data have been processed; After all image data are processed and all difference regions are obtained, step S206 is executed; If all image data have not been processed, execute step S201; S206, continuously tracking and selecting the position change of the current difference region in subsequent image data to obtain a series of candidate dynamic regions, wherein the current difference region is any difference region; The subsequent image data refers to the image data that has not been processed in steps S201-S205 and is later than the current difference area in time. Continuous tracking refers to continuously locating and identifying the same target area in consecutive image frames. The candidate dynamic area represents the suspected moving target area found in the tracking process, and its dynamic properties need to be further determined. The position change is used to represent the motion trajectory of the target area in the image sequence.

[0061] In some embodiments, continuous tracking of the difference area can be achieved in a variety of ways: Optionally, a tracking method based on feature matching is adopted. The significant feature points of the difference area are extracted, such as SIFT, ORB, etc., and similar feature points are searched in subsequent frames to complete the area matching. The matching area is the candidate dynamic area, and the motion state is judged according to the change of the area position. Optionally, a tracking method based on correlation filtering is adopted. The difference area is used as a template for the correlation filter, and a sliding window similarity calculation is performed in subsequent frames. The area with the highest similarity is used as a candidate dynamic area, and the motion state is judged according to the change of the area position. It can be understood that other target tracking methods can also be used, such as tracking based on deep learning, tracking based on optical flow, etc., which are not limited here.

[0062] S207, counting the frequency of each candidate dynamic region being identified as a difference region in all frame image pairs; In some embodiments, for each candidate dynamic region, all frame image pairs are traversed. It is counted whether the candidate region is identified as a difference region in each image pair, and the total number of its occurrences is calculated. The number of occurrences is divided by the total number of image pairs to obtain the dynamic frequency of the candidate region. The higher the frequency, the more likely the candidate region is to be a true dynamic target.

[0063] S208: When the frequency is greater than the frequency threshold, determine the current difference area as a dynamic object; S209: Have all difference areas been processed? After processing all the difference regions and obtaining all the dynamic objects, execute step S210; If all the difference regions have not been processed, execute step S206; Processing refers to the process of dynamic attribute judgment and marking of the difference area, namely steps S206 to S208. Processing all the difference areas means that tracking, statistics and judgment are completed for each difference area, and finally it is determined whether it is a dynamic object.

[0064] Step S209 is executed after processing a difference region each time, in order to check whether there are any difference regions that have not been processed. Specifically, all difference regions obtained in step S204 are traversed to check whether they have been tracked and dynamically judged. If there are any difference regions that have not been processed, one of them is selected and the process goes to step S206. S210: Define other areas as static objects.

[0065] It can be seen that by continuously tracking the position changes of the difference areas in the continuous image data, a series of candidate dynamic areas can be obtained. These candidate areas represent the areas that are continuously changing in the image sequence, and they are likely to correspond to real dynamic objects. Then, by counting the frequency of candidate dynamic areas being identified as difference areas in all image pairs, their dynamic properties can be further judged. The higher the frequency, the more likely it is a real dynamic target. Finally, the candidate areas whose frequency exceeds the threshold are determined as the final dynamic objects. By combining tracking and frequency statistics, the interference of temporary and random image changes on dynamic object recognition can be eliminated, and the robustness and reliability of recognition can be improved. At the same time, defining other areas as static objects can comprehensively describe the static and dynamic distribution in the environment, providing the robot with clear and complete environmental cognition.

[0066] In actual use, in actual application scenarios, environmental perception data usually comes from multiple sensors, such as cameras, radars, lidars, etc. with multiple viewing angles. At the same time, these sensors will provide data from different viewing angles or modalities, thus forming a comprehensive perception of the environment. However, if the processing flow from S103 to S108 is executed separately for the data of each sensor, the amount of calculation will increase significantly.

[0067] In some embodiments, after S102 and before S103, the process further includes the following steps: S1, detecting feature points of multimodal data with the same time; Feature points refer to representative key locations in the data, which usually have unique texture, structure or geometric properties within a certain neighborhood.

[0068] S2, using the feature points to calculate the transformation matrix of the multimodal data; The transformation matrix refers to the matrix parameters that describe the spatial transformation relationship between different modal data. The matrix can be used to align the data of one modality with another. The matrix usually includes two parts: the rotation matrix and the translation vector. Feature points are used to estimate the transformation matrix and are the basis for establishing data association.

[0069] In some embodiments, the transformation matrix calculation of multimodal data can be implemented in a variety of ways: Optionally, an estimation method based on least squares method is adopted. According to the matching pairs of feature points, the error function is listed, and the transformation matrix is ​​differentiated to obtain a closed-form solution. This method is simple to calculate, but sensitive to mismatches. Optionally, an estimation method based on RANSAC is adopted. Randomly sample a small number of feature point pairs, estimate the transformation matrix, evaluate the consistency of other point pairs, and iterate and optimize. This method can effectively eliminate mismatches and improve the robustness of the estimation. It can be understood that other transformation matrix estimation methods, such as nonlinear optimization, Kalman filtering, etc., can also be used, which are not limited here.

[0070] S3, using the transformation matrix to perform geometric changes on the multimodal data; Common transformations include rotation, translation, scaling, etc. By using the transformation matrix to perform operations, data of one modality can be mapped to the coordinate system of another modality to achieve data registration.

[0071] The S3 step is performed after the transformation matrix is ​​estimated. Its purpose is to spatially align data of different modalities, eliminate geometric differences, and prepare for data fusion. Specifically, for each modal data, the estimated transformation matrix is ​​used to transform its coordinates. By multiplying the data by the rotation matrix and adding the translation vector, it is mapped to a unified spatial coordinate system. The transformed data is aligned in space, reflecting the different attributes of the same scene. This step is the key to achieving multimodal data fusion, which can unify data from different perspectives and scales to facilitate subsequent information extraction and analysis.

[0072] In some embodiments, the geometric changes of multimodal data can be achieved in a variety of ways: Optionally, a method based on rigid body transformation is adopted. Assuming that there are only rotation and translation differences between different modal data, the coordinates of the data points are linearly transformed using the transformation matrix to achieve fast alignment. Optionally, a method based on non-rigid body transformation is adopted. Considering the scale and perspective differences between different modal data, more general transformation models such as affine transformation and projective transformation are introduced to perform nonlinear transformation on the data to achieve more accurate alignment. It is understandable that other geometric transformation methods can also be used, such as deep learning-based alignment, surface matching-based alignment, etc., which are not limited here.

[0073] S4. Fuse the multimodal data after geometric changes.

[0074] The data layer fusion method is used to align different modal data to the same coordinate system, uniformly interpolate to the same grid or pixel, and splice point by point to form fused data.

[0075] It can be seen that by detecting the feature points in different sensor data, the correspondence between them can be found. Feature points often represent prominent locations in the environment, are consistent in different modal data, and can be used as a benchmark for fusion. Then, by calculating the transformation matrix between the feature points, the geometric correspondence between different modal data can be determined. The transformation matrix describes the mapping method from one data to another, reflecting their registration relationship in space. Next, the multimodal data can be geometrically transformed using the transformation matrix to unify them into a common coordinate system and eliminate geometric deviations. Finally, the transformed multimodal data is fused to obtain geometrically consistent and complementary enhanced environmental perception data.

[0076] In some embodiments, after step S108, the method further includes: S109, after receiving the new historical environment map, identifying the changed part of the new historical environment map; S1010. Update the historical environment map according to the changed part so that the updated historical environment map is the same as the new historical environment map.

[0077] It can be seen that by continuously receiving new historical environment maps after the cleaning decision, the robot can continuously learn and update its environmental cognition. Once the new map is found to be different from the old map, the robot's historical map will be automatically adjusted by identifying the changed part to keep it synchronized with the latest environmental status. This incremental learning method allows the robot to adapt to dynamic changes in the environment in a timely manner, such as newly added fixed facilities and removed old obstacles.

[0078] The following introduces an exemplary cleaning robot 100 provided in an embodiment of the present application. Figure 3 It is a schematic diagram of an exemplary hardware structure of the cleaning robot 100 provided in an embodiment of the present application.

[0079] In some embodiments, the cleaning robot 100 is a computer device or the cleaning robot 100 includes a computer device. The computer device includes a processor, a memory and a network interface connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with other external terminals or servers through a network connection. In some embodiments, the network interface can be a wired network interface, and in some embodiments, the network interface can also be a wireless network interface. When the computer program is executed by the processor, the method in the embodiment of the present application is implemented.

[0080] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0081] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0082] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0083] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0084] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A sanitation cleaning decision-making method based on a cleaning robot, characterized in that: include: Acquire multimodal data of the sanitation area to be cleaned, wherein the multimodal data is data including geometric features of the sanitation area to be cleaned obtained by detecting the sanitation area to be cleaned by different sensors; Performing spatiotemporal synchronization and coordinate system one on the multimodal data to obtain environmental perception data; dividing the environmental perception data into dynamic objects and static objects; Removing all dynamic objects from the environmental perception data and filling the removed areas to obtain static environmental data; Comparing the static environmental data with the historical environmental map, and identifying suspected new targets through difference analysis; Determine whether the suspected new target is obtained by filling; If it is not filled, the suspected new target is determined as a cleaning target; Output a cleaning decision result according to the cleaning target.

2. The method according to claim 1, characterized in that: The step of dividing the environmental perception data into dynamic objects and static objects specifically includes: Selecting two consecutive frames of image data from the environmental perception data to determine them as a first image and a second image; performing matching and registration on the first image and the second image; cropping unregistered regions in the registered first image and the registered second image; identifying a difference region between the cropped first registered image and the cropped second registered image, determining the difference region as a dynamic object, and defining other regions as static objects; Repeat the above steps until all image data are processed and all dynamic objects and static objects are obtained.

3. The method according to claim 1, characterized in that The step of dividing the environmental perception data into dynamic objects and static objects specifically includes: Selecting two consecutive frames of image data from the environmental perception data to be the third image and the fourth image; performing matching and registration on the third image and the fourth image; cropping unregistered regions in the registered third image and the registered fourth image; identifying a difference region between the cropped third registered image and the cropped fourth registered image; Repeat the above steps until all image data are processed and all difference areas are obtained; Continuously tracking and selecting position changes of the current difference region in subsequent image data to obtain a series of candidate dynamic regions, wherein the current difference region is any difference region; Count the frequency of each candidate dynamic region being identified as a difference region in all frame image pairs; When the frequency is greater than a frequency threshold, determining the current difference region as a dynamic object; Repeat the above steps until all difference areas are processed and all dynamic objects are obtained; Define other areas as static objects.

4. The method according to claim 1, characterized in that: After the step of performing spatiotemporal synchronization and coordinate system 1 on the multimodal data to obtain environmental perception data, the method further includes: Detecting feature points of multimodal data with the same time; Using the feature points to calculate a transformation matrix of the multimodal data; Using the transformation matrix to geometrically transform the multimodal data; Fusion of multimodal data after geometric changes.

5. The method according to claim 1, characterized in that: The step of outputting the cleaning decision result according to the cleaning target specifically includes: Calculate the shortest cleaning path according to the cleaning target; Output the cleaning decision result according to the cleaning path.

6. The method according to claim 1, characterized in that After the step of determining the suspected newly added target as a cleaning target if the target is not filled, the method further includes: If it is filled, the suspected new target will be determined as a pending target.

7. The method according to claim 1, characterized in that After the step of outputting the cleaning decision result according to the cleaning target, the method further includes: After receiving the new historical environment map, identifying the changed parts of the new historical environment map; Update the historic environment map based on the changed parts so that the updated historic environment map is the same as the new historic environment map.

8. A cleaning robot, characterized in that: The cleaning robot comprises: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions so that the cleaning robot executes the method as described in any one of claims 1-7.

9. A computer program product comprising instructions, characterized in that When the computer program product runs on a cleaning robot, the cleaning robot is enabled to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on the cleaning robot, the cleaning robot is caused to perform the method according to any one of claims 1 to 7.

Images