Semantic map generation method, cleaning control method, cleaning robot, medium and product

Through the semantic map generation method, low objects are identified and marked, which solves the problem that traditional maps cannot be recognized, and achieves the satisfaction of cleaning needs of different users and improves user experience.

CN118936451APending Publication Date: 2024-11-12MIDEA ROBOZONE TECH CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411026191.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Traditional occupancy grid maps cannot effectively identify low objects on the ground that cannot be observed by lidar, causing the robot to enter the wrong area and the cleaning method is fixed, making it difficult to adapt to the changing needs of different users and have poor user experience.

Method used

The semantic map generation method is adopted to obtain scene images through the image acquisition device, determine the semantic information of the target object, and mark semantic blocks in the map to generate a map with rich scene information for precise cleaning area specification and adaptive cleaning mode.

Benefits of technology

It realizes the identification of low objects and precise cleaning area designation, meets the variable cleaning needs of different users and improves the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118936451A_ABST
    Figure CN118936451A_ABST
Patent Text Reader

Abstract

The invention discloses a semantic map generation method, a cleaning control method, a cleaning robot, a medium and a product. The cleaning robot comprises an image acquisition device and a radar, and the method comprises the steps that a scene image acquired by the image acquisition device is acquired, and the scene image at least comprises image information of a scene below the height of the radar; determining a target object in the scene image and semantic information of the target object; and based on the semantic information of the target objects, marking each target object in the map through the semantic blocks to generate a semantic map. The target objects and the semantic information of the target objects are recognized by collecting the scene images including the scenes which cannot be collected by the radar, so that the target objects are marked in the map through the semantic blocks, the semantic map with more scene information can be generated, and the cleaning efficiency is improved when the cleaning robot conducts cleaning based on the semantic map. The cleaning area can be specified more accurately and specifically, and variable cleaning requirements of different users can be met.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of cleaning technology, and in particular relates to a method for generating a semantic map, a cleaning control method based on a semantic map, a cleaning robot, a computer-readable storage medium, and a computer program product. Background Art

[0002] Traditional occupancy grid maps are constructed using laser radar, which includes three states: "occupied", "free", and "unobserved". "Free" means there are no obstacles and the robot can pass through, while "occupied" means the opposite. For sweeping robots, laser radars are usually installed on top of them, that is, "free" at the height of the laser radar means that the sweeping robot can pass through, which will miss low objects on the ground that cannot be observed by lasers, such as carpets, wires, etc. This missed recognition will cause the robot to enter the wrong "travelable area" and run over low objects that are not on the ground.

[0003] The traditional grid map has single information and cannot provide the robot with rich scene information. Users can only clean based on several preset cleaning modes. The cleaning method is relatively fixed and difficult to adapt to the changing cleaning needs of different users, resulting in a poor user experience. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a method for generating a semantic map, a cleaning control method based on a semantic map, a cleaning robot, a computer-readable storage medium, and a computer program product, which can meet the changing cleaning needs of different users and provide a good user experience.

[0005] In a first aspect, the present application provides a method for generating a semantic map, which is applied to a cleaning robot, wherein the cleaning robot includes an image acquisition device and a radar, and the method includes acquiring a scene image acquired by the image acquisition device, wherein the scene image includes at least image information of the scene below the height of the radar; determining a target object in the scene image and semantic information of the target object; and based on the semantic information of the target object, marking each target object in a map through semantic blocks to generate a semantic map.

[0006] In the second aspect, the present application provides a cleaning control method based on a semantic map, which is used to control a cleaning robot. The semantic map contains a map and semantic information of each area of ​​the map. The method includes, upon receiving a cleaning task, identifying the semantic information in the cleaning task; determining a target area in the semantic map whose semantic information matches the semantic information of the cleaning task; and executing the cleaning task based on the target area.

[0007] In a third aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned semantic map generation method or the semantic map-based cleaning control method.

[0008] In a fourth aspect, the present application provides a cleaning robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for generating a semantic map or the cleaning control method based on the semantic map is implemented.

[0009] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned semantic map generation method or the semantic map-based cleaning control method.

[0010] In the semantic map generation method, the cleaning control method based on the semantic map, the cleaning robot, the computer-readable storage medium and the computer program product provided in the embodiments of the present application, scene images including scenes that cannot be collected by radar are collected to identify the target objects and the semantic information of the target objects, so that each target object is marked in the map through semantic blocks, and a semantic map with more scene information can be generated. When the cleaning robot cleans based on the semantic map, the cleaning area designation can be more accurate and specific, which can meet the changing cleaning needs of different users.

[0011] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0013] Figure 1 is a schematic diagram of the structure of the cleaning robot provided in an embodiment of the present application;

[0014] Figure 2 It is a process scene diagram of the method for generating a semantic map provided in an embodiment of the present application;

[0015] Figure 3 is a schematic diagram of a semantic map of an indoor scene provided in an embodiment of the present application;

[0016] Figure 4 It is a process scene diagram of the method for generating a semantic map provided in an embodiment of the present application;

[0017] Figure 5 It is a process scenario diagram of the method for generating a semantic map provided in an embodiment of the present application;

[0018] Figure 6 It is a process scenario diagram of the cleaning control method provided in the embodiment of the present application;

[0019] Figure 7 It is a process scenario diagram of the cleaning control method provided in the embodiment of the present application;

[0020] Figure 8 It is a process scenario diagram of the cleaning control method provided in the embodiment of the present application;

[0021] Fig. 9 It is a process scenario diagram of the cleaning control method provided in the embodiment of the present application;

[0022] Fig.10 It is a process scenario diagram of the cleaning control method provided in the embodiment of the present application;

[0023] Fig.11 is a schematic diagram of a module of a generating device provided in an embodiment of the present application; and

[0024] Fig.12 It is a module schematic diagram of the control device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0026] For ease of understanding, the cleaning robot of this application is first introduced below:

[0027] Self-moving robot: A machine device that performs work automatically. It can accept human commands, run pre-programmed programs, or act according to principles formulated by artificial intelligence technology.

[0028] A sweeping robot is a type of self-propelled robot. A sweeping robot, also known as an automatic sweeper, smart vacuum cleaner, robot vacuum cleaner, etc., is a type of smart home appliance that can automatically clean the floor in the room with the help of artificial intelligence. It generally uses brushing and vacuuming to absorb the debris on the ground into its own garbage collection box, thereby completing the function of floor cleaning. Generally speaking, robots that complete sweeping, vacuuming, and mopping are collectively classified as sweeping robots.

[0029] The cleaning robot of the present application may be a self-moving robot or a device including a self-moving robot. The self-moving robot may be a sweeping robot, or an autonomous mobile robot (AMR), etc. For the sake of brevity, the present application takes the self-moving robot as a sweeping robot as an example. The principles of the self-moving robot as other types of robots are similar and will not be repeated here.

[0030] In some embodiments, the cleaning robot 100 includes an image acquisition device 10, a radar 20, a processor 30, a memory 40 and a body 50; the processor 30 communicates with the image acquisition device 10 and the radar 20 respectively, and the image acquisition device 10, the radar 20 and the processor 30 are arranged on the body 50 of the cleaning robot 100.

[0031] The image acquisition device 10 is used to acquire scene images. The radar 20 is used to acquire point cloud information of objects in the scene.

[0032] The cleaning robot 100 can realize scene recognition (such as recognizing objects in the scene), positioning and navigation through devices such as the image acquisition device 10 and the radar 20.

[0033] Based on the introduction of the above-mentioned related scenarios, the embodiment of the present application provides a method for generating a semantic map, and the generation method is described in detail below:

[0034] See also Figure 2 , a method for generating a semantic map provided in an embodiment of the present application is applied to a cleaning robot. The method is implemented by steps 011 to 013, which are described in detail below.

[0035] Step 021: Acquire a scene image acquired by an image acquisition device, where the scene image at least includes image information of the scene below the height of the radar;

[0036] The cleaning robot includes an image acquisition device, which is used to acquire scene images, and the scene images at least include image information of the scene below the height of the radar of the cleaning robot.

[0037] Among them, semantic map is an advanced form of map representation method, which not only contains geographic space information (i.e. map), such as the location of roads, furniture, buildings, etc., but also incorporates rich semantic information about the environment, such as the types of roads, furniture, buildings, and the form of contact with the ground, etc. Semantic map plays a vital role in the fields of autonomous driving, robot navigation, virtual reality and augmented reality.

[0038] See also Figure 3 Optionally, the semantic map includes semantic information of the map and various regions of the map.

[0039] The area division of the map can be based on objects, such as the area covered by each object as a map area (in this case, the area is clustered into a semantic block with the same semantics). Alternatively, the map can be divided into grids of fixed size to obtain a grid map, where each grid of the grid map is a map area, and one or more grids with the same semantic information are clustered into a semantic block. That is, each semantic block corresponds to an object.

[0040] like Figure 3 As shown, the semantic information of each grid covered by the sofa in the living room is the sofa, and the semantic information of each grid covered by the table in bedroom 1 is the table.

[0041] Optionally, the semantic information of each region is semantic information of an object set on the region, such as object type information and contact type information with the ground.

[0042] The object type information of the region refers to the type of the object set in the region, such as the object is a building: wall, furniture: sofa, bed, wardrobe, table and chair legs, door, coffee table, etc., electrical appliances: refrigerator, washing machine, etc., clothing: shoes, backpack, etc.

[0043] The contact type information of the area includes at least one of the following types: covering type, obstacle type and suspended type. The covering type corresponds to an area where objects cover the ground and are lower than the obstacle crossing height of the cleaning robot. The cleaning robot can pass over covering objects, such as carpets and floor mats. The obstacle type corresponds to an area where objects are higher than the obstacle crossing height. The cleaning robot can only bypass obstacle objects, but cannot pass over or cross obstacle objects, such as trash cans, shoes, sofas, coffee tables, etc. The suspended type corresponds to an area where objects are suspended and are higher than the cleaning robot. The cleaning robot can pass under the suspended part (the part not in contact with the ground) of the suspended object, such as tables, chairs, beds, etc.

[0044] Specifically, since the radar of the cleaning robot is generally set on the top of the cleaning robot and has a certain installation height, objects below the installation height cannot be scanned. Therefore, the image acquisition device of the cleaning robot can be used to make up for this defect and obtain objects below the installation height of the radar. For example, there are generally multiple image acquisition devices, which are set at the front end and both sides of the cleaning robot. The installation height of the image acquisition device is lower than the radar sensor, and the image acquisition device has a vertical field of view, so that the scene image captured by the image acquisition device includes at least the image information of the scene below the height of the cleaning robot's radar (specifically the lowest height of the radar's field of view), thereby obtaining a scene image containing low objects on the ground.

[0045] Optionally, the image acquisition device may be a monocular camera. The monocular camera is relatively small in size, which is beneficial for reducing the size of the cleaning robot. The monocular camera is also relatively low in cost, which can also reduce the cost of the cleaning robot.

[0046] Step 022: determining a target object in the scene image and semantic information of the target object;

[0047] Step 023: Based on the semantic information of the target object, each target object is marked in the map through a semantic block to generate a semantic map.

[0048] The target object may be any object in the scene.

[0049] Specifically, the scene image is recognized to obtain the semantics and position of each object (including low objects), and the corresponding area in the map is semantically annotated according to the position and semantics of each object, so that each semantic block corresponding to each object is annotated in the map. In this way, a semantic map annotated with semantic information can be obtained, and the creation of a semantic map can be realized.

[0050] See also Figure 4 , step 022, comprising:

[0051] Step 0221: semantically segment the scene map to determine each target object in the scene and the semantic information of each target object;

[0052] Step 023:

[0053] Step 0231: Based on the semantic information of the target object and the image coordinates in the scene image, semantic projection is performed to mark the semantic blocks of each target object in the map to generate a semantic map.

[0054] Specifically, when creating a semantic map, the scene map is firstly semantically segmented to obtain each object in the scene, that is, the objects in the scene map are recognized. Semantic information of each object can also be determined, such as object type information and contact type information with the ground. For example, semantic segmentation can be achieved by a pre-trained semantic segmentation model.

[0055] Afterwards, based on the image coordinates of the object in the scene image, the position coordinate range corresponding to the object in the map is determined. For example, the image coordinates are projected onto the map through the internal and external references of the image acquisition device to obtain the various regions corresponding to the object in the map, and then semantic projection is performed based on the semantic information and position coordinate range of the object to mark the semantic blocks of each object in the map to generate a semantic map.

[0056] Optionally, the camera is kept on during the driving of the cleaning robot; or, the scene image is acquired in real time or by extracting key frames during the driving of the cleaning robot. It is understandable that due to factors such as scene changes or poor accuracy of scene images, semantic annotation may not be accurate once. Therefore, semantic annotation is a continuous process that requires continuous acquisition of scene images.

[0057] In the semantic map generation method, semantic map-based cleaning control method, control device, cleaning robot, computer-readable storage medium and computer program product provided in the embodiments of the present application, scene images including scenes that cannot be collected by radar are collected to identify target objects and semantic information of target objects, so that each target object is marked in the map through semantic blocks, and a semantic map with more scene information can be generated. When the cleaning robot cleans based on the semantic map, the cleaning area designation can be more accurate and specific, which can meet the changing cleaning needs of different users.

[0058] See also Figure 5 In some embodiments, the semantic map includes a plurality of regions, the semantic block covers one or more regions, and the method further includes:

[0059] Step 024: Acquire various semantic information when multiple semantic projections are performed on the current area, where the current area is any map area;

[0060] Step 025: based on the probability distribution of different semantic information in each semantic information;

[0061] Step 026: Update the semantic information of the current area based on the semantic information with the highest distribution probability.

[0062] Specifically, after each scene image is acquired, a semantic projection may be performed, that is, the semantic information of the recognized object is projected onto the map, and each area (or grid) in the map may be projected multiple times.

[0063] Taking the current area (i.e., the map area currently to be semantically annotated) as an example, the number of projections of different semantic information in multiple semantic projections of the current area can be counted, and the probability distribution of different semantic information can be determined based on the number of projections of different semantic information. For example, the distribution probability of any different semantic information is the ratio of the projection number of the semantic information to the total projection number of the current area, so that the semantic information with the largest distribution probability is determined as the semantic information of the current area (or target grid). In this way, the semantic annotation of each area can be completed, and the accuracy of the semantic annotation of each area can be improved through multiple semantic projections.

[0064] Optionally, when the probability of any different semantic information is too small (such as less than a preset threshold), the semantic information can be determined to be invalid and can be removed as noise. In this way, invalid semantic information can be removed to improve the accuracy of semantic annotation.

[0065] Optionally, after the semantic map is generated, the semantic map can be displayed on a terminal that is in communication with the cleaning robot. In this way, the user can intuitively view the semantic map and quickly find the area to be cleaned through the indication of the semantic block, thereby realizing fast and accurate designation of the cleaning area.

[0066] Based on the introduction of the above-mentioned related scenarios, the embodiment of the present application provides a cleaning control method, which is described in detail below:

[0067] See also Figure 6 A cleaning control method based on a semantic map provided in an embodiment of the present application is used to control a cleaning robot. The method is implemented by steps 011 to 013, which are described in detail below.

[0068] Step 011: When a cleaning task is received, semantic information in the cleaning task is identified.

[0069] Among them, the cleaning task is the task of controlling the cleaning robot to clean a designated area.

[0070] The cleaning task can be a preset task for the cleaning robot (such as the cleaning robot automatically cleans each room in the house every day), or it can be a user-defined setting (such as the area near the table is dirty during user activities, and the area needs to be cleaned).

[0071] Optionally, the cleaning robot can be communicatively connected with an electronic device, and the electronic device may be provided with an application for controlling the cleaning robot. Cleaning tasks can be created through the application, thereby customizing the cleaning parameters (such as cleaning area, cleaning time, cleaning mode, etc.).

[0072] For example, users can input semantic information in the application interface and customize the cleaning area (such as the door, under the bed, carpet, etc.) to generate a cleaning task. For another example, users can quickly create a cleaning task through voice commands. For example, if a user speaks "clean the door", the cleaning robot and / or electronic device can perform voice recognition to generate a cleaning task with the semantic information corresponding to the voice.

[0073] After the application generates a cleaning task, the application sends the cleaning task to the cleaning robot, and the cleaning robot can receive the cleaning task. Alternatively, the cleaning robot can generate a cleaning task when receiving a voice, and at this time, the cleaning robot is deemed to have received the cleaning task.

[0074] After the cleaning robot receives a cleaning task, in order to execute it accurately, it needs to first perform semantic recognition on the cleaning task, thereby determining the semantic information of the cleaning task.

[0075] Step 012: In the semantic map, determine a target area whose semantic information matches the semantic information of the cleaning task.

[0076] Specifically, after the semantic information of the cleaning task is acquired, semantic matching may be performed based on the semantic map. For example, in the semantic map, a map area whose semantic information matches the semantic information of the cleaning task is determined as a target area.

[0077] For example, the semantic information of each area in the semantic map is matched with the semantic information of the cleaning task. When the semantic information of any area (or grid) matches the semantic information of the cleaning task, the area (or grid) is determined to be at least a part of the target area, thereby determining one or more map areas (or map grids) that match the semantic information of the cleaning task as the target area.

[0078] Optionally, the matching of semantic information of the region and semantic information of the cleaning task may be matching of object type information of the region and object type of the object to be cleaned by the cleaning task.

[0079] For example, the semantic information of the cleaning task is "clean the carpet". The carpet is the object type information, and at this time, each area where the carpet is located in the semantic map can be determined as the target area.

[0080] Optionally, the map can be divided into multiple preset partitions. For indoor scenes, each partition can be a divided space, such as a living room, kitchen, bathroom, bedroom, balcony, etc. The semantic information of the area also includes partition information, that is, the partition to which each area belongs. Therefore, the semantic information of the area matches the semantic information of the cleaning task, or the partition to which the area belongs matches the partition information of the cleaning task.

[0081] For example, the semantic information of the cleaning task is "clean the living room". The living room is the partition information, and at this time, each area where the living room is located in the semantic map can be determined as the target area.

[0082] It can be understood that for a map area (grid), it can include not only partition information, but also object type information and contact type information. For example, the area where the sofa in the living room is located contains partition information (living room), object type information (sofa) and contact type information (covered type) at the same time; the ground in the living room where no objects are set only contains partition information.

[0083] See also Figure 7Optionally, the semantic information of the cleaning task includes object type information or object type information and orientation information. Step 012 includes:

[0084] Step 0121: determining a first matching area in the semantic map where semantic information matches object type information;

[0085] Step 0122: Determine the target area based on the first matching area and the orientation information.

[0086] Specifically, since the dirt may be located in the area covered by the object or near the object, when the dirt may be located in the area covered by the object (such as under the table, on the carpet, etc.), the user can specify the corresponding object to be cleaned in the cleaning task (that is, only set the object type information), such as cleaning the table, cleaning the carpet, etc.

[0087] When the dirt is near an object, in order to ensure the accuracy of cleaning, the user can combine the "object + orientation" method (i.e. specify the object type information and orientation information in the cleaning task) to determine the area to be cleaned when setting the cleaning task.

[0088] When determining the target area, the matching area where the semantic information matches the object type information can be first determined in the semantic map, that is, the area where the object in the semantic information of the cleaning task is located. Then, the matching area and the orientation information are combined to accurately determine the target area.

[0089] For example, the user sets the semantic information of the cleaning task as "clean the area between the sofa and the coffee table", so the object type information can be determined as the sofa and the coffee table, and the orientation is between the two. At this time, the areas where the sofa and the coffee table are located are found in the semantic map as matching areas, and then the area between the matching area of ​​the sofa and the matching area of ​​the coffee table is determined as the target area.

[0090] See also Figure 8 Optionally, the semantic information of the cleaning task also includes partition information. Step 012 further includes:

[0091] Step 0123: determining a second matching area in which the semantic information in the semantic map matches the partition information;

[0092] Step 0122: Determine a target area based on at least one of the first matching area and the orientation information, including:

[0093] Step 01221: determining a third matching area based on at least one of the first matching area and the orientation information, wherein the third matching area is located in the second matching area;

[0094] Step 01222: Determine the target area based on the semantic information of the cleaning task, the second matching area and the third matching area.

[0095] Specifically, since different partitions may have the same objects (such as there are tables in the living room and the kitchen), the user can also specify the partition to be cleaned in the cleaning task, thereby combining "partition + object" or "partition + object + orientation" to more accurately determine the area to be cleaned.

[0096] Therefore, we can first determine the second matching area (i.e., the corresponding partition in the semantic map) where the semantic information in the semantic map matches the partition information, and then determine the third matching area (i.e., the area corresponding to the object) in combination with at least one of the orientation information of the first matching area where the object is located. It can be understood that the third matching area is generally located in the second matching area.

[0097] After the second matching area and the third matching area are determined, the target area can be determined in combination with the semantic information of the cleaning task.

[0098] See also Fig. 9 Optionally, step 01222 includes:

[0099] Step 01223: Determine whether to clean the third matching area based on the semantic information of the cleaning task;

[0100] Step 01224: If yes, determine the second matching area as the target area;

[0101] Step 01225: If not, determine that the area in the second matching area except the third matching area is the target area.

[0102] Specifically, after the user specifies the partition and the object in the partition, there may be two cleaning requirements, one is to clean the area corresponding to the object in the partition (ie, the third matching area), and the other is to clean the area in the partition except the area corresponding to the object.

[0103] Therefore, it can be determined whether the third matching area is to be cleaned according to the semantic information of the cleaning task. If yes, it can be determined that the user wants to clean the area corresponding to the object in the partition, and if no, it can be determined that the user wants to clean the area other than the area corresponding to the object in the partition.

[0104] For example, when the user wants to clean an area corresponding to an object in a designated partition, such as cleaning under a table in a living room or a carpet in a living room, the third matching area in the second matching area can be determined as the target area.

[0105] For another example, when the user wants to clean an area other than an area corresponding to an object in a specified partition, such as cleaning an area other than a carpet in a living room, the area other than the third matching area in the second matching area can be determined as the target area.

[0106] In this way, the area that the user wants to clean can be accurately determined based on the semantic information of the cleaning task, with high cleaning accuracy, and targeted cleaning can be achieved according to the different cleaning needs of different users. The richer the semantic information in the semantic map, the richer the cleaning needs that the user can achieve, and the cleaning robot can easily understand and achieve the user's cleaning needs.

[0107] Step 013: Perform a cleaning task based on the target area.

[0108] Specifically, after the target area, ie, the area to be cleaned, is determined, the cleaning robot can perform the cleaning task to clean the target area.

[0109] See also Fig.10 Optionally, step 013 includes:

[0110] Step 0131: In the process of performing the cleaning task, a cleaning path is planned based on the semantic information of each area in the target area;

[0111] Step 0132: Execute the cleaning task based on the cleaning path.

[0112] Specifically, after determining the target area, since the contact type between the object and the ground is different in the target area, the contact type between the object and the ground needs to be considered when planning the cleaning path to avoid collision with the object or to avoid missing the cleanable area. For example, when cleaning, you can bypass obstacle-type objects, pass on the surface of covered objects, pass through suspended objects, etc.

[0113] Afterwards, based on the planned cleaning path, the cleaning robot moves along the cleaning path and performs cleaning to achieve the cleaning task of the target area.

[0114] Optionally, when performing cleaning, the cleaning mode can be adaptively adjusted according to the semantic information of the area to be cleaned. For example, the cleaning mode for a carpet is different from the cleaning mode for a bottom surface without a carpet. When cleaning a carpet, the cleaning mode is adaptively adjusted to improve the cleaning effect of the carpet. In this way, for areas with different semantics, targeted cleaning modes are used for cleaning, which can improve the cleaning effect.

[0115] In the cleaning control method of the present application, the cleaning robot can perform cleaning tasks based on a semantic map that can provide map information and semantic information. Therefore, the user can make full use of the semantic information of the semantic map by setting the semantic information of the cleaning task, thereby generating more diverse cleaning tasks. After receiving the cleaning task, the cleaning robot can match the semantic information in the cleaning task with the semantic information in the semantic map, thereby determining the matching target area (such as the area to be cleaned), so that the area corresponding to the location of any object in the map (such as under the table, at the door, etc.) can be cleaned in a targeted manner, which can meet the changing cleaning needs of different users and provide a good user experience.

[0116] According to the method described in the above embodiment, the present application embodiment also provides a control device 300 for executing the steps in the above cleaning control method. Fig.11 , Fig.11 4 is a schematic diagram of the structure of a generating device 400 provided in an embodiment of the present application. The generating device 400 includes:

[0117] An acquisition module 401 is used to acquire a scene image acquired by an image acquisition device, wherein the scene image at least includes image information of the scene below the height of the radar;

[0118] A determination module 402, used to determine a target object in a scene image and semantic information of the target object;

[0119] The generation module 403 is used to mark each target object in the map through a semantic block based on the semantic information of the target object to generate a semantic map.

[0120] According to the method described in the above embodiment, the present application embodiment also provides a control device 300 for executing the steps in the above cleaning control method. Fig.12 , Fig.12 : is a schematic diagram of the structure of a control device 300 provided in an embodiment of the present application. The control device 300 includes:

[0121] The recognition module 301 is used to recognize semantic information in the cleaning task when receiving the cleaning task;

[0122] A matching module 302 is used to determine, in the semantic map, a target area whose semantic information matches the semantic information of the cleaning task;

[0123] The execution module 303 is used to execute the cleaning task based on the target area and the semantic information of the cleaning task.

[0124] It should be noted that the specific details of each module unit in the above-mentioned control device 300 have been described in detail in the embodiment of the above-mentioned cleaning control method, and will not be repeated here.

[0125] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0126] In some embodiments, the control device in the embodiments of the present application can be implemented in hardware, such as a cleaning robot, or a component in the cleaning robot, such as an integrated circuit or a chip; the control device can also be implemented in software, such as as an application installed in the cleaning robot.

[0127] In some embodiments, please refer again to Figure 1 The cleaning robot 100 includes a processor 30 and a memory 40. The memory 40 stores a computer program that can be run on the processor 30. When the program is executed by the processor 30, each process of the embodiment of the above-mentioned cleaning control method is implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0128] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned cleaning control method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0129] The processor may be the processor in the cleaning robot in the above-mentioned embodiment.

[0130] Among them, storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other solid-state storage technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that computer storage media are not limited to the above-mentioned ones.

[0131] The embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned cleaning control method when executed by a processor. The processor may be a processor in the cleaning robot in the above-mentioned embodiment. When the computer program is executed by the processor, each process of the embodiment of the above-mentioned cleaning control method is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0132] It is understandable that in the specific implementation of this application, data related to user identity or characteristics is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0133] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A method for generating a semantic map, It is characterized in that Applied to a cleaning robot, the cleaning robot includes an image acquisition device and a radar, and the method includes: Acquire a scene image acquired by the image acquisition device, wherein the scene image at least includes image information of a scene below the height of the radar; Determining a target object in the scene image and semantic information of the target object; Based on the semantic information of the target objects, each of the target objects is marked in the map by a semantic block to generate a semantic map.

2. The generation method according to claim 1, It is characterized in that The method further comprises: The semantic map is displayed on a terminal that is communicatively connected to the cleaning robot.

3. The generation method according to claim 1, It is characterized in that The semantic information of the target object includes object type information and contact type information with the ground, the contact type information includes at least one of covering type, obstacle type and suspended type. The objects in the area corresponding to the covering type cover the ground and are lower than the obstacle crossing height of the cleaning robot. The objects in the area corresponding to the obstacle type are higher than the obstacle crossing height. The objects in the area corresponding to the suspended type are suspended and are higher than the height of the cleaning robot.

4. The generation method according to any one of claims 1 to 3, It is characterized in that The determining the target object in the scene image and the semantic information of the target object includes: Performing semantic segmentation on the scene map to determine each of the target objects in the scene and semantic information of each of the target objects; The step of marking each of the target objects in a map by using semantic blocks based on the semantic information of the target objects to generate a semantic map includes: Based on the semantic information of the target object and the image coordinates in the scene image, semantic projection is performed to mark the semantic blocks of each target object in the map to generate the semantic map.

5. The generation method according to claim 4, It is characterized in that The semantic map includes a plurality of regions, the semantic block covers one or more of the regions, and the method further includes: Acquire various semantic information of a current region when multiple semantic projections are performed, wherein the current region is any of the regions; Based on the probability distribution of different semantic information in the various semantic information; The semantic information of the current region is updated based on the semantic information with the largest distribution probability.

6. The generation method according to claim 1, It is characterized in that The map is a grid map.

7. The generation method according to claim 4, It is characterized in that One or more grid clusters form a semantic block corresponding to the target object.

8. The generation method according to claim 1, It is characterized in that The image acquisition device includes a monocular camera.

9. A cleaning control method based on semantic map, It is characterized in that The semantic map is generated based on the generation method according to any one of claims 1 to 8, the cleaning control method is used to control a cleaning robot, the semantic map includes a map and semantic information of each area of ​​the map, and the method includes: Upon receiving a cleaning task, identifying semantic information in the cleaning task; In the semantic map, determining a target area whose semantic information matches the semantic information of the cleaning task; The cleaning task is performed based on the target area.

10. The cleaning control method according to claim 9, It is characterized in that The map is a grid map, each grid of the grid map is used as a region, and in the semantic map, determining a target region whose semantic information matches the semantic information of the cleaning task includes: In the semantic map, determining each target grid whose semantic information matches the semantic information of the cleaning task; Based on the target grid, the target area is determined.

11. The cleaning control method according to claim 9, It is characterized in that The semantic information of the cleaning task includes object type information or includes the object type information and orientation information, and in the semantic map, determining a target area whose semantic information matches the semantic information of the cleaning task includes: Determine a first matching area in the semantic map where semantic information matches the object type information; The target area is determined based on at least one of the first matching area and the orientation information.

12. The cleaning control method according to claim 11, It is characterized in that The semantic information of the cleaning task further includes partition information, and in the semantic map, determining a target area whose semantic information matches the semantic information of the cleaning task further includes: Determine a second matching area in the semantic map where semantic information matches the partition information; The determining the target area based on at least one of the first matching area and the orientation information includes: Determine a third matching area based on at least one of the first matching area and the orientation information, wherein the third matching area is located in the second matching area; The target area is determined based on the semantic information of the cleaning task, the second matching area, and the third matching area.

13. The cleaning control method according to claim 12, It is characterized in that The determining the target area based on the semantic information of the cleaning task, the second matching area, and the third matching area includes: determining whether the third matching area is to be cleaned based on the semantic information of the cleaning task; If yes, determining the second matching area as the target area; If not, it is determined that the area in the second matching area except the third matching area is the target area.

14. The cleaning control method according to claim 9, It is characterized in that The semantic information of the cleaning task includes partition information, and determining, in the semantic map, a target area whose semantic information matches the semantic information of the cleaning task includes: Determine a second matching area in the semantic map where semantic information matches the partition information; The target area is determined based on the second matching area.

15. A non-transitory computer readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the generation method or the cleaning control method as described in any one of claims 1 to 14 is implemented.

16. A cleaning robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the generation method or the cleaning control method as described in any one of claims 1 to 14 is implemented.

17. A computer program product, It is characterized in that It comprises a computer program, which, when executed by a processor, implements the generation method or the cleaning control method as described in any one of claims 1 to 14.

Citation Information

Cited By

  • Accurate sensing and positioning method, device and equipment for sweeping robot and medium

    CN120284162A

  • Control method, cleaning robot, cleaning system and computer readable storage medium

    CN121465444A