Map updating method and cleaning robot

Abstract

The invention relates to a map updating method and a cleaning robot. The map updating method comprises the following steps: acquiring an image which is acquired by a camera and comprises a to-be-detected object; in a three-dimensional model library, searching a target object model matched with the to-be-measured object based on the image; determining position information of the to-be-measured object based on radar data acquired by a laser radar and the image; and mapping the target object model to an environment map according to the position information to obtain an updated environment map. By adopting the method, even if the position of the indoor object changes, the accuracy of the environment map can be ensured.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a map updating method and a cleaning robot. Background Art

[0002] With the rapid development of smart homes, robots used for indoor cleaning are widely used. Due to the advantages of automation, high efficiency and intelligence, robots are loved by the majority of users.

[0003] When the robot performs a cleaning task for the first time, it will collect indoor environmental data during the cleaning process, and build a map based on the collected environmental data, so as to plan the path according to the obtained environmental map. However, considering that the position of indoor objects may change, the environmental map may be inaccurate. Summary of the invention

[0004] Based on this, it is necessary to provide a map updating method and a cleaning robot to address the above technical problems, which can ensure the accuracy of the environmental map even if the positions of indoor objects change.

[0005] In a first aspect, the present application provides a map updating method, the method comprising:

[0006] Acquire an image including the object to be measured captured by a camera;

[0007] In the 3D model library, search for a target object model that matches the object to be tested based on the image;

[0008] Determine the position information of the object to be measured based on the radar data and images collected by the laser radar;

[0009] The target object model is mapped to the environment map according to the position information to obtain an updated environment map.

[0010] In one embodiment, searching for a target object model matching the object to be detected in a three-dimensional model library based on an image includes:

[0011] Extracting object features of the object to be tested from the image;

[0012] Acquire text information corresponding to the object to be tested, and extract text features from the text information;

[0013] In the 3D model library, a target object model matching the object to be tested is searched based on object features and text features.

[0014] In one embodiment, extracting the object features of the object to be detected from the image includes:

[0015] Extract local object features and global object features of the object to be tested from the image;

[0016] Obtaining text information corresponding to the object to be tested includes:

[0017] Based on local object features and global object features, identify the category of the object to be tested;

[0018] Get the text information corresponding to the category.

[0019] In one embodiment, searching for a target object model matching the object to be tested in a three-dimensional model library based on object features and text features includes:

[0020] If the category meets the model search conditions, the object features and text features are mapped to the embedding space to obtain the object features and text features of the embedding space;

[0021] The object features and text features in the embedded space are matched with the features of each object model in the 3D model library to obtain the target object model.

[0022] In one embodiment, obtaining an image including the object to be measured captured by a camera includes:

[0023] If the object to be measured is a dynamic object, an image of the object to be measured is acquired by the camera according to the acquisition frequency corresponding to the dynamic object; wherein the acquisition frequency corresponding to the dynamic object is greater than the acquisition frequency corresponding to the non-dynamic object, and the dynamic object is a biological body or other self-moving object;

[0024] The position information of the object to be measured based on the radar data and images collected by the lidar includes:

[0025] Obtain radar data collected by the laser radar according to the collection frequency corresponding to the dynamic object;

[0026] The position information of the object to be detected is determined based on radar data and images.

[0027] In one embodiment, the method further comprises:

[0028] Align the target object model with the object to be measured to obtain an aligned object model;

[0029] Mapping the target object model to the environment map according to the location information includes:

[0030] Map the aligned object model to the environment map according to the position information.

[0031] In one embodiment, aligning the target object model with the object to be measured to obtain the aligned object model includes:

[0032] Extracting key features from the target object model; the key features include at least one of corner points, edge points or geometric features;

[0033] The key features are matched with the object features of the object to be measured, so that the target object model is aligned with the object to be measured, and an aligned object model is obtained.

[0034] In one embodiment, the method further comprises:

[0035] Determine the size information of the object to be measured based on the radar data collected by the laser radar;

[0036] Mapping the aligned object model to the environment map according to the position information includes:

[0037] According to the position information, size information and the pose of the aligned object model, the aligned object model is mapped to the environment map.

[0038] In one embodiment, mapping the aligned object model to the environment map according to the position information, the size information and the pose of the aligned object model includes:

[0039] Processing the aligned object model according to the size information to obtain a processed object model;

[0040] converting the processed object model into ground data according to the pose of the processed object model;

[0041] Update the ground data to the environment map based on the location information.

[0042] In one embodiment, the object to be detected is an object in an uncleaned area of ​​the environment map;

[0043] Updating ground data to the environment map based on location information includes:

[0044] Based on the location information, the ground data is updated to the uncleaned area of ​​the environment map.

[0045] In one embodiment, the environment map includes an uncleanable area; the method further includes:

[0046] Compare the environment map before updating with the environment map after updating to obtain a comparison result;

[0047] If it is determined according to the comparison result that the state of the uncleanable area has changed, the uncleanable area is cleaned separately, or the uncleanable area is included in the cleaning plan of the room to which it belongs, so that the uncleanable area is cleaned when the room is cleaned.

[0048] In a second aspect, the present application further provides a map updating device, the device comprising:

[0049] An acquisition module, used to acquire an image including an object to be measured captured by a camera;

[0050] A search module, used to search for a target object model matching the object to be tested in a three-dimensional model library based on an image;

[0051] A determination module, used to determine the position information of the object to be measured based on the radar data and images collected by the laser radar;

[0052] The mapping module is used to map the target object model into the environment map according to the position information to obtain an updated environment map.

[0053] In one of the embodiments, the search module is also used to extract object features of the object to be tested from the image; obtain text information corresponding to the object to be tested, and extract text features from the text information; and search for a target object model that matches the object to be tested in the three-dimensional model library based on the object features and the text features.

[0054] In one of the embodiments, the search module is further used to extract local object features and global object features of the object to be detected from the image; identify the category of the object to be detected based on the local object features and the global object features; and obtain text information corresponding to the category.

[0055] In one of the embodiments, the search module is also used to map the object features and text features to the embedding space if the category meets the model search conditions, so as to obtain the object features and text features of the embedding space; and match the object features and text features of the embedding space with the features of each object model in the three-dimensional model library to obtain the target object model.

[0056] In one embodiment, the search module is further used to obtain an image of the object to be detected obtained by the camera according to the acquisition frequency corresponding to the dynamic object if the object to be detected is a dynamic object; wherein the acquisition frequency corresponding to the dynamic object is greater than the acquisition frequency corresponding to the non-dynamic object, and the dynamic object is a biological body or other self-moving object;

[0057] The determination module is also used to obtain radar data collected by the laser radar according to the collection frequency corresponding to the dynamic object; and determine the position information of the object to be measured based on the radar data and the image.

[0058] In one embodiment, the device further comprises:

[0059] An alignment module is used to align the target object model with the object to be measured to obtain an aligned object model;

[0060] The mapping module is also used to map the aligned object model into the environment map according to the position information.

[0061] In one embodiment, the alignment module is further used to extract key features from the target object model; the key features include at least one of corner points, edge points or geometric features; the key features are matched with the object features of the object to be measured to align the target object model with the object to be measured to obtain an aligned object model.

[0062] In one embodiment, the device further comprises:

[0063] The determination module is also used to determine the size information of the object to be measured based on the radar data collected by the laser radar;

[0064] The mapping module is also used to map the aligned object model to the environment map according to the position information, size information and the pose of the aligned object model.

[0065] In one of the embodiments, the mapping module is further used to process the aligned object model according to the size information to obtain a processed object model; convert the processed object model into ground data according to the posture of the processed object model; and update the ground data to the environment map according to the position information.

[0066] In one embodiment, the object to be detected is an object in an uncleaned area of ​​the environment map;

[0067] The mapping module is further used to update the ground data into the uncleaned area of ​​the environment map according to the location information.

[0068] In one embodiment, the environment map includes an uncleanable area; the device further includes:

[0069] A comparison module, used for comparing the environment map before updating with the environment map after updating to obtain a comparison result;

[0070] The cleaning module is used to clean the uncleanable area separately if it is determined according to the comparison result that the state of the uncleanable area has changed, or to include the uncleanable area in the cleaning plan of the room to which it belongs, so that the uncleanable area can be cleaned when the room is cleaned.

[0071] In a third aspect, the present application also provides a cleaning robot, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned map updating method when executing the computer program.

[0072] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the map updating method are implemented.

[0073] In a fifth aspect, the present application also provides a computer program product, the computer program product comprising a computer program, which implements the steps of the map updating method when executed by a processor.

[0074] The above-mentioned map updating method and cleaning robot obtain an image including the object to be measured collected by the camera; in the three-dimensional model library, search for the target object model matching the object to be measured based on the image; determine the position information of the object to be measured based on the radar data and image collected by the lidar; map the target object model to the environmental map according to the position information to obtain an updated environmental map, so that the environmental map can be updated in real time during each cleaning. Even if the position of indoor objects changes, the accuracy of the environmental map can be ensured, which is conducive to improving the accuracy of the robot's path planning during the cleaning process, thereby effectively avoiding collisions with indoor objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 A diagram showing an application environment of a map updating method in an embodiment;

[0076] Figure 2 is a flowchart of a map updating method in one embodiment;

[0077] Figure 3 A schematic diagram of identifying a chair and a two-dimensional boundary of the chair in an image in one embodiment;

[0078] Figure 4 is a schematic diagram of an environment map in one embodiment;

[0079] Figure 5 A schematic diagram showing a target object model displayed on a map page in one embodiment;

[0080] Figure 6 is a flowchart of a map updating method in another embodiment;

[0081] Figure 7 is a structural block diagram of a map updating device in one embodiment;

[0082] Figure 8 is a structural block diagram of a map updating device in another embodiment;

[0083] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0084] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0085] It should be noted that in the following description, the terms "first, second and third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first, second and third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0086] The map updating method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the robot 104 through a network.

[0087] The terminal 102 can initiate a cleaning instruction to the robot 104. The robot 104 responds to the cleaning instruction and performs a cleaning task indoors. In the process of performing the cleaning task, the robot 104 obtains an image including the object to be measured captured by the camera; in the three-dimensional model library, the target object model matching the object to be measured is searched based on the image; the position information of the object to be measured is determined based on the radar data and image collected by the lidar; the target object model is mapped to the environmental map according to the position information to obtain an updated environmental map; thus, path planning can be performed based on the updated environmental map.

[0088] Among them, the terminal 102 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, an Internet of Things device and a portable wearable device. The Internet of Things device can be a smart speaker, a smart TV, a smart air conditioner and a smart car device, etc.

[0089] The robot 104 may be a cleaning robot such as a sweeping robot, a mopping robot, a sweeping and mopping robot, or a vacuuming robot.

[0090] In one embodiment, Figure 2 As shown, a map updating method is provided, which is composed of Figure 1 The robot execution in is taken as an example to illustrate, which includes the following steps:

[0091] S202: Acquire an image including the object to be measured captured by a camera.

[0092] The camera may be a camera installed on the robot, and the number of the camera may be one or more.

[0093] The objects to be tested can be static objects or dynamic objects in the room. Static objects can be objects that do not move automatically. The static objects can be furniture, home appliances, walls and doors; in addition, they can also be small objects, debris on the ground, carpets, trash cans, trash bags, curtains, windows, stairs and height differences, reflective or transparent objects, special areas or landmarks, etc. Among them, furniture can be tables, chairs, sofas, bookshelves and beds, etc.; home appliances can be televisions, refrigerators, washing machines and microwave ovens, etc.

[0094] Dynamic objects can be automatically moving objects, such as people in the room (such as family members or visitors), pets (such as cats, dogs) or other automatically moving objects (such as children's electric toys). By identifying dynamic objects, the robot can avoid them when planning the path, thereby avoiding collision with the dynamic objects. It should be pointed out that for the robot, the above-mentioned objects to be measured are obstacles in the direction of the robot's travel, so the objects to be measured in this application can also be called obstacles.

[0095] In one embodiment, the robot can receive a cleaning instruction sent by the terminal, and then perform a cleaning task indoors based on the cleaning instruction, and obtain an image of the object to be tested captured by the camera in real time during the cleaning task.

[0096] When the camera collects images in real time, different collection frequencies can be used for image collection according to different types of objects in the room.

[0097] The cleaning instruction can be an instruction to clean the whole house, or an instruction to clean a specific room or area.

[0098] For example, when the user needs to clean the master bedroom, he can send a cleaning instruction to the robot through the mobile phone. After receiving the cleaning instruction, the robot cleans the master bedroom and obtains the image of the object to be tested captured by the camera in real time during the cleaning process.

[0099] In one embodiment, if the object to be measured is a dynamic object, the camera acquires an image of the object to be measured according to the acquisition frequency corresponding to the dynamic object; wherein the acquisition frequency corresponding to the dynamic object is greater than the acquisition frequency corresponding to the non-dynamic object, and the dynamic object is a biological body or other self-moving object. The self-moving object includes an object that can be moved (such as a table, a chair), and an object that moves itself (such as a cleaning robot, etc.).

[0100] For example, if the images collected by the robot during the cleaning task include dynamic objects (such as family members), in order to avoid collisions caused by the movement of dynamic objects, a high acquisition frequency can be used during image acquisition. This allows the robot to track and predict the movement trajectory of dynamic objects when they move indoors, update the map in time, and make path planning quickly to avoid collisions between the robot and dynamic objects during the cleaning process.

[0101] In one embodiment, after obtaining the image captured by the camera, the robot can also make a confidence judgment on the image. If the confidence of the image is less than the preset confidence, the image capture continues; if the confidence of the image is greater than or equal to the preset confidence, S204 can be executed.

[0102] In another embodiment, after obtaining the image captured by the camera, the robot can also determine whether the object to be measured in the currently obtained image is the same object as the object in the image captured at other times when performing the cleaning task. If so, it can be determined that the object to be measured is a movable object, and S204 can be executed at this time.

[0103] Among them, objects can be divided into movable objects and immovable objects according to their properties. Movable objects can be divided into frequently moving objects and infrequently moving objects according to their movement frequency.

[0104] Frequently moving objects can be dynamic objects that move automatically, such as people in the room (including family members and visitors) and electronic toys, etc. Infrequently moving objects can be objects that can be moved by external forces (static objects, such as furniture or household appliances). Immovable objects can be objects that cannot be moved even by external forces, such as walls.

[0105] For frequently moving objects, the corresponding collection frequency is greater than the collection frequency of infrequently moving objects, thereby increasing the detection frequency of frequently moving objects and adjusting the cleaning path in time to cope with the ever-changing environment.

[0106] S204, searching the three-dimensional model library for a target object model that matches the object to be measured based on the image.

[0107] The 3D model library may be a model library composed of various 3D object models. The 3D model library may be provided with corresponding directories. When searching, the matching target object model may be searched under the corresponding directories of the 3D model library. For example, the directories may be living rooms, bedrooms, dining rooms, and bathrooms. The object models in the 3D model library may be models formed by 3D modeling of various objects in the room. The target object model may be an object model in the 3D model library that matches the object to be tested.

[0108] In one embodiment, the robot can extract the object features of the object to be measured from the image, and then search for a target object model matching the object to be measured in the three-dimensional model library based on the object features.

[0109] The object feature may be a visual feature of the image about the object to be measured, and may include local object features and global object features. Local object features may include geometric features (such as shape, including round and square), texture and color of the object to be measured, and may also include corners and edges of the object to be measured. Global object features may include the overall structure of the object to be measured.

[0110] For example, the robot can use deep learning models (such as the YOLO model) to extract object features such as the shape, texture, color, and overall structure of the object to be tested from the image. The object features can be high-dimensional vectors, and then search for the target object model that matches the object to be tested in the three-dimensional model library based on the object features.

[0111] In one embodiment, the robot extracts object features of the object to be measured from the image; obtains text information corresponding to the object to be measured, and extracts text features from the text information; and searches for a target object model matching the object to be measured in a three-dimensional model library based on the object features and the text features.

[0112] The text information may be information used to describe the object to be tested. For example, if the object to be tested is a chair, the text information may be a chair with four legs or a five-legged ergonomic chair, or a chair with a backrest, etc. The text information may be preset in the form of a script, and may be obtained according to the object to be tested when necessary.

[0113] When extracting text features, the robot can use the natural language processing (NLP) model for encoding, converting text information into a high-dimensional vector representation to obtain text features.

[0114] In one embodiment, the robot responds to an information selection operation, selects first text information corresponding to the object to be tested from the candidate text information, and extracts text features from the first text information; when performing a model search, the robot can first search for matching candidate object models in the three-dimensional model library based on object features and text features of the first text information, and the number of the candidate object models can be multiple; then, in response to a second information selection operation, the robot selects second text information corresponding to the object to be tested from the candidate text information, extracts text features from the second text information, and then selects a matching object model from the candidate object models as the target object model based on the object features and the text features of the second text information.

[0115] Among them, the first text information can be text information selected according to the category of the object to be tested. For example, if the category of the object to be tested is a chair, the first text information can be a chair with four legs; the second text information can be further screened out based on the first text information, such as having a backrest (that is, the chair has a backrest).

[0116] In one embodiment, the robot can extract local object features and global object features of the object to be tested from the image; identify the category of the object to be tested based on the local object features and the global object features; and obtain text information corresponding to the category.

[0117] For example, the robot uses the YOLO model to extract the shape, texture, color, and overall structure of the object to be tested from the image, and then identifies the category of the object to be tested based on these object features. For example, if the object to be tested is a chair, the two-dimensional boundary of the object to be tested can be generated, such as Figure 3 As shown, on the other hand, text information corresponding to the chair can be obtained, such as a four-legged chair or a five-legged ergonomic chair, thereby obtaining a semantically rich information description (Prompt).

[0118] In one embodiment, if the category meets the model search conditions, the robot maps the object features and text features to the embedded space to obtain the object features and text features of the embedded space; the object features and text features of the embedded space are matched with the features of each object model in the three-dimensional model library to obtain the target object model, so that even if the robot only captures a part of the object to be tested through the image, the object to be tested can be completed by matching and searching. In other words, even if the camera can only capture a part of the object to be tested, the part that the camera cannot capture can be completed in the environment map by matching and searching.

[0119] In order to achieve effective matching between image and text information, the features corresponding to the image and text information are mapped to a common embedding space. In this embedding space, the features corresponding to the image and text information are as close as possible, so that they can be retrieved through the contrastive learning method. Contrastive learning can be: in the CLIP model, the features corresponding to the image and text information are mapped to the same embedding space, and the contrast loss function (such as InfoNCE loss) is used to minimize the distance between images and text information with the same content, and maximize the distance between unrelated images and text information. In this embedding space, if the text information matches the content of the image, then the distance between their features in the embedding space will be very close.

[0120] For example, the robot can use multimodal feature learning technology to convert object features and text features into vector representations embedded in the space, and use a vector retrieval algorithm to search in the three-dimensional model library based on the vector representation, such as matching the vector representation with the features of the object model stored in the three-dimensional model library to obtain a target object model that matches the object to be tested. The target object model can accurately obtain the ground projection of the entire obstacle, helping the robot to plan a more accurate path.

[0121] The vector retrieval algorithm may be a k-NN (k-nearest neighbors algorithm), a Faiss algorithm for efficiently searching for a vector most similar to a query vector in a large vector set, or a HNSW (Hierarchical Navigable Small World), where HNSW is an efficient approximate nearest neighbor search algorithm.

[0122] In one embodiment, if a target object model that matches the object to be measured cannot be found in the three-dimensional model library, a three-dimensional target object model is generated based on the object features of the object to be measured, and the target object model is updated to the three-dimensional model library, so that the next time an image of the object to be measured is acquired, a target object model that matches the object to be measured can be found in the three-dimensional model library.

[0123] S206, determining the position information of the object to be measured based on the radar data and images collected by the laser radar.

[0124] The radar data can be a radar point cloud (also known as a laser point cloud) obtained by a laser radar emitting laser light and receiving the reflected laser light. The radar data can provide depth data, which can be used to indicate the distance between the robot and each area or point in the object to be measured.

[0125] The location information may be the location coordinates of the object to be detected in the environment map. The environment map may be a two-dimensional radar map, such as Figure 4 shown.

[0126] It should be pointed out that the radar data can be a radar point cloud obtained by the laser radar emitting a horizontal laser and receiving the reflected laser; it can also be a radar point cloud obtained by the laser radar emitting a laser at a certain angle on a vertical horizontal line and receiving the reflected laser.

[0127] In one embodiment, the robot obtains radar data collected by the laser radar at a collection frequency corresponding to the dynamic object; and determines the position information of the object to be measured based on the radar data and the image.

[0128] When collecting radar data, the collection frequency corresponding to dynamic objects is greater than the collection frequency corresponding to non-dynamic objects. For example, when it is detected that the object to be measured is a moving electric toy, the collection frequency is increased to collect radar data.

[0129] For example, the radar data collected by the lidar can be used to add depth information to each identified object, thereby helping the sweeping robot determine the precise location information of the object to be measured; in addition, it can also help the sweeping robot determine the precise shape of the object to be measured.

[0130] In one embodiment, after finding the target object model, the robot can display the target object model on a relevant page of the client. For example, the robot sends the target object model to the client, so that the client displays the target object model on a corresponding model page. In addition, the target object model can also be displayed on a corresponding map page. Figure 5 , thereby improving the realism and facilitating user understanding. The client may be a terminal (such as a mobile phone) that establishes a communication connection with the robot, and the user may adjust the orientation (such as direction and position) of the target object model on the display page.

[0131] S208, mapping the target object model into the environment map according to the position information to obtain an updated environment map.

[0132] The environment map may be a two-dimensional radar map historically constructed by the robot, and the environment map may be updated each time a cleaning task is performed.

[0133] In one embodiment, the robot can determine the size information of the object to be measured based on the radar data collected by the laser radar; and map the target object model into the environment map according to the position information and size information.

[0134] Among them, for this cleaning task (i.e., cleaning plan), the environment map may include cleaned areas and uncleaned areas. When updating the environment map, the uncleaned areas in the environment map may be updated. For example, in the same cleaning task, the cleaned areas and uncleaned areas are distinguished. For the uncleaned areas, the environment map is updated, and for the cleaned areas, the environment map is not updated. For example, when the robot passes through the cleaned area, the cleaned area in the environment map will not be updated.

[0135] Specifically, the robot can process the target object model according to the size information to obtain a processed object model; convert the processed object model into ground data according to the posture of the processed object model; update the ground data to the environment map according to the position information; wherein, when updating, the ground data can be updated to the uncleaned area of ​​the environment map according to the position information.

[0136] For example, the target object model is scaled according to the size information so that the size of the target object model is consistent with the size of the object to be measured. For example, if the length and width of the object to be measured are both 20 centimeters (cm), then after scaling, the length and width of the target object model are also 20 cm. Therefore, after updating to the environmental map, the problem of path planning errors caused by inconsistent sizes can be avoided. Then, the ground data is updated to the uncleaned area of ​​the environmental map according to the location information.

[0137] In one embodiment, before updating the environment map, the robot may also align the target object model with the object to be measured to obtain an aligned object model; and map the aligned object model to the environment map according to the position information.

[0138] The alignment process may specifically include: the robot extracts key features from the target object model; the key features include at least one of corner points, edge points or geometric features; the key features are matched with the object features of the object to be measured to align the target object model with the object to be measured to obtain an aligned object model.

[0139] Among them, corner points can be the endpoints of the target object model, such as the corner points of a chair. Edge points can be points on the edge of the target object model, such as the points on the edge line of a chair. Geometric features can be features used to describe the shape of the target object model.

[0140] For example, the 6DOF posture of the object to be tested is inferred by matching the object features of the image with the key features of the target object model using a geometric algorithm. Specifically, a deep learning model (such as a convolutional neural network (CNN)) is used to detect the key points of the object to be tested in the image, and then a geometric algorithm is used to match the pose of the object to be tested with the target object model, such as matching the key points of the object to be tested with the key points of the target object model, so that the object to be tested is aligned with the target object model. Among them, the above-mentioned key points can be corner points, edge points or specific geometric features (such as a circle or a square, etc.). After the target object model is aligned with the object to be tested, on the one hand, the aligned target object model can be obtained, and on the other hand, the 6DOF pose of the object in the camera coordinate system can be estimated, that is, the position (Translation) and orientation (Rotation) of the object to be tested.

[0141] In one embodiment, the robot can also determine the size information of the object to be measured based on the radar data collected by the laser radar; and map the aligned object model to the environment map according to the position information, size information and the posture of the aligned object model.

[0142] Specifically, the robot processes the aligned object model according to the size information to obtain the processed object model; converts the processed object model into ground data according to the posture of the processed object model; and updates the ground data to the environment map according to the position information.

[0143] The object to be detected is an object in an uncleaned area in the environment map.

[0144] For example, the aligned object model is scaled according to the size information so that the size of the processed object model is consistent with the size of the object to be measured. For example, if the length and width of the object to be measured are both 20 cm, then after scaling, the length and width of the processed object model are also 20 cm. Therefore, after updating to the environment map, the problem of path planning errors caused by inconsistent sizes can be avoided; then, the processed object model is converted into ground data, and the ground data is updated to the uncleaned area of ​​the environment map according to the location information.

[0145] For this cleaning task, considering that the environmental map includes cleaned areas and uncleaned areas, when the environmental map is updated, the uncleaned areas in the environmental map can be updated. Therefore, the robot can update the ground data to the uncleaned areas of the environmental map based on the location information.

[0146] In one embodiment, after completing the update of the environmental map, the robot can perform path planning according to the updated environmental map to obtain the currently planned cleaning path, so as to perform the cleaning task according to the cleaning path.

[0147] Among them, the cleaning task may be to clean the indoor floor (such as floor tiles and wooden boards) or carpet. The floor cleaning may be sweeping, mopping or vacuuming; the carpet cleaning may be to adjust the robot's cleaning mop to a smooth surface to prevent the mop from contaminating the carpet, and then sweep or vacuum the carpet.

[0148] In one embodiment, the robot can also update the target area of ​​the environment map, and correspondingly, when performing image acquisition, the object to be detected is also an object in the target area. For example, for some areas with more stains such as kitchens or bathrooms that need special cleaning, multiple cleanings can be performed, and when updating the map, the target area representing the kitchen or bathroom in the environment map is mainly updated.

[0149] As an example, in order to more clearly understand the solution of the present application, the object to be tested is a chair as an example. Figure 3 , Figure 5 and Figure 6 The scheme of this application is introduced as follows:

[0150] like Figure 6As shown in the figure, when the robot performs the cleaning task in the living room, it obtains an image containing a chair captured by the camera. The image can be referred to as Figure 3 ; Use a deep learning model to recognize the image, such as using a deep learning model to extract features of a chair in the image, and perform recognition based on the extracted features, thereby identifying the chair in the image.

[0151] Taking into account the viewing angle of the camera, the image may only contain part of the structure of the chair, such as only one foot of the chair (i.e., the chair leg) in the image. After identifying it as a chair, the text information can be obtained. According to the text features of the text information and the object features of the object to be tested, the chair model that matches the chair can be searched in the 3D model library. Therefore, even if only a part of the object to be tested is detected, the entire chair model can be obtained by searching.

[0152] The position information of the chair is determined based on the radar data and images collected by the laser radar; in addition, the chair model is aligned with the chair data, and then the chair model is processed according to the radar data and images to obtain a chair model that matches the size ratio of the chair, and then the chair model is converted into ground data. Figure 5 As shown, the four legs of the chair model are converted into ground data, that is, the ground area occupied by the four legs in the environment map, so as to achieve feature completion of the chair, that is, when one leg of the chair is collected, the other three legs can also be completed.

[0153] The ground data is updated to the environment map according to the position information. Therefore, even if the camera only obtains one leg of the chair, the other legs of the chair can be completed on the environment map to obtain an updated environment map so that the robot can plan the path more accurately and thus perform obstacle avoidance operations accurately.

[0154] In the above embodiment, an image including the object to be measured is acquired by the camera; in the three-dimensional model library, a target object model matching the object to be measured is searched based on the image; the position information of the object to be measured is determined based on the radar data and image collected by the lidar; the target object model is mapped to the environmental map according to the position information to obtain an updated environmental map, so that the environmental map can be updated in real time each time cleaning is performed. Even if the position of indoor objects changes, the accuracy of the environmental map can be ensured, which is beneficial to improving the accuracy of the robot's path planning during the cleaning process, thereby effectively avoiding collisions with indoor objects.

[0155] In one embodiment, a new uncleanable area can be added, that is, the environment map includes the uncleanable area; therefore, the method also includes: the robot compares the environment map before the update with the environment map after the update to obtain a comparison result; if it is determined according to the comparison result that the state of the uncleanable area has changed, the uncleanable area is cleaned separately, or the uncleanable area is included in the cleaning plan of the room to which it belongs, so that the uncleanable area can be cleaned when the room is cleaned.

[0156] The uncleanable area may be an area that cannot be cleaned, such as an area that is blocked by movable objects and cannot be reached by the robot, making it impossible to clean. For example, an area in a restaurant is blocked by a dining table. In this case, the blocked area is an uncleanable area. If the dining table is moved to another area during the cleaning process, the state of the uncleanable area changes, that is, from an uncleanable area to a cleanable area.

[0157] In actual application scenarios, when cleaning is sweeping, the uncleanable area can be called an unsweeping area; when cleaning is mopping, the uncleanable area can be called an unmopping area; when cleaning is vacuuming, the uncleanable area can be called an undustable area.

[0158] For example, the robot can compare the environmental map before and after the update to obtain the changes before and after the update, so as to determine the disappeared unsweepable area, so that the area can be cleaned separately, or the area can be included in the cleaning plan of the room to which it belongs and cleaned together.

[0159] In the above embodiment, by adding an uncleanable area, when performing a cleaning task, the uncleanable area may not be cleaned, and when the state of the uncleanable area changes, the uncleanable area may be cleaned, which is beneficial to improving cleaning efficiency.

[0160] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0161] Based on the same inventive concept, the embodiment of the present application also provides a map updating device for implementing the map updating method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more map updating device embodiments provided below can refer to the limitations on the map updating method above, and will not be repeated here.

[0162] In one embodiment, Figure 7 As shown, a map updating device is provided, including: an acquisition module 702, a search module 704, a determination module 706 and a mapping module 708, wherein:

[0163] An acquisition module 702 is used to acquire an image including an object to be measured captured by a camera;

[0164] A search module 704 is used to search for a target object model matching the object to be tested in the three-dimensional model library based on the image;

[0165] A determination module 706 is used to determine the position information of the object to be measured based on the radar data and images collected by the laser radar;

[0166] The mapping module 708 is used to map the target object model into the environment map according to the position information to obtain an updated environment map.

[0167] In one embodiment, the search module 704 is also used to extract object features of the object to be tested from the image; obtain text information corresponding to the object to be tested, and extract text features from the text information; and search for a target object model that matches the object to be tested in the three-dimensional model library based on the object features and the text features.

[0168] In one embodiment, the search module 704 is further used to extract local object features and global object features of the object to be detected from the image; identify the category of the object to be detected based on the local object features and the global object features; and obtain text information corresponding to the category.

[0169] In one of the embodiments, the search module 704 is also used to map the object features and text features to the embedding space if the category meets the model search conditions to obtain the object features and text features of the embedding space; match the object features and text features of the embedding space with the features of each object model in the three-dimensional model library to obtain the target object model.

[0170] In one embodiment, the search module 704 is further used to obtain an image of the object to be detected obtained by the camera according to the acquisition frequency corresponding to the dynamic object if the object to be detected is a dynamic object; wherein the acquisition frequency corresponding to the dynamic object is greater than the acquisition frequency corresponding to the non-dynamic object, and the dynamic object is a biological body or other self-moving object;

[0171] The determination module 706 is also used to obtain radar data collected by the laser radar according to the collection frequency corresponding to the dynamic object; and determine the position information of the object to be measured based on the radar data and the image.

[0172] In one embodiment, Figure 8 As shown, the device also includes:

[0173] An alignment module 710 is used to align the target object model with the object to be measured to obtain an aligned object model;

[0174] The mapping module 708 is further configured to map the aligned object model into the environment map according to the position information.

[0175] In one embodiment, the alignment module 710 is also used to extract key features from the target object model; the key features include at least one of corner points, edge points or geometric features; the key features are matched with the object features of the object to be measured to align the target object model with the object to be measured to obtain an aligned object model.

[0176] In one embodiment, the determination module 706 is further used to determine the size information of the object to be measured based on the radar data collected by the laser radar;

[0177] The mapping module 708 is further used to map the aligned object model into the environment map according to the position information, size information and the pose of the aligned object model.

[0178] In one embodiment, the mapping module 708 is further used to process the aligned object model according to the size information to obtain a processed object model; convert the processed object model into ground data according to the posture of the processed object model; and update the ground data to the environment map according to the position information.

[0179] In one embodiment, the object to be detected is an object in an uncleaned area of ​​the environment map;

[0180] The mapping module 708 is further configured to update the ground data to the uncleaned area of ​​the environment map according to the location information.

[0181] In the above embodiment, an image including the object to be measured is acquired by the camera; in the three-dimensional model library, a target object model matching the object to be measured is searched based on the image; the position information of the object to be measured is determined based on the radar data and image collected by the lidar; the target object model is mapped to the environmental map according to the position information to obtain an updated environmental map, so that the environmental map can be updated in real time each time cleaning is performed. Even if the position of indoor objects changes, the accuracy of the environmental map can be ensured, which is beneficial to improving the accuracy of the robot's path planning during the cleaning process, thereby effectively avoiding collisions with indoor objects.

[0182] In one embodiment, the environment map includes uncleanable areas; Figure 8 As shown, the device also includes:

[0183] A comparison module 712, used to compare the environment map before updating with the environment map after updating to obtain a comparison result;

[0184] The cleaning module 714 is used to clean the uncleanable area separately if it is determined according to the comparison result that the state of the uncleanable area has changed, or to include the uncleanable area in the cleaning plan of the room to which it belongs, so that the uncleanable area can be cleaned when the room is cleaned.

[0185] In the above embodiment, by adding an uncleanable area, when performing a cleaning task, the uncleanable area may not be cleaned, and when the state of the uncleanable area changes, the uncleanable area may be cleaned, which is beneficial to improving cleaning efficiency.

[0186] Each module in the above-mentioned map updating device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0187] In one embodiment, a computer device is provided. The computer device may be a cleaning robot, and its internal structure diagram may be as follows: Fig. 9As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store images and radar data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a map updating method is implemented.

[0188] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0189] In one embodiment, a cleaning robot is provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned map updating method when executing the computer program.

[0190] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned map updating method are implemented.

[0191] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the above map updating method when executed by a processor.

[0192] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0193] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0194] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0195] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A map updating method, characterized in that: The map updating method comprises: Acquire an image including the object to be measured captured by a camera; In a three-dimensional model library, searching for a target object model matching the object to be measured based on the image; Determine the position information of the object to be measured based on the radar data collected by the laser radar and the image; The target object model is mapped into an environment map according to the position information to obtain an updated environment map.

2. The map updating method according to claim 1, characterized in that: The step of searching the target object model matching the object to be detected in the three-dimensional model library based on the image comprises: Extracting local object features and global object features of the object to be measured from the image; Based on the local object features and the global object features, identifying the category of the object to be detected, and acquiring text information corresponding to the category; In the three-dimensional model library, a target object model matching the object to be measured is searched based on the object features and the text features of the text information.

3. The map updating method according to claim 2, characterized in that: The step of searching, in the three-dimensional model library, a target object model matching the object to be detected based on the object features and the text features of the text information comprises: If the category satisfies the model search condition, mapping the object feature and the text feature to an embedding space to obtain the object feature and the text feature of the embedding space; The object features and the text features in the embedding space are matched with the features of each object model in the three-dimensional model library to obtain a target object model.

4. The map updating method according to claim 1, characterized in that: The step of obtaining an image of the object to be measured collected by a camera comprises: If the object to be measured is a dynamic object, an image of the object to be measured is acquired by the camera according to the acquisition frequency corresponding to the dynamic object; wherein the acquisition frequency corresponding to the dynamic object is greater than the acquisition frequency corresponding to the non-dynamic object, and the dynamic object is a biological body or other self-moving object; The determining the position information of the object to be measured based on the radar data collected by the laser radar and the image includes: Acquire radar data collected by the laser radar according to the collection frequency corresponding to the dynamic object; The position information of the object to be detected is determined based on the radar data and the image.

5. The map updating method according to any one of claims 1 to 4, characterized in that: Mapping the target object model to the environment map according to the position information includes: Aligning the target object model with the object to be measured to obtain an aligned object model; The aligned object model is mapped into an environment map according to the position information.

6. The map updating method according to claim 5, characterized in that: The aligning the target object model with the object to be measured to obtain an aligned object model comprises: Extracting key features from the target object model; the key features include at least one of corner points, edge points or geometric features; The key features are matched with the object features of the object to be measured, so that the target object model is aligned with the object to be measured, and an aligned object model is obtained.

7. The map updating method according to claim 5, characterized in that: Mapping the aligned object model to the environment map according to the position information includes: Determine the size information of the object to be measured based on radar data collected by the laser radar; Processing the aligned object model according to the size information to obtain a processed object model; Converting the processed object model into ground data according to the pose of the processed object model; The ground data is updated into an environment map according to the position information.

8. The map updating method according to claim 7, characterized in that: If the object to be detected is an object in an uncleaned area in the environmental map, then updating the ground data to the environmental map according to the position information includes: The ground data is updated into the uncleaned area of ​​the environment map according to the position information.

9. The map updating method according to any one of claims 1 to 4, characterized in that: If the environment map contains uncleanable areas, obtaining the updated environment map further includes: Comparing the environment map before updating with the environment map after updating to obtain a comparison result; If it is determined according to the comparison result that the state of the uncleanable area has changed, the uncleanable area is cleaned separately, or the uncleanable area is included in the cleaning plan of the room to which it belongs, so that the uncleanable area is cleaned when the room is cleaned.

10. A cleaning robot, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, each step of the map updating method according to any one of claims 1 to 9 is implemented.

Images