Floor sweeping robot mapping positioning method and device, related equipment and computer program product

Through the encoding and fusion of digital cameras and TOF cameras, the neural network model is used to predict the position and environmental point cloud information of the sweeping robot, which solves the problems of increased fuselage height and high cost caused by radar modules in the existing technology, and realizes radar-free The module's sweeping robot map construction and positioning functions.

CN118936455APending Publication Date: 2024-11-12IFLYTEK CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411151217.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Existing sweeping robots rely on radar modules for mapping and positioning, resulting in increased fuselage height, inability to pass low space and high cost.

Method used

Digital cameras and TOF cameras are used to replace radar modules, and the plane image and point cloud data are obtained for encoding and fusion, and the robot positioning and environmental point cloud information are predicted using the pre-trained target neural network model to achieve synchronous map construction and positioning.

Benefits of technology

The height of the robot's fuselage is reduced, the traffic capacity to various terrain spaces is improved, and the hardware cost of equipment is reduced, and the mapping and positioning functions without radar modules are realized.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118936455A_ABST
    Figure CN118936455A_ABST
Patent Text Reader

Abstract

The invention discloses a floor mopping robot mapping positioning method and device, related equipment and a computer program product, a radar does not need to be arranged on a floor mopping robot, only a digital camera and a TOF camera are arranged, and the setting positions of the digital camera and the TOF camera meet the condition that a union set of view field ranges of the digital camera and the TOF camera in the horizontal direction is not smaller than a set view field value. The set field-of-view value is the minimum field-of-view value of the surrounding environment required by mapping. Based on an improved sweeping robot, a digital camera is used for shooting a plane image, a TOF camera is used for shooting point cloud data, coding and feature fusion are carried out on the plane image and the point cloud data respectively, the current pose of the sweeping robot and environment point cloud information at a set height are predicted based on fused features, and therefore the functions of mapping and positioning can be synchronously carried out. According to the scheme, a radar at the top of the sweeping robot can be omitted, the height of the robot body is reduced, the passing capacity of the sweeping robot to various terrain spaces is improved, and the equipment hardware cost can be further reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of machine mapping and positioning, and more specifically, to a method, device, related equipment and computer program product for mapping and positioning of a sweeping robot. Background Art

[0002] A sweeping robot refers to an intelligent device with autonomous movement and cleaning functions. Existing sweeping machines are generally equipped with a radar module on the top of the machine. The radar module generally uses a single-line radar, which can scan point cloud data in a fixed-height environment, and then use its own algorithm to process and analyze the collected data, thereby locating the robot's own position and building a map of the surrounding environment, commonly known as SLAM (Simultaneous Localization and Mapping).

[0003] The current mapping and positioning process of sweeping robots relies on radar. However, if the radar is set on the top of the body, the height of the sweeping robot will increase, making some low spaces inaccessible. In addition, the cost of the radar module is also high. Summary of the invention

[0004] In view of the above problems, this application is proposed to provide a method, device, related equipment and computer program product for mapping and positioning of a sweeping robot, so as to realize the mapping and positioning functions of the sweeping robot without relying on radar. The specific scheme is as follows:

[0005] In a first aspect, a method for mapping and positioning a sweeping robot is provided, wherein the sweeping robot is provided with at least one digital camera and at least one TOF camera for collecting planar images, and the setting positions of the digital camera and the TOF camera on the sweeping robot satisfy that the union of the field of view ranges of the two in the horizontal direction is not less than a set field of view value, and the method comprises:

[0006] Acquire a planar image of the environment in which the cleaning robot is located taken by the digital camera, and acquire point cloud data of the environment in which the cleaning robot is located taken by the TOF camera;

[0007] Encoding the plane image and the point cloud data respectively, and fusing the encoded features to obtain fused features;

[0008] Predicting the current posture of the sweeping robot and the environmental point cloud information at a set height based on the fusion features, wherein the set height is the height of the digital camera and the TOF camera within the vertical field of view;

[0009] Mapping and positioning are performed synchronously based on the current posture of the robot and the environmental point cloud information at a set height.

[0010] In a possible design, in another implementation of the first aspect of the embodiment of the present application, obtaining the fusion feature and predicting the current posture of the sweeping robot and the environmental point cloud information at a set height based on the fusion feature are implemented by a pre-trained target neural network model;

[0011] The target neural network model is trained using the plane image samples taken by the digital camera and the point cloud data samples taken by the TOF camera as training samples, and using the marked current posture of the sweeping robot and the environmental point cloud information at a set height as sample labels.

[0012] In one possible design, in another implementation of the first aspect of the embodiments of the present application, the target neural network model includes a first encoder, a second encoder, a first output layer, and a second output layer;

[0013] The first encoder is used to encode the planar image to obtain image coding features;

[0014] The second encoder is used to encode the point cloud data to obtain point cloud coding features, and the point cloud coding features are fused with the image coding features as fused features;

[0015] The first output layer is used to predict the current posture of the cleaning robot based on the fusion feature;

[0016] The second output layer is used to predict the environmental point cloud information at a set height based on the fused features.

[0017] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the training process of the target neural network model includes:

[0018] Obtaining plane image samples collected by a digital camera on a sweeping robot prototype, point cloud data samples collected by a TOF camera, and radar data collected by a radar, and using the plane image samples and the point cloud data samples as training samples;

[0019] Based on the radar data, a synchronous mapping and positioning SLAM algorithm is used to calculate the positioning and mapping prediction results of the sweeping robot prototype and their confidence, wherein the positioning and mapping prediction results include the current posture of the sweeping robot prototype and the environmental point cloud information at a set height;

[0020] The positioning and mapping prediction results are screened according to the confidence level, and the prediction results with confidence levels higher than a set confidence threshold are retained as sample labels corresponding to the training samples;

[0021] The target neural network model is trained using the training samples and corresponding sample labels.

[0022] In a possible design, in another implementation of the first aspect of the embodiments of the present application, after training the target neural network model using the training samples and the corresponding sample labels, the method further includes:

[0023] Obtain the plane image samples collected by the digital camera of the sweeping robot prototype in the pre-built truth value system environment, and the point cloud data samples collected by the TOF camera, and obtain the real posture of the sweeping robot through the truth value system;

[0024] The plane image samples and the point cloud data samples acquired in a true value system environment constitute training samples, the real posture is used as a sample label, and the target neural network model is fine-tuned through the training samples and the sample labels to obtain a trained target neural network model.

[0025] In a second aspect, a sweeping robot mapping and positioning device is provided, wherein the sweeping robot is provided with at least one digital camera and at least one TOF camera for collecting planar images, and the setting positions of the digital camera and the TOF camera on the sweeping robot satisfy that the union of the field of view ranges of the two in the horizontal direction is not less than a set field of view value, and the device comprises:

[0026] A sensor data acquisition unit, used to acquire a plane image of the environment in which the sweeping robot is located taken by the digital camera, and to acquire point cloud data of the environment in which the sweeping robot is located taken by the TOF camera;

[0027] A coding fusion unit, used for encoding the plane image and the point cloud data respectively, and fusing the coding features to obtain fusion features;

[0028] A prediction unit, used to predict the current posture of the sweeping robot and the environmental point cloud information at a set height based on the fusion feature, where the set height is the height within the field of view of the digital camera and the TOF camera in the vertical direction;

[0029] A mapping and positioning unit is used to simultaneously perform mapping and positioning based on the current posture of the robot and the environmental point cloud information at a set height.

[0030] In a third aspect, a sweeping robot is provided, comprising:

[0031] Equipment body;

[0032] At least one digital camera and at least one TOF camera are arranged on the device body, and the arrangement positions of the digital camera and the TOF camera on the sweeping robot satisfy that the union of the field of view ranges of the two in the horizontal direction is not less than the set field of view value, the digital camera is used to collect plane images, and the TOF camera is used to collect point cloud data;

[0033] The processor connected to the digital camera and the TOF camera is used to complete the mapping and positioning tasks by adopting the sweeping robot mapping and positioning method described in any one of the first aspects of the present application.

[0034] In a possible design, in another implementation of the third aspect of the embodiment of the present application, at least one of the TOF cameras is disposed on the front side of the sweeping robot, and at least one of the digital cameras is disposed on the rear side of the sweeping robot.

[0035] In one possible design, in another implementation of the third aspect of the embodiments of the present application, the digital camera is an RGB camera, a grayscale camera, or an infrared camera.

[0036] In a fourth aspect, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, each step of the method for mapping and positioning a sweeping robot described in any one of the first aspects of the present application is implemented.

[0037] In a fifth aspect, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the computer program implements the various steps of the sweeping robot mapping and positioning method described in any one of the first aspects of the present application.

[0038] By means of the above technical solution, the present application provides an improved sweeping robot, on which a radar module may not be provided, and only a digital camera and a TOF camera may be provided. The digital camera and the TOF camera are provided at positions that satisfy the union of their field of view ranges in the horizontal direction is not less than a set field of view value, and the set field of view value is the minimum field of view value of the surrounding environment required for mapping. Based on this improved sweeping robot, a mapping and positioning method based on multimodal data is further provided, which obtains a plane image of the environment taken by a digital camera and point cloud data of the environment taken by a TOF camera, encodes the plane image and the point cloud data respectively, fuses the encoded features to obtain fused features, and predicts the current posture of the sweeping robot and the environmental point cloud information at a set height based on the fused features, thereby realizing the simultaneous mapping and positioning functions. The method of the present application reasonably sets the positions of the digital camera and the TOF camera to ensure that the union of their field of view in the horizontal direction is not less than the set field of view value, that is, it can cover most of the surrounding environment. On this basis, the current position of the robot and the environmental point cloud information at the set height can be predicted through the collected plane images and point cloud data, which replaces the function of the radar module in the traditional solution. On the basis of ensuring that the mapping and positioning functions can be completed, the radar on the top of the sweeping robot can be cancelled, thereby reducing the height of the fuselage and improving the sweeping robot's ability to pass through various terrain spaces. In addition, it can further reduce the hardware cost of the equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0040] Figure 1 An example of a top view of a sweeping robot carrying a radar module is shown;

[0041] Figure 2 An example of a front view of a sweeping robot carrying a radar module is shown;

[0042] Figure 3 A front view of a sweeping robot without a radar module provided for the implementation of the present application;

[0043] Figure 4 A rear view of a sweeping robot without a radar module provided for the implementation of the present application;

[0044] Figure 5 A schematic diagram of a flow chart of a method for mapping and positioning a sweeping robot provided in an embodiment of the present application;

[0045] Figure 6 A schematic diagram of the structure of a target neural network model provided in an embodiment of the present application;

[0046] Figure 7 A schematic diagram of a special coding for a truth value system environment and visual positioning provided in an embodiment of the present application;

[0047] Figure 8 This is a schematic diagram of the structure of a sweeping robot mapping and positioning device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0048] Before introducing this application solution, the relevant concepts involved in this article are first explained:

[0049] Sweeping robot: Also known as automatic cleaning robot or intelligent sweeping robot, it is a smart home device specially designed to automatically complete the cleaning work of the floor. It integrates technologies such as microcomputer system, sensor, motor drive system and dust collection system, and can autonomously plan the cleaning path without manual operation or with less manual intervention, and clean and vacuum the floor. Some models even have the function of mopping the floor. Many sweeping robots also support remote control through mobile phone APP, Bluetooth speakers, etc., setting cleaning plans, monitoring cleaning status, etc., which improves the convenience and intelligence level of use.

[0050] SLAM: The full name is Simultaneous Localization and Mapping, which means simultaneous positioning and map building or concurrent mapping and positioning. Its main function is to enable the robot to complete functions such as positioning (Localization) and mapping (Mapping) in an unknown environment.

[0051] Specifically, SLAM technology uses the radar sensor carried by the robot to perceive the surrounding environment in real time, and uses its own algorithm to process and analyze the sensor data, thereby locating the robot's own position and building a map of the surrounding environment. This process is carried out simultaneously, that is, the robot continuously updates its own position information and map information during the movement.

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] Reference Figure 1-Figure 2As shown, the existing sweeping machine is generally provided with a radar module 103 on the top of the fuselage 100. The radar module generally adopts a single-line radar and can scan the point cloud data of a fixed height environment. In addition, a digital camera (such as an RGB camera, an infrared camera, a grayscale camera, etc.) 101 and a TOF camera can also be provided on the front side of the fuselage.

[0054] The sweeping robot can use its own algorithms (such as open source cartography solutions) to process and analyze data collected by radar to locate the robot's own position and build a map of the surrounding environment, which is referred to as mapping and positioning.

[0055] Digital cameras can capture images of the environment in front of them to assist in object recognition, semantic obstacle avoidance and other functions.

[0056] The TOF camera can directly obtain the three-dimensional point cloud of the object in front through active light, which is mainly used to assist in realizing general obstacle avoidance and other functions.

[0057] Since current sweeping robots need to rely on radar for mapping and positioning, installing a radar on the top of the sweeping robot will increase the height of the body, making some low spaces inaccessible for cleaning. In addition, the cost of radar modules is generally high, which will increase the overall equipment cost of the sweeping robot.

[0058] To this end, the present application hopes to provide a sweeping robot that can complete mapping and positioning functions without relying on a radar module. In this way, the radar module on the sweeping robot can be removed, the height of the sweeping robot can be reduced, its ability to pass through various terrain spaces can be improved, and the equipment cost can be further reduced.

[0059] In some embodiments of the present application, Figure 3-Figure 4 As shown, an optional structure of a sweeping robot is provided, which may specifically include:

[0060] Device body 100;

[0061] At least one digital camera 101 and at least one TOF camera 102 are disposed on the device body 100 , and a processor (not shown in the figure) connected to the digital camera 101 and the TOF camera 102 .

[0062] The digital camera 101 is used to collect a planar image of the surrounding environment. It can use a camera capable of capturing planar images, such as an RGB camera, an infrared camera, or a grayscale camera. The corresponding planar image can be an RGB image, an infrared image, or a grayscale image.

[0063] The TOF camera 102 is used to collect point cloud data of the environment.

[0064] The digital camera 101 and the TOF camera 102 are arranged at positions on the cleaning robot such that the union of their field of view ranges in the horizontal direction is not less than a set field of view value.

[0065] The field of view value may be set based on the design objectives of the product and the requirements of the application scenario, and may be the minimum field of view value that satisfies the environmental information required for mapping.

[0066] In order to meet the above-mentioned setting position requirements, an optional setting method is as follows Figure 3 and Figure 4 As shown, at least one TOF camera 102 can be arranged on the front side of the cleaning robot, and at least one digital camera 101 can be arranged on the rear side of the cleaning robot.

[0067] In addition, at least one TOF camera 102 may be disposed on the rear side of the cleaning robot, and at least one digital camera 101 may be disposed on the front side of the cleaning robot.

[0068] Of course, the digital camera 101 and the TOF camera 102 are not strictly limited to being on two opposite sides of the front and back of the body, as long as the setting positions meet the above requirements.

[0069] In addition, the number of digital cameras 101 and TOF cameras 102 can be one or more. For example, if two digital cameras 101 and one TOF camera are included, the two digital cameras 101 can be both arranged on one side of the body (such as on the rear side of the body), or one digital camera 101 is arranged on one side of the body (such as on the rear side of the body), and the other digital camera 101 and the TOF camera are jointly arranged on the other side of the body (such as on the front side of the body).

[0070] The processor on the sweeping robot serves as a control center and can receive the plane image captured by the digital camera 101 and the point cloud data captured by the TOF camera 102. On this basis, the mapping and positioning method designed in this application is adopted to complete the mapping and positioning tasks.

[0071] In some embodiments of the present application, a method for mapping and positioning a sweeping robot is provided. The method can be applied to a sweeping robot (such as applied to a processor on a sweeping robot), or to a system consisting of a sweeping robot and a control terminal thereof, wherein the control terminal can be an intelligent terminal capable of communicating with the sweeping robot, such as a smart phone, a smart speaker, etc.

[0072] The mapping and positioning method for the sweeping robot of the present embodiment can complete the mapping and positioning task without relying on radar. Therefore, the structure of the sweeping robot to which the method is adapted can be as described above, and is provided with at least one digital camera and at least one TOF camera. The setting positions of the digital camera and the TOF camera on the sweeping robot satisfy that the union of their field of view ranges in the horizontal direction is not less than the set field of view value. There is no need to set a radar module on the sweeping robot. Of course, this does not limit the mapping and positioning method for the sweeping robot of the present embodiment to be applicable only to sweeping robots that do not carry radar modules, and it can also be applied to sweeping robots that carry radar modules. Next, in combination with Figure 5 , the mapping and positioning method of the sweeping robot of the present application is introduced, which includes the following steps:

[0073] Step S100: obtaining a planar image of the environment in which the sweeping robot is located taken by a digital camera, and obtaining point cloud data of the environment in which the sweeping robot is located taken by a TOF camera.

[0074] Step S110: Encode the plane image and the point cloud data respectively, and fuse the encoded features to obtain fused features.

[0075] Specifically, the plane image is encoded to obtain image coding features, and the point cloud data is encoded to obtain point cloud coding features. Further, the image coding features and the point cloud coding features are fused, such as splicing the two, to obtain fused features. The fused features contain both the plane image information and the point cloud information.

[0076] Step S120: predicting the current posture of the sweeping robot and the environmental point cloud information at a set height based on the fusion features.

[0077] The set height is the height of the digital camera and the TOF camera within the vertical field of view. Generally, the set height can be selected as the body height of the sweeping robot, or the body height can fluctuate up and down by a certain distance.

[0078] Since the fused features contain both plane images and point cloud information, and the horizontal coverage angle of the two is not less than the set field of view value, it is guaranteed that the two can cover most of the environmental information. Therefore, the current posture of the sweeping robot and the environmental point cloud information at the set height can be predicted based on the fused features, which mentions the function of the radar module.

[0079] Step S130: synchronously perform mapping and positioning based on the current posture of the robot and the environmental point cloud information at the set height.

[0080] Specifically, through the aforementioned steps, the current posture of the robot and the environmental point cloud information at a set height can be obtained without relying on the radar module, and on this basis, the mapping and positioning functions of the sweeping robot can be completed.

[0081] The mapping and positioning method for a sweeping robot provided in an embodiment of the present application can obtain plane images through a digital camera and point cloud data through a TOF camera without relying on a radar module, and predict the current posture of the robot and the environmental point cloud information at a set height based on the multimodal information, thereby completing the mapping and positioning functions of the sweeping robot.

[0082] In some embodiments of the present application, the above steps S110-S120 can be implemented by a pre-trained target neural network model.

[0083] In this embodiment, the target neural network model can be trained using plane image samples taken by a digital camera and point cloud data samples taken by a TOF camera as training samples, and using the labeled current posture of the sweeping robot and the environmental point cloud information at a set height as sample labels.

[0084] Combination Figure 6 As shown, this embodiment provides an optional structure of the target neural network model, which may include: a first encoder, a second encoder, a first output layer, and a second output layer.

[0085] The first encoder is used to encode the planar image to obtain image coding features.

[0086] The second encoder is used to encode the point cloud data to obtain point cloud coding features. The point cloud coding features and image coding features are fused as fused features.

[0087] The first output layer is used to predict the current posture of the sweeping robot based on the fused features.

[0088] The second output layer is used to predict the environmental point cloud information at a set height based on the fused features.

[0089] Among them, the first output layer and the second output layer can be two parallel network structures, or the first output layer and the second output layer can share the same prefix network, such as sharing the same DNN (Deep Neural Network) network, and the outputs of the prefix network are connected to different output layers respectively, which are used to respectively predict the current posture of the sweeping robot, that is, the environmental point cloud information at a set height.

[0090] For the training samples and sample labels during the training of the target neural network model, an optional way to obtain them is as follows:

[0091] A sweeping robot prototype is set up, and the sweeping robot prototype may include a digital camera, a TOF camera, and a radar at the same time. Then, a plane image sample collected by the digital camera on the sweeping robot prototype, a point cloud data sample collected by the TOF camera, and radar data collected by the radar are obtained. The plane image sample and the point cloud data sample constitute a training sample.

[0092] For the radar data collected by the radar, a synchronous mapping and positioning SLAM algorithm (such as a cartography solution) can be used to calculate the positioning and mapping prediction results of the sweeping robot prototype and their confidence. The positioning and mapping prediction results include the current posture of the sweeping robot prototype and the environmental point cloud information at a set height.

[0093] Taking into account possible errors in the SLAM algorithm, in this embodiment, prediction results with a confidence level exceeding a set confidence threshold (such as 0.8 or other values) can be selected from the positioning and mapping prediction results, and other data can be discarded, and the selected prediction results can be used as sample labels corresponding to the training samples.

[0094] The target neural network model is trained according to the training samples and sample labels obtained in the above steps.

[0095] By adopting the above scheme, during the training phase of the target neural network model, we can use the robot vacuum cleaner sample equipped with a radar module, collect radar data through the radar module, and calculate the positioning and mapping prediction results of the robot vacuum cleaner prototype in combination with the traditional SLAM algorithm, and select the prediction results with high confidence as sample labels to guide the model training process. This training strategy can greatly reduce the workload of manually annotating training data.

[0096] Furthermore, after completing the training of the target neural network model using the above training data, in order to further improve the performance of the model, the following process of fine-tuning the accuracy of the target neural network model can be further added.

[0097] Specifically, considering that the above sample labels are screened using confidence, there may still be some noise data. In this embodiment, a true value system can be pre-built. Figure 7 As shown in the figure, the truth system is based on a high-definition high-speed camera (usually set on the roof of the room). A special coded picture is attached to the prototype of the sweeping robot, and the real posture of the sweeping robot prototype is obtained by matching visual features. The posture of the sweeping robot prototype obtained by the truth system is very accurate and can be considered to be a real posture.

[0098] Based on the above-mentioned truth system, in this embodiment, plane image samples collected by a digital camera of the sweeping robot prototype in the truth system environment and point cloud data samples collected by a TOF camera can be obtained, and the true posture of the sweeping robot can be obtained through the truth system.

[0099] The training samples are composed of the plane image samples and point cloud data samples acquired in the true value system environment, the real posture is used as the sample label, and the target neural network model is fine-tuned through the training samples and the sample labels to obtain the trained target neural network model.

[0100] The accuracy of the target neural network model obtained by the above fine-tuning training has been further improved. The target neural network model can be deployed on the sweeping robot or used for inference in the control terminal. In this way, the mapping and positioning function of the sweeping robot based on the digital camera and TOF camera can be realized, thereby achieving the purpose of removing the radar.

[0101] The following is a description of a sweeping robot mapping and positioning device provided in an embodiment of the present application. The sweeping robot mapping and positioning device described below and the sweeping robot mapping and positioning method described above can correspond to each other.

[0102] See also Figure 8 , Figure 8 This is a schematic diagram of the structure of a sweeping robot mapping and positioning device disclosed in an embodiment of the present application.

[0103] like Figure 8 As shown, the device may include:

[0104] The sensor data acquisition unit 11 is used to acquire the plane image of the environment where the sweeping robot is located photographed by the digital camera, and to acquire the point cloud data of the environment where the sweeping robot is located photographed by the TOF camera;

[0105] The encoding fusion unit 12 is used to encode the plane image and the point cloud data respectively, and fuse the encoding features to obtain fusion features;

[0106] A prediction unit 13 is used to predict the current posture of the sweeping robot and the environmental point cloud information at a set height based on the fusion feature, where the set height is the height within the field of view of the digital camera and the TOF camera in the vertical direction;

[0107] The mapping and positioning unit 14 is used to simultaneously perform mapping and positioning based on the current posture of the robot and the environmental point cloud information at a set height.

[0108] In one possible implementation, the implementation process of the above-mentioned encoding fusion unit and the prediction unit is achieved by calling a pre-trained target neural network model. The target neural network model uses the plane image samples taken by the digital camera and the point cloud data samples taken by the TOF camera as training samples, and uses the annotated current posture of the sweeping robot and the environmental point cloud information at a set height as sample labels for training.

[0109] In a possible implementation, the target neural network model includes a first encoder, a second encoder, a first output layer, and a second output layer;

[0110] The first encoder is used to encode the planar image to obtain image coding features;

[0111] The second encoder is used to encode the point cloud data to obtain point cloud coding features, and the point cloud coding features are fused with the image coding features as fused features;

[0112] The first output layer is used to predict the current posture of the cleaning robot based on the fusion feature;

[0113] The second output layer is used to predict the environmental point cloud information at a set height based on the fused features.

[0114] In a possible implementation, the apparatus of the present application may further include: a model training unit, configured to train a target neural network model. The process may include:

[0115] Obtaining plane image samples collected by a digital camera on a sweeping robot prototype, point cloud data samples collected by a TOF camera, and radar data collected by a radar, and using the plane image samples and the point cloud data samples as training samples;

[0116] Based on the radar data, a synchronous mapping and positioning SLAM algorithm is used to calculate the positioning and mapping prediction results of the sweeping robot prototype and their confidence, wherein the positioning and mapping prediction results include the current posture of the sweeping robot prototype and the environmental point cloud information at a set height;

[0117] The positioning and mapping prediction results are screened according to the confidence level, and the prediction results with confidence levels higher than a set confidence threshold are retained as sample labels corresponding to the training samples;

[0118] The target neural network model is trained using the training samples and corresponding sample labels.

[0119] In a possible implementation, the model training unit is further used to:

[0120] After the target neural network model is trained using the training samples and the corresponding sample labels, a planar image sample collected by a digital camera and a point cloud data sample collected by a TOF camera of the sweeping robot prototype in a pre-built truth system environment are obtained, and the real posture of the sweeping robot is obtained through the truth system;

[0121] The plane image samples and the point cloud data samples acquired in a true value system environment constitute training samples, the real posture is used as a sample label, and the target neural network model is fine-tuned through the training samples and the sample labels to obtain a trained target neural network model.

[0122] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the sweeping robot mapping and positioning methods provided in the embodiments of the present application.

[0123] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the sweeping robot mapping and positioning methods provided in the embodiments of the present application.

[0124] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0125] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0126] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for mapping and positioning a sweeping robot, characterized in that: The cleaning robot is provided with at least one digital camera and at least one TOF camera for collecting planar images, and the setting positions of the digital camera and the TOF camera on the cleaning robot satisfy that the union of the field of view ranges of the two in the horizontal direction is not less than the set field of view value, and the method includes: Acquire a planar image of the environment in which the cleaning robot is located taken by the digital camera, and acquire point cloud data of the environment in which the cleaning robot is located taken by the TOF camera; Encoding the plane image and the point cloud data respectively, and fusing the encoded features to obtain fused features; Predicting the current posture of the sweeping robot and the environmental point cloud information at a set height based on the fusion features, wherein the set height is the height of the digital camera and the TOF camera within the vertical field of view; Mapping and positioning are performed synchronously based on the current posture of the robot and the environmental point cloud information at a set height.

2. The method according to claim 1, characterized in that Obtaining fusion features, and predicting the current posture of the sweeping robot and the environmental point cloud information at a set height based on the fusion features, are achieved through a pre-trained target neural network model; The target neural network model is trained using the plane image samples taken by the digital camera and the point cloud data samples taken by the TOF camera as training samples, and using the marked current posture of the sweeping robot and the environmental point cloud information at a set height as sample labels.

3. The method according to claim 2, characterized in that The target neural network model includes a first encoder, a second encoder, a first output layer and a second output layer; The first encoder is used to encode the planar image to obtain image coding features; The second encoder is used to encode the point cloud data to obtain point cloud coding features, and the point cloud coding features are fused with the image coding features as fused features; The first output layer is used to predict the current posture of the cleaning robot based on the fusion feature; The second output layer is used to predict the environmental point cloud information at a set height based on the fused features.

4. The method according to claim 2 or 3, characterized in that: The training process of the target neural network model includes: Obtaining plane image samples collected by a digital camera on a sweeping robot prototype, point cloud data samples collected by a TOF camera, and radar data collected by a radar, and using the plane image samples and the point cloud data samples as training samples; Based on the radar data, a synchronous mapping and positioning SLAM algorithm is used to calculate the positioning and mapping prediction results of the sweeping robot prototype and their confidence, wherein the positioning and mapping prediction results include the current posture of the sweeping robot prototype and the environmental point cloud information at a set height; The positioning and mapping prediction results are screened according to the confidence level, and the prediction results with confidence levels higher than a set confidence threshold are retained as sample labels corresponding to the training samples; The target neural network model is trained using the training samples and corresponding sample labels.

5. The method according to claim 4, characterized in that After the target neural network model is trained using the training samples and the corresponding sample labels, the method further includes: Obtain the plane image samples collected by the digital camera of the sweeping robot prototype in the pre-built truth value system environment, and the point cloud data samples collected by the TOF camera, and obtain the real posture of the sweeping robot through the truth value system; The plane image samples and the point cloud data samples acquired in a true value system environment constitute training samples, the real posture is used as a sample label, and the target neural network model is fine-tuned through the training samples and the sample labels to obtain a trained target neural network model.

6. A mapping and positioning device for a sweeping robot, characterized in that: The cleaning robot is provided with at least one digital camera and at least one TOF camera for collecting planar images. The setting positions of the digital camera and the TOF camera on the cleaning robot satisfy that the union of the field of view ranges of the two in the horizontal direction is not less than the set field of view value. The device includes: A sensor data acquisition unit, used to acquire a plane image of the environment in which the sweeping robot is located taken by the digital camera, and to acquire point cloud data of the environment in which the sweeping robot is located taken by the TOF camera; A coding fusion unit, used to encode the plane image and the point cloud data respectively, and fuse the coding features to obtain fusion features; A prediction unit, used to predict the current posture of the sweeping robot and the environmental point cloud information at a set height based on the fusion feature, where the set height is the height within the field of view of the digital camera and the TOF camera in the vertical direction; A mapping and positioning unit is used to simultaneously perform mapping and positioning based on the current posture of the robot and the environmental point cloud information at a set height.

7. A sweeping robot, characterized in that: include: Equipment body; At least one digital camera and at least one TOF camera are arranged on the device body, and the arrangement positions of the digital camera and the TOF camera on the sweeping robot satisfy that the union of the field of view ranges of the two in the horizontal direction is not less than the set field of view value, the digital camera is used to collect plane images, and the TOF camera is used to collect point cloud data; The processor connected to the digital camera and the TOF camera is used to complete the mapping and positioning tasks by using the sweeping robot mapping and positioning method according to any one of claims 1 to 5.

8. The sweeping robot according to claim 7, characterized in that: At least one of the TOF cameras is disposed on the front side of the cleaning robot, and at least one of the digital cameras is disposed on the rear side of the cleaning robot.

9. The sweeping robot according to claim 7, characterized in that: The digital camera is an RGB camera, a grayscale camera or an infrared camera.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, each step of the method for mapping and positioning a sweeping robot as claimed in any one of claims 1 to 5 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, each step of the method for mapping and positioning a sweeping robot as claimed in any one of claims 1 to 5 is implemented.