Indoor robot positioning method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an indoor robot positioning method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining sensor data of an indoor robot; performing visual inertial positioning on a target object according to the sensor data to obtain original pose data; mapping the target object according to the sensor data to obtain point cloud data; identifying the point cloud data and the sensor data to obtain reference pose data; and adjusting the original pose data according to the reference pose data to obtain a positioning result. By implementing the embodiment of the invention, efficient indoor robot positioning can be realized, the influence of the surrounding environment on positioning is avoided, moving objects can be reasonably and accurately identified, indoor positioning is accurately realized without errors, and the positioning efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and more specifically, to an indoor robot positioning method, device, electronic device and storage medium. Background Art

[0002] Robot positioning usually uses SLAM technology. Traditional SLAM technology can achieve high accuracy and efficiency in ideal environments. However, in actual scenarios, traditional SLAM methods have a variety of environmental complexity problems: for example, traditional SLAM technology relies on basic geometric features (such as points, lines, and surfaces) and lacks the ability to perceive macroscopic objects and scene understanding. If there are not enough features in the environment, the SLAM algorithm may not converge, resulting in inaccurate positioning or incomplete maps. In addition, SLAM methods usually require the use of sensors such as lidar and cameras to obtain environmental information, which also requires that the objective conditions in the environment are suitable for the operation of the sensor (visual algorithms are affected by light, laser algorithms are affected by reflectivity, etc.). Traditional SLAM technology is difficult to cope with dynamic or temporary environments. For some dynamic environments, such as people, animals, moving objects, etc., the SLAM algorithm is difficult to process this information correctly, and may incorrectly include it in the positioning or mapping process, resulting in errors in subsequent positioning. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide an indoor robot positioning method, device, electronic device and storage medium, which can achieve efficient indoor robot positioning, avoid the influence of the surrounding environment on positioning, reasonably and accurately identify moving objects, accurately and correctly achieve indoor positioning, and improve positioning efficiency.

[0004] In a first aspect, an embodiment of the present application provides an indoor robot positioning method, the method comprising:

[0005] Acquiring sensor data of the indoor robot;

[0006] Perform visual inertial positioning on the target object according to the sensor data to obtain original position and posture data;

[0007] Mapping the target object according to the sensor data to obtain point cloud data;

[0008] Identify the point cloud data and the sensor data to obtain reference pose data;

[0009] The original posture data is adjusted according to the reference posture data to obtain a positioning result.

[0010] In the above implementation process, visual inertial positioning and mapping of sensor data are performed before identification and re-positioning, which can achieve efficient indoor robot positioning, avoid the influence of the surrounding environment on positioning, reasonably and accurately identify moving objects, accurately and correctly achieve indoor positioning, and improve positioning efficiency.

[0011] Furthermore, the step of performing visual inertial positioning on the target object according to the sensor data to obtain original posture data includes:

[0012] Reading inertial measurement unit data from the sensor data;

[0013] Visual inertial positioning is performed on the target object according to the inertial measurement unit data to obtain the original position and posture data.

[0014] In the above implementation process, visual inertial positioning of the target object is performed according to the inertial measurement unit data, so that the obtained original posture data is more accurate and closer to the actual positioning, reducing the subsequent adjustment process.

[0015] Furthermore, the step of mapping the target object according to the sensor data to obtain point cloud data includes:

[0016] Reading an RGB-D image from the sensor data;

[0017] Performing color depth mapping on the target object according to the RGB-D image to obtain the point cloud data.

[0018] In the above implementation process, a color depth map of the target object is constructed according to the RGB-D image, so that the obtained point cloud data contains the color information of the target object, which is convenient for accurate positioning of the target object.

[0019] Furthermore, the step of identifying the point cloud data and the sensor data to obtain reference pose data includes:

[0020] Performing three-dimensional point cloud recognition on the point cloud data to obtain point cloud tower data;

[0021] Performing two-dimensional detection on the RGB image in the sensor data to obtain image tower data;

[0022] The point cloud tower data and the image tower data are fused to obtain the reference pose data.

[0023] In the above implementation process, three-dimensional point cloud recognition is performed on the point cloud data, two-dimensional detection is performed on the RGB image, and then the obtained point cloud tower data and image tower data are fused, so that the obtained reference pose data contains the two-dimensional information and three-dimensional information of the sensor data, which can more accurately locate the target object.

[0024] Furthermore, the step of performing three-dimensional point cloud recognition on the point cloud data to obtain point cloud tower data includes:

[0025] Extracting deep features of the point cloud data;

[0026] Classify and vote on the deep features to obtain potential object centers;

[0027] Aggregating the potential object centers to obtain classified object objects;

[0028] Performing non-maximum suppression processing on the classified object to obtain a two-dimensional bounding box of the target object and a label of the target object;

[0029] The point cloud tower data is obtained according to the two-dimensional bounding box of the target object and the label of the target object.

[0030] In the above implementation process, the deep features of the point cloud data are classified, voted, aggregated and processed with non-maximum suppression to obtain point cloud tower data, which can be used to extract more detailed features of the point cloud data and avoid the influence of the surrounding environment on the point cloud data.

[0031] Furthermore, the step of performing two-dimensional detection on the RGB image in the sensor data to obtain image tower data includes:

[0032] Obtaining a connection vector between the two-dimensional bounding box of the target object and the potential object center;

[0033] Performing three-dimensional space projection on the connection vector to obtain a projection vector;

[0034] Back-projecting the deep features onto the RGB image to obtain semantic information;

[0035] Obtaining an RGB texture according to the RGB image;

[0036] The image tower data is obtained according to the projection vector, the semantic information and the RGB texture.

[0037] In the above implementation process, the connection vector is projected into three-dimensional space, and then the deep features are back-projected into the RGB image to obtain semantic information. Image tower data is obtained according to the projection vector, semantic information and RGB texture, so that the image tower data contains more semantic information and image information, which can make the subsequent positioning more accurate.

[0038] Furthermore, the step of adjusting the original posture data according to the reference posture data to obtain a positioning result includes:

[0039] Fusing the reference pose data and the original pose data to obtain a pose graph;

[0040] The pose graph is repositioned to obtain the positioning result.

[0041] In the above implementation process, the reference pose data and the original pose data are fused, and the obtained pose graph is then repositioned, which can avoid unreasonable positioning in the original pose data and make the positioning result more accurate.

[0042] In a second aspect, an embodiment of the present application further provides an indoor robot positioning device, the device comprising:

[0043] An acquisition module, used to acquire sensor data of the indoor robot;

[0044] A visual inertial positioning module, used to perform visual inertial positioning on the target object according to the sensor data to obtain original posture data;

[0045] A mapping module, used to map the target object according to the sensor data to obtain point cloud data;

[0046] An identification module, used for identifying the point cloud data and the sensor data to obtain reference pose data;

[0047] The positioning module is used to adjust the original posture data according to the reference posture data to obtain a positioning result.

[0048] In the above implementation process, visual inertial positioning and mapping of sensor data are performed before identification and re-positioning, which can achieve efficient indoor robot positioning, avoid the influence of the surrounding environment on positioning, reasonably and accurately identify moving objects, accurately and correctly achieve indoor positioning, and improve positioning efficiency.

[0049] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in any one of the first aspects when executing the computer program.

[0050] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which instructions are stored. When the instructions are executed on a computer, the computer executes the method as described in any one of the first aspects.

[0051] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when running on a computer, enables the computer to execute the method as described in any one of the first aspects.

[0052] Other features and advantages of the present disclosure will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present disclosure.

[0053] And it can be implemented according to the contents of the specification. The following is a detailed description of the preferred embodiments of the present application with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the range values. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 A schematic diagram of a flow chart of an indoor robot positioning method provided in an embodiment of the present application;

[0056] Figure 2 A structural diagram of a posture graph provided in an embodiment of the present application;

[0057] Figure 3 A schematic diagram of the structure of an indoor robot positioning device provided in an embodiment of the present application;

[0058] Figure 4 A schematic diagram of the structural composition of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0060] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0061] The specific implementation methods of the present application are further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present application, but are not used to limit the scope of the present application.

[0062] Embodiment 1

[0063] Figure 1 is a flow chart of the indoor robot positioning method provided by the embodiment of the present application, such as Figure 1 As shown, the method includes:

[0064] S1, obtain sensor data of indoor robots;

[0065] S2, visual inertial positioning of the target object according to the sensor data to obtain the original posture data;

[0066] S3, mapping the target object based on the sensor data to obtain point cloud data;

[0067] S4, identifying the point cloud data and sensor data to obtain reference pose data;

[0068] S5, adjusting the original posture data according to the reference posture data to obtain a positioning result.

[0069] In the above implementation process, visual inertial positioning and mapping of sensor data are performed before identification and re-positioning, which can achieve efficient indoor robot positioning, avoid the influence of the surrounding environment on positioning, reasonably and accurately identify moving objects, accurately and correctly achieve indoor positioning, and improve positioning efficiency.

[0070] The embodiment of the present application proposes a real-time robot positioning method, which can enhance semantic information and reduce cumulative errors in robot positioning in a building scene.

[0071] In the S1, the sensor consists of a smart camera equipped with two grayscale lenses, one color lens, and a built-in inertial measurement unit (IMU). The real-time sensor data generated is used as input.

[0072] Furthermore, S2 includes:

[0073] Read the inertial measurement unit data from the sensor data;

[0074] The target object is visually inertially positioned according to the inertial measurement unit data to obtain the original posture data.

[0075] In the above implementation process, visual inertial positioning of the target object is performed according to the inertial measurement unit data, so that the obtained original posture data is more accurate and closer to the actual positioning, reducing the subsequent adjustment process.

[0076] Furthermore, S3 includes:

[0077] Read RGB-D images from sensor data;

[0078] The target object is mapped in color and depth based on the RGB-D image to obtain point cloud data.

[0079] In the above implementation process, a color depth map of the target object is constructed according to the RGB-D image, so that the obtained point cloud data contains the color information of the target object, which is convenient for accurate positioning of the target object.

[0080] The embodiment of the present application performs visual inertial positioning and color depth mapping through visual SLAM. RGB-D images and IMU data are read from sensor data to generate raw pose data to be optimized and point cloud data for object recognition.

[0081] Before identifying the target object, the embodiment of the present application also constructs an object database, that is, generates a pre-constructed object database as a priori landmark reference. The object database is constructed using industrial foundation classification (IFC) information extraction and graphics processing algorithms. It is convenient for the subsequent semantic recognition of the target object.

[0082] In traditional SLAM technology, loop detection is usually performed by a bag-of-words model that stores visual features. The geometric and semantic information in the macroscopic object landmarks is represented by the data structure "semantic object", as shown in Table 1. The semantic object class contains the basic attributes shared by objects from both sources (e.g., label, position, direction, etc.), as well as unique attributes that are different for each class (e.g., mesh structure of IFC objects, cumulative confidence of deep neural network (DNN) objects).

[0083] Table 1 Data structure: semantic object

[0084]

[0085] For IFC objects, class attributes are extracted by parsing the prior BIM in IFC format. Taking the indoor furniture components of the embodiment of the present application as an example, except for the "source" and "serial number" which are automatically determined by the program when creating a new object, other class attributes are extracted from the corresponding IFC attributes. The semantic information is contained in Name and ObjectType, which are extracted as object names and labels. The geometric information is contained in ObjectPlacement and Representation, which define the position and shape of the object respectively. Position information (position and direction) is represented by IfcAxis2Placement3D, which defines the local coordinates in three-dimensional space relative to the reference coordinates (such as the floor origin). On the other hand, shape information requires further data conversion to match the object point cloud detected in real time. The present application adopts a polyhedron structure as the representation format of geometric data. The polyhedron format can be extracted from the BIM through triangulation, and the table is further converted into a point cloud through sampling operations.

[0086] Furthermore, S4 includes:

[0087] Perform three-dimensional point cloud recognition on the point cloud data to obtain point cloud tower data;

[0088] Perform two-dimensional detection on the RGB image in the sensor data to obtain image tower data;

[0089] The point cloud tower data and image tower data are fused to obtain reference pose data.

[0090] In the above implementation process, three-dimensional point cloud recognition is performed on the point cloud data, two-dimensional detection is performed on the RGB image, and then the obtained point cloud tower data and image tower data are fused, so that the obtained reference pose data contains the two-dimensional information and three-dimensional information of the sensor data, which can more accurately locate the target object.

[0091] The recognition of this application consists of a three-dimensional point cloud detection branch and a two-dimensional image detection branch. This approach combines the advantages of the point cloud recognition network ImVoteNet and the image recognition network YOLO, greatly improving efficiency while maintaining high accuracy.

[0092] Furthermore, the step of performing three-dimensional point cloud recognition on the point cloud data to obtain point cloud tower data includes:

[0093] Extract deep features of point cloud data;

[0094] Classify and vote on deep features to obtain potential object centers;

[0095] Aggregate potential object centers to obtain classified object objects;

[0096] Perform non-maximum suppression on the classified object to obtain the two-dimensional bounding box of the target object and the label of the target object;

[0097] The point cloud tower data is obtained based on the two-dimensional bounding box of the target object and the label of the target object.

[0098] In the above implementation process, the deep features of the point cloud data are classified, voted, aggregated and processed with non-maximum suppression to obtain point cloud tower data, which can be used to extract more detailed features of the point cloud data and avoid the influence of the surrounding environment on the point cloud data.

[0099] 3D point cloud recognition is processed through the following steps: use PointNet++ to extract deep features as seed points; use the MLP network to let the seed points vote for classification to obtain potential object centers; aggregate the obtained potential object centers into clusters, and use shared PointNet to determine and classify object objects; use non-maximum suppression to generate bounding boxes and labels of target objects for DNN-derived semantic object creation.

[0100] Furthermore, the step of performing two-dimensional detection on the RGB image in the sensor data to obtain image tower data includes:

[0101] Obtain the connection vector between the 2D bounding box of the target object and the potential object center;

[0102] Project the connection vector into three-dimensional space to obtain a projection vector;

[0103] Back-project the deep features to the RGB image to obtain semantic information;

[0104] Get RGB texture from RGB image;

[0105] Image tower data is obtained based on projection vectors, semantic information and RGB texture.

[0106] In the above implementation process, the connection vector is projected into three-dimensional space, and then the deep features are back-projected into the RGB image to obtain semantic information. Image tower data is obtained according to the projection vector, semantic information and RGB texture, so that the image tower data contains more semantic information and image information, which can make the subsequent positioning more accurate.

[0107] The 2D detection branch receives an RGB image and a 2D bounding box to produce an image prediction, and assists in point cloud prediction from three aspects: geometry, semantics, and texture. In terms of geometry, the connection vector from each pixel in the 2D bounding box to the center of the object can be directly projected into 3D space through the camera parameters. These projection vectors can be used to narrow the search range for the center of the object. In terms of semantics and texture, the seed point can be back-projected to the image plane to obtain the semantic label and corresponding probability within the region, and the RGB texture can be obtained based on the original pixel value.

[0108] The two-dimensional detection model of this application refers to the one-stage structure of YOLO, which shortens the two-dimensional detection time and can generate and classify bounding boxes simultaneously through a single convolutional neural network. Therefore, this application can integrate the pre-trained model into the real-time sensor system, maintaining detection accuracy while being efficient and convenient.

[0109] Furthermore, S5 includes:

[0110] The reference pose data and the original pose data are fused to obtain a pose graph;

[0111] Reposition the pose graph and obtain the positioning result.

[0112] In the above implementation process, the reference pose data and the original pose data are fused, and the obtained pose graph is then repositioned, which can avoid unreasonable positioning in the original pose data and make the positioning result more accurate.

[0113] The structure of the pose graph is as follows Figure 2 As shown in Figure 1, each node represents a body state (timestamp, position, and orientation), and each edge represents a constraint between two nodes. There are two types of edges in the proposed pose graph: sequential edges (solid lines) connecting consecutive keyframe nodes and relative edges (dashed lines) connecting keyframe nodes with object recognition frame nodes. To ensure computational efficiency, each keyframe node is connected to four previous keyframe nodes and one recognition frame node.

[0114] The relocalization process can be transformed into a nonlinear least squares problem, minimizing the residual terms (6 degrees of freedom) of all edges in the pose graph. The residual represents the error between the predicted value and the target value of the motion. It contains three dimensions of translation and rotation respectively. Here, quaternion (Q) is selected to represent rotation. For each edge, the residual between nodes i and j can be calculated by the formula. For each residual, the translation error (3 degrees of freedom) is the vector difference between the predicted value and the target value, while the rotation error (3 degrees of freedom) is the vector part of the result of the relative rotation quaternion division between the predicted value and the target value (following the quaternion operation rules), and the second to fourth elements are taken. The results of both are reflected in the world coordinate system, and the sum of squares of the residual terms of all edges is minimized. The loss function is used to reduce the impact of potential mismatches. By solving the objective function, the key frame nodes on the pose graph will be "stretched" toward the recognition frame nodes while retaining the original continuous constraints. The pose graph of the entire object can be relocalized as a result.

[0115] The application scenario of the embodiment of the present application can be any building facility (such as a residence, office building) that already has a BIM model, and the application platform can be any mobile robot platform (such as a sweeping robot, etc.) equipped with a color depth camera (RGB-D). Here, the Turtlebot3 robot equipped with a RealSense D435i smart camera and a laptop with an RTX2060 graphics card is taken as an example to demonstrate the beneficial effects of this application on traditional SLAM in an office building environment. The experimental scene is a public rest area in an office building, which contains indoor furniture components, building structure components, HVAC equipment, etc.

[0116] During the mobile scanning process of the robot platform, the prior information provided by BIM and the object recognition ability of the artificial intelligence model can enhance the performance of traditional SLAM technology in the following aspects:

[0117] Enhanced environmental perception: Deep learning technology can more accurately identify objects in the environment, thereby improving the robot's environmental perception. BIM can provide more comprehensive static environmental information, such as the room layout of the building, the location of doors and windows, etc. Deep learning object recognition can simultaneously obtain static and dynamic related information, providing more references for the SLAM algorithm, thereby eliminating the influence of complex environments.

[0118] Improve positioning accuracy: Building components in BIM can provide a priori target references, and deep learning models can infer the robot's position in the building by identifying objects in the building. By combining this information, the positioning accuracy of the SLAM algorithm can be improved.

[0119] Enhanced map building capabilities: Building information in BIM can provide three-dimensional building structure information, and deep learning technology can improve the accuracy of map building by identifying different scenes. For example, robots can use deep learning technology to identify different rooms or objects, and match this information with the building information in BIM to build a more accurate map.

[0120] This application utilizes prior knowledge related to buildings and the object recognition capabilities of deep learning to effectively improve the performance of traditional SLAM technology in terms of improving environmental perception, positioning accuracy, and map building capabilities. It can also be used in a wider range of robot intelligent applications, providing effective solutions for intelligent decision-making, human-computer interaction and other functions under robot operating conditions.

[0121] Embodiment 2

[0122] In order to execute the method corresponding to the above embodiment 1 to achieve the corresponding functions and technical effects, an indoor robot positioning device is provided below, such as Figure 3 As shown, the device comprises:

[0123] Acquisition module 1, used to acquire sensor data of the indoor robot;

[0124] The visual inertial positioning module 2 is used to perform visual inertial positioning on the target object according to the sensor data to obtain the original posture data;

[0125] Mapping module 3, used to map the target object according to the sensor data to obtain point cloud data;

[0126] Recognition module 4, used to recognize point cloud data and sensor data to obtain reference pose data;

[0127] The positioning module 5 is used to adjust the original posture data according to the reference posture data to obtain a positioning result.

[0128] In the above implementation process, visual inertial positioning and mapping of sensor data are performed before identification and re-positioning, which can achieve efficient indoor robot positioning, avoid the influence of the surrounding environment on positioning, reasonably and accurately identify moving objects, accurately and correctly achieve indoor positioning, and improve positioning efficiency.

[0129] Furthermore, the visual inertial positioning module 2 is also used for:

[0130] Read the inertial measurement unit data from the sensor data;

[0131] The target object is visually inertially positioned according to the inertial measurement unit data to obtain the original posture data.

[0132] In the above implementation process, visual inertial positioning of the target object is performed according to the inertial measurement unit data, so that the obtained original posture data is more accurate and closer to the actual positioning, reducing the subsequent adjustment process.

[0133] Furthermore, the mapping module 3 is also used for:

[0134] Read RGB-D images from sensor data;

[0135] The target object is mapped in color and depth based on the RGB-D image to obtain point cloud data.

[0136] In the above implementation process, a color depth map of the target object is constructed according to the RGB-D image, so that the obtained point cloud data contains the color information of the target object, which is convenient for accurate positioning of the target object.

[0137] Furthermore, the identification module 4 is also used for:

[0138] Perform three-dimensional point cloud recognition on the point cloud data to obtain point cloud tower data;

[0139] Perform two-dimensional detection on the RGB image in the sensor data to obtain image tower data;

[0140] The point cloud tower data and image tower data are fused to obtain reference pose data.

[0141] In the above implementation process, three-dimensional point cloud recognition is performed on the point cloud data, two-dimensional detection is performed on the RGB image, and then the obtained point cloud tower data and image tower data are fused, so that the obtained reference pose data contains the two-dimensional information and three-dimensional information of the sensor data, which can more accurately locate the target object.

[0142] Furthermore, the identification module 4 is also used for:

[0143] Extract deep features of point cloud data;

[0144] Classify and vote on deep features to obtain potential object centers;

[0145] Aggregate potential object centers to obtain classified object objects;

[0146] Perform non-maximum suppression on the classified object to obtain the two-dimensional bounding box of the target object and the label of the target object;

[0147] The point cloud tower data is obtained according to the two-dimensional bounding box of the target object and the label of the target object.

[0148] In the above implementation process, the deep features of the point cloud data are classified, voted, aggregated and processed with non-maximum suppression to obtain point cloud tower data, which can be used to extract more detailed features of the point cloud data and avoid the influence of the surrounding environment on the point cloud data.

[0149] Furthermore, the identification module 4 is also used for:

[0150] Obtain the connection vector between the 2D bounding box of the target object and the potential object center;

[0151] Project the connection vector into three-dimensional space to obtain a projection vector;

[0152] Back-project the deep features to the RGB image to obtain semantic information;

[0153] Get RGB texture from RGB image;

[0154] Image tower data is obtained based on projection vectors, semantic information and RGB texture.

[0155] In the above implementation process, the connection vector is projected into three-dimensional space, and then the deep features are back-projected into the RGB image to obtain semantic information. Image tower data is obtained according to the projection vector, semantic information and RGB texture, so that the image tower data contains more semantic information and image information, which can make the subsequent positioning more accurate.

[0156] Furthermore, the positioning module 5 is also used for:

[0157] The reference pose data and the original pose data are fused to obtain a pose graph;

[0158] Reposition the pose graph and obtain the positioning result.

[0159] In the above implementation process, the reference pose data and the original pose data are fused, and the obtained pose graph is then repositioned, which can avoid unreasonable positioning in the original pose data and make the positioning result more accurate.

[0160] The indoor robot positioning device described above can implement the method of the above embodiment 1. The options in the above embodiment 1 are also applicable to this embodiment and will not be described in detail here.

[0161] The rest of the contents of the embodiments of the present application can refer to the contents of the above-mentioned embodiment 1, and will not be repeated in this embodiment.

[0162] Embodiment 3

[0163] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the indoor robot positioning method of embodiment 1.

[0164] Optionally, the above-mentioned electronic device may be a server.

[0165] See also Figure 4 , Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 41, a communication interface 42, a memory 43 and at least one communication bus 44. The communication bus 44 is used to realize direct connection and communication between these components. The communication interface 42 of the device in the embodiment of the present application is used to communicate signaling or data with other node devices. The processor 41 can be an integrated circuit chip with signal processing capabilities.

[0166] The processor 41 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor 41 can also be any conventional processor, etc.

[0167] The memory 43 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electric erasable programmable read-only memory (EEPROM), etc. The memory 43 stores computer readable instructions. When the computer readable instructions are executed by the processor 41, the device can execute the above-mentioned Figure 1 The method embodiment involves various steps.

[0168] Optionally, the electronic device may further include a storage controller and an input / output unit. The memory 43, the storage controller, the processor 41, the peripheral interface, and the input / output unit are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components may be electrically connected to each other via a communication bus 44. The processor 41 is used to execute executable modules stored in the memory 43, such as software function modules or computer programs included in the device.

[0169] The input and output unit is used to provide users with the task creation and to create a start optional time period or preset execution time for the task to realize the interaction between the user and the server. The input and output unit can be, but is not limited to, a mouse and a keyboard.

[0170] Understandably, Figure 4 The structure shown is for illustration only. The electronic device may also include Figure 4 More or fewer components as shown, or with Figure 4 Different configurations are shown. Figure 4 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0171] In addition, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the indoor robot positioning method of embodiment 1.

[0172] The embodiment of the present application also provides a computer program product, which, when running on a computer, enables the computer to execute the method described in the method embodiment.

[0173] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart, can be implemented with a dedicated hardware-based device that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0174] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0175] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0176] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application. It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0177] The above is only a specific implementation of the present application, but the protection range of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical range disclosed in the present application, which should be included in the protection range of the present application. Therefore, the protection range of the present application should be based on the protection range of the claims.

[0178] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for positioning an indoor robot, characterized in that: The method comprises: Acquiring sensor data of the indoor robot; Perform visual inertial positioning on the target object according to the sensor data to obtain original position and posture data; Mapping the target object according to the sensor data to obtain point cloud data; Identify the point cloud data and the sensor data to obtain reference pose data; The original posture data is adjusted according to the reference posture data to obtain a positioning result.

2. The indoor robot positioning method according to claim 1, characterized in that: The step of performing visual inertial positioning on the target object according to the sensor data to obtain original posture data includes: Reading inertial measurement unit data from the sensor data; Visual inertial positioning is performed on the target object according to the inertial measurement unit data to obtain the original position and posture data.

3. The indoor robot positioning method according to claim 1, characterized in that: The step of mapping the target object according to the sensor data to obtain point cloud data includes: Reading an RGB-D image from the sensor data; Performing color depth mapping on the target object according to the RGB-D image to obtain the point cloud data.

4. The indoor robot positioning method according to claim 1, characterized in that: The step of identifying the point cloud data and the sensor data to obtain reference pose data comprises: Performing three-dimensional point cloud recognition on the point cloud data to obtain point cloud tower data; Performing two-dimensional detection on the RGB image in the sensor data to obtain image tower data; The point cloud tower data and the image tower data are fused to obtain the reference pose data.

5. The indoor robot positioning method according to claim 4, characterized in that: The step of performing three-dimensional point cloud recognition on the point cloud data to obtain point cloud tower data includes: Extracting deep features of the point cloud data; Classify and vote on the deep features to obtain potential object centers; Aggregating the potential object centers to obtain classified object objects; Performing non-maximum suppression processing on the classified object to obtain a two-dimensional bounding box of the target object and a label of the target object; The point cloud tower data is obtained according to the two-dimensional bounding box of the target object and the label of the target object.

6. The indoor robot positioning method according to claim 4, characterized in that: The step of performing two-dimensional detection on the RGB image in the sensor data to obtain image tower data comprises: Obtaining a connection vector between the two-dimensional bounding box of the target object and the potential object center; Performing three-dimensional space projection on the connection vector to obtain a projection vector; Back-projecting the deep features onto the RGB image to obtain semantic information; Obtaining an RGB texture according to the RGB image; The image tower data is obtained according to the projection vector, the semantic information and the RGB texture.

7. The indoor robot positioning method according to claim 1, characterized in that: The step of adjusting the original posture data according to the reference posture data to obtain a positioning result comprises: Fusing the reference pose data and the original pose data to obtain a pose graph; The pose graph is repositioned to obtain the positioning result.

8. An indoor robot positioning device, characterized in that: The device comprises: An acquisition module, used to acquire sensor data of the indoor robot; A visual inertial positioning module, used to perform visual inertial positioning on the target object according to the sensor data to obtain original posture data; A mapping module, used to map the target object according to the sensor data to obtain point cloud data; An identification module, used for identifying the point cloud data and the sensor data to obtain reference pose data; The positioning module is used to adjust the original posture data according to the reference posture data to obtain a positioning result.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the indoor robot positioning method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the indoor robot positioning method as described in any one of claims 1 to 7.

Images