Indoor positioning method and apparatus based on target detection and visual slam

Abstract

The application provides an indoor positioning method and device based on target detection and visual SLAM, wherein the method comprises the following steps: acquiring an environment image sequence of the whole indoor space by collecting an RGB-D camera carried by a robot; constructing an indoor three-dimensional point cloud map; performing semantic segmentation on a key frame and detecting the key frame by using a target detection model to obtain category information of objects in the key frame; fusing the category information of the objects in the key frame into the indoor three-dimensional point cloud map to construct an indoor three-dimensional semantic map; acquiring an environment image of a current position of a robot to be positioned by using the RGB-D camera; acquiring category information of objects in the environment image of the current position of the robot to be positioned; estimating a rough position of the robot to be positioned in the indoor space; acquiring a distance between a photographed object and the robot to be positioned by using the RGB-D camera, and combining the rough position of the robot to be positioned in the indoor space to obtain an accurate position of the robot to be positioned in the indoor space.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more specifically to an indoor positioning method and an indoor positioning device based on target detection and visual SLAM. Background Technology

[0002] With the rapid development and increasing maturity of robotics technology, the application of robots in indoor environments is becoming more and more widespread. Indoor positioning is one of the key technologies for indoor robot applications.

[0003] Various technologies have emerged in the field of indoor robot positioning, such as ultra-wideband (UWB) technology, ultrasonic navigation and positioning technology, wireless network-based positioning systems, and laser ranging and positioning technology. However, each positioning technology has its own drawbacks and limitations. For example, ultrasonic positioning technology, due to the relatively low speed of sound waves and long transmission time, cannot achieve real-time information transmission; wireless network-based positioning systems, although having low hardware costs, often suffer from positioning accuracy issues due to the instability of WLAN signals; and UWB technology, due to its high cost, short communication distance, and the requirement that UWB device power be below a certain threshold to avoid interfering with other communication systems, inevitably leads to errors in indoor positioning. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides an indoor positioning method and apparatus based on target detection and visual SLAM, which can improve the accuracy and real-time performance of robot positioning.

[0005] The technical solution adopted in this invention is as follows:

[0006] An indoor localization method based on object detection and visual SLAM includes the following steps: acquiring a sequence of environmental images of the entire indoor environment using an RGB-D camera carried by a robot; constructing an indoor 3D point cloud map using a SLAM algorithm based on the environmental image sequence; performing semantic segmentation on keyframes in the environmental image sequence and detecting the keyframes using an object detection model to obtain the category information of objects in the keyframes; fusing the category information of objects in the keyframes into the indoor 3D point cloud map to construct an indoor 3D semantic map; and acquiring the image of the robot to be localized using an RGB-D camera carried by the robot in the indoor environment. The environmental image of the current location is used; the target detection model is used to detect the environmental image of the current location of the robot to be located to obtain the category information of objects in the environmental image of the current location of the robot to be located; based on the category information of objects in the environmental image of the current location of the robot to be located and the indoor 3D semantic map, the approximate position of the robot to be located in the room is estimated; the distance between the object and the robot to be located is obtained by the RGB-D camera carried by the robot to be located, and combined with the approximate position of the robot to be located in the room, the precise position of the robot to be located in the room is obtained.

[0007] The Deeplab V3 plus algorithm is used to perform semantic segmentation on keyframes in the environmental image sequence, and the target detection model is the SSD model.

[0008] The indoor 3D point cloud map is stored using an octree structure, and object category labels are incorporated into the indoor 3D point cloud map to construct the indoor 3D semantic map.

[0009] The objects in the room are fixed objects, including one or more of the following: sofa, coffee table, dining table, bed, wardrobe, washing machine, and refrigerator.

[0010] The distance between the photographed object and the robot to be located is obtained using the time-of-flight method.

[0011] An indoor localization device based on object detection and visual SLAM includes: a first acquisition module, configured to acquire a sequence of environmental images of the entire indoor environment using an RGB-D camera carried by a robot; a first construction module, configured to construct an indoor 3D point cloud map based on the environmental image sequence using a SLAM algorithm; a first detection module, configured to perform semantic segmentation on keyframes in the environmental image sequence and detect the keyframes using an object detection model to obtain object category information in the keyframes; a second construction module, configured to fuse the object category information in the keyframes into the indoor 3D point cloud map to construct an indoor 3D semantic map; and a second acquisition module, configured to acquire images of the entire indoor environment using an RGB-D camera carried by a robot to be localized within the indoor environment. The system comprises: an RGB-D camera to acquire an environmental image of the current location of the robot to be located; a second detection module, which uses the target detection model to detect the environmental image of the current location of the robot to obtain the category information of objects in the environmental image of the current location of the robot to be located; an estimation module, which estimates the approximate location of the robot to be located in the indoor environment based on the category information of objects in the environmental image of the current location of the robot to be located and the indoor 3D semantic map; and a positioning module, which uses the RGB-D camera carried by the robot to acquire the distance between the photographed objects and the robot to be located, and combines this distance with the approximate location of the robot to be located in the indoor environment to obtain the precise location of the robot to be located in the indoor environment.

[0012] The first detection module uses the Deeplab V3 plus algorithm to perform semantic segmentation on keyframes in the environmental image sequence, and the target detection model is the SSD model.

[0013] The second construction module uses an octree structure to store the indoor 3D point cloud map and incorporates object category labels into the indoor 3D point cloud map to construct the indoor 3D semantic map.

[0014] The objects in the room are fixed objects, including one or more of the following: sofa, coffee table, dining table, bed, wardrobe, washing machine, and refrigerator.

[0015] The positioning module obtains the distance between the photographed object and the robot to be positioned using the time-of-flight method.

[0016] The beneficial effects of this invention are:

[0017] This invention constructs an indoor 3D semantic map using the SLAM algorithm, acquires environmental images of the robot's location using an RGB-D camera, and employs a target detection model to accurately locate the robot, thereby improving the accuracy and real-time performance of robot localization. Attached Figure Description

[0018] Figure 1 This is a flowchart of an indoor positioning method based on target detection and visual SLAM according to an embodiment of the present invention;

[0019] Figure 2 This is a schematic diagram of the ORB-SLAM2 algorithm according to an embodiment of the present invention;

[0020] Figure 3 This is a schematic diagram of the network structure of the Deeplab V3 plus algorithm according to an embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram of an octree storage structure according to an embodiment of the present invention;

[0022] Figure 5 This is a block diagram of an indoor positioning device based on target detection and visual SLAM according to an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] like Figure 1 As shown, the indoor positioning method based on target detection and visual SLAM according to an embodiment of the present invention includes the following steps:

[0025] S1 acquires a sequence of indoor environmental images by using an RGB-D camera carried by the robot.

[0026] In one embodiment of the present invention, an environmental image sequence can be formed by a robot carrying an RGB-D camera moving indoors and collecting environmental images of various locations indoors.

[0027] S2. Based on the environmental image sequence, the SLAM algorithm is used to construct an indoor 3D point cloud map.

[0028] In one embodiment of the present invention, the ORB-SLAM2 algorithm can be used to construct an indoor 3D point cloud map by using the above-mentioned environmental image sequence as the input of the entire visual SLAM system.

[0029] like Figure 2 As shown, the ORB-SLAM2 algorithm comprises three threads: a tracking thread, a local mapping thread, and a global loop closure detection thread. Specifically, in the tracking thread, ORB feature points are extracted from each input frame, compared with the nearest keyframe, the position of the feature points is calculated, and the camera pose is roughly estimated. If tracking fails, re-localization is performed using the bag-of-words method, ultimately deciding whether to insert a keyframe. If the number of tracked feature points in the current image is very small, it indicates a significant change in the current environment compared to the previous keyframe; in this case, a new keyframe should be inserted to avoid tracking failure. In the local mapping thread, Bundle Adjustment optimizes the feature points and camera pose in the local space, removing redundant keyframes and low-quality map points to obtain a more accurate camera pose and spatial position of feature points. In the global loop closure detection thread, the bag-of-words method can be used to detect loop closures on the global map and keyframes. If a loop closure is detected, the cumulative error of the odometry is calculated using Sim3 optimization. Then, all matched point clouds are aligned and duplicate points are merged. Finally, graph optimization is performed on the similarity constraints to achieve global consistency and obtain a sparse 3D point cloud map. Keyframes are images that are clear, have a sufficient number of feature points that are evenly distributed, and have a certain degree of co-view relationship with other keyframes. At the same time, the repetition rate must not be too high. This can be understood as requiring constraints between keyframes while minimizing redundant information.

[0030] S3 performs semantic segmentation on keyframes in the environmental image sequence and uses an object detection model to detect keyframes in order to obtain the category information of objects in the keyframes.

[0031] In one embodiment of the present invention, the Deeplab V3 plus algorithm can be used to perform semantic segmentation on keyframes in an environmental image sequence. The indoor objects to be detected by the object detection model are fixed-position objects, including one or more of the following: sofa, coffee table, dining table, bed, wardrobe, washing machine, and refrigerator.

[0032] The network structure of the Deeplab V3 plus algorithm is as follows: Figure 3As shown, the encoder architecture of the model consists of a ResNet101 backbone network and an ASPP module. The backbone network extracts image features to generate high-level semantic feature maps. The ASPP module uses the high-level semantic feature maps obtained from the backbone network to perform multi-scale sampling, generating multi-scale feature maps. At the end of the encoder, the multi-scale high-level semantic feature maps are combined along the channel dimension using a 1×1 convolution to reduce the channel dimensionality, maintaining the weight of the high-level semantic feature maps when concatenated, thus increasing the network's learning ability. Then, a 3×3 convolution is used to extract features, and upsampling is performed at the end of the encoder to produce the final semantic segmentation map.

[0033] In one embodiment of the present invention, the object detection model is the SSD model. Using the SSD model, category information of objects in the environmental image can be obtained. Different objects are given different colors and bounding boxes are drawn to generate labels. A regression method is then used to establish the object detection framework, eliminating the step of generating candidate boxes and significantly improving the object detection speed.

[0034] S4 integrates the object category information from the keyframes into the indoor 3D point cloud map to construct an indoor 3D semantic map.

[0035] In one embodiment of the present invention, an octree structure can be used to store an indoor 3D point cloud map, and object category labels can be incorporated into the indoor 3D point cloud map to construct an indoor 3D semantic map. Here, an octree is a hierarchical data structure in 3D space, based on which space can be infinitely divided.

[0036] In one embodiment of the present invention, such as Figure 4 As shown, each node in an octree represents a voxel, and its value is the size of the space occupied by that voxel. Each voxel can be divided into eight voxels of the same size, and this process is repeated until the smallest square is formed. The entire large square can be regarded as the root node, and the smallest square can be regarded as the "leaf node". In an octree, the map volume can be increased eightfold when moving from the lower level to the upper level. If all nodes of a square are occupied or not occupied, there is no need to expand that node. In the octree space, the bottom-level leaves represent the lowest resolution of the map. Occupation and unoccupiedness are expressed in probabilistic form. Let there be a node n, and its corresponding observation data z. The prior probability of a node being occupied is generally taken as 0.5, indicating that in an unknown environment, the prior probability of a node being occupied and unoccupied is equal, denoted by P(n|z). 1:T Let t represent the observation probability of this node from the initial time to time t. Then, the probability of this node from the initial time to time t+1 is:

[0037]

[0038] The probability logarithm can be expressed as:

[0039] L(n|z 1:t+1 )=L(n|z 1:t )+L(n|z t+1 (2)

[0040] Wherein L(n|z 1:t L(n|z) represents the logarithmic probability of observation at that node from the initial time to time t. t+1 ) represents the logarithmic probability of the node from time t+1.

[0041] S5 acquires environmental images of the robot's current location using an RGB-D camera carried by the robot to be located indoors.

[0042] In one embodiment of the present invention, the robot to be located and the data acquisition robot can be the same robot or different robots. The purpose of the data acquisition robot is to acquire environmental image sequences for constructing an indoor 3D semantic map, while the robot to be located is the robot that uses the environmental image sequences of the indoor 3D semantic map constructed in the above steps to achieve its own localization.

[0043] S6. An object detection model is used to detect the environmental image of the current location of the robot to be located in order to obtain the category information of objects in the environmental image of the current location of the robot to be located.

[0044] S7. Based on the category information of objects in the environmental image of the robot's current location and the indoor 3D semantic map, estimate the approximate location of the robot indoors.

[0045] In one embodiment of the present invention, since the position of the identified object is fixed, the position of the robot to be located indoors can be roughly estimated based on the position of the indoor object and in combination with the previously constructed three-dimensional semantic map.

[0046] S8 obtains the distance between the object being photographed and the robot being located using the RGB-D camera carried by the robot being located, and combines this distance with the approximate location of the robot being located indoors to obtain the precise location of the robot being located indoors.

[0047] In one embodiment of the present invention, the distance between the photographed object and the robot to be located can be obtained by the time-of-flight method.

[0048] In one embodiment of the present invention, the infrared emitter in the RGB-D camera continuously projects infrared measurement modulated light onto the object being measured within its field of view. An internal timer calculates the emission time of each beam, and an infrared detector searches for the beam reflected back from the object in the incident direction. When the detector receives the returned beam, the timer stops. Using the known speed of light, by measuring the depth value of a specified pixel in the depth image, the distance information between the actual object corresponding to that pixel and the depth camera plane can be obtained. The formula for TOF measurement technology is:

[0049]

[0050] Where d is the distance between the object being measured and the depth camera, c is the flight speed of the emitted light beam in the measurement environment, f is the modulation frequency of the infrared light speed, and Δψ is the phase difference between the incident and reflected light beams. Since the approximate location of the robot to be located indoors is known, the planar distance between the robot and the photographed object is calculated using the time-of-flight method. Therefore, the precise location of the robot indoors can be calculated from the planar distances between the robot and any two objects indoors.

[0051] The indoor localization method based on target detection and visual SLAM according to embodiments of the present invention constructs an indoor three-dimensional semantic map through SLAM algorithm, acquires environmental images of the location of the robot to be localized using an RGB-D camera, and uses a target detection model to accurately localize the robot, thereby improving the accuracy and real-time performance of robot localization.

[0052] To realize the indoor positioning method based on target detection and visual SLAM in the above embodiments, the present invention also proposes an indoor positioning device based on target detection and visual SLAM.

[0053] like Figure 5As shown, the indoor positioning device based on target detection and visual SLAM according to an embodiment of the present invention includes: a first acquisition module 10, a first construction module 20, a first detection module 30, a second construction module 40, a second acquisition module 50, a second detection module 60, an estimation module 70, and a positioning module 80. The system comprises the following modules: a first acquisition module 10 acquires a sequence of indoor environmental images using an RGB-D camera carried by a robot; a first construction module 20 constructs an indoor 3D point cloud map using a SLAM algorithm based on the environmental image sequence; a first detection module 30 performs semantic segmentation on keyframes in the environmental image sequence and uses an object detection model to detect keyframes to obtain object category information in the keyframes; a second construction module 40 integrates the object category information in the keyframes into the indoor 3D point cloud map to construct an indoor 3D semantic map; a second acquisition module 50 acquires an environmental image of the robot's current location using an RGB-D camera carried by the robot; a second detection module 60 uses an object detection model to detect the environmental image of the robot's current location to obtain object category information in the environmental image; an estimation module 70 estimates the approximate location of the robot indoors based on the object category information in the environmental image and the indoor 3D semantic map; and a positioning module 80 uses an RGB-D camera carried by the robot to acquire an environmental image of the robot's current location. The D camera acquires the distance between the photographed object and the robot to be located, and combines this distance with the approximate location of the robot indoors to obtain the precise location of the robot indoors.

[0054] In one embodiment of the present invention, the objects indoors are fixed-position objects, including one or more of a sofa, coffee table, dining table, bed, wardrobe, washing machine, and refrigerator. The first acquisition module 10 can acquire environmental images of various locations indoors by moving the RGB-D camera carried by the acquisition robot, thus forming an environmental image sequence.

[0055] In one embodiment of the present invention, the first construction module 20 can construct an indoor 3D point cloud map using the ORB-SLAM2 algorithm. The ORB-SLAM2 algorithm comprises three threads: a tracking thread, a local mapping thread, and a global loop closure detection thread. Specifically, in the tracking thread, ORB feature points are extracted from each input image frame, compared with the nearest keyframe, the position of the feature points is calculated, and the camera pose is roughly estimated. If tracking fails, re-localization is performed using the bag-of-words method, ultimately deciding whether to insert a keyframe. If the number of tracked feature points in the current image is very small, it indicates that the current environment has changed significantly from the environment of the previous keyframe; in this case, a new keyframe should be inserted to avoid tracking failure. In the local mapping thread, Bundle Adjustment can be used to optimize the feature points and camera pose in the local space, deleting redundant keyframes and low-quality map points to obtain a more accurate camera pose and spatial position of feature points. In the global loop closure detection thread, the bag-of-words method can be used to detect loop closures on the global map and keyframes. If a loop closure is detected, the cumulative error of the odometry is calculated using Sim3 optimization. Then, all matched point clouds are aligned and duplicate points are merged. Finally, graph optimization is performed on the similarity constraints to achieve global consistency and obtain a sparse 3D point cloud map. Keyframes are images that are clear, have a sufficient number of feature points that are evenly distributed, and have a certain degree of co-view relationship with other keyframes. Simultaneously, the repetition rate must not be too high. This can be understood as requiring constraints between keyframes while minimizing redundant information.

[0056] In one embodiment of the present invention, the first detection module 30 may use the Deeplab V3 plus algorithm to perform semantic segmentation on keyframes in the environmental image sequence.

[0057] In one embodiment of the present invention, the Deeplab V3 plus algorithm can be used to perform semantic segmentation on keyframes in an environmental image sequence. Specifically, the encoder architecture of the model consists of a backbone network ResNet101 and an ASPP module. The backbone network extracts image features to generate high-level semantic feature maps. The ASPP module uses the high-level semantic feature maps obtained from the backbone network to perform multi-scale sampling, generating multi-scale feature maps. At the end of the encoder, the multi-scale high-level semantic feature maps are combined along the channel dimension, and 1×1 convolution is used to reduce the channel dimensionality, maintaining the weight of the high-level semantic feature maps when concatenated, thus increasing the network's learning ability. Then, 3×3 convolution is used to extract features, and upsampling is performed at the end of the encoder to generate the final semantic segmentation map.

[0058] In one embodiment of the present invention, the object detection model is the SSD model. Using the SSD model, category information of objects in the environmental image can be obtained. Different objects are given different colors and bounding boxes are drawn to generate labels. A regression method is then used to establish the object detection framework, eliminating the step of generating candidate boxes and significantly improving the object detection speed.

[0059] In one embodiment of the present invention, the second construction module 40 uses an octree structure to store the indoor 3D point cloud map and incorporates object category labels into the indoor 3D point cloud map to construct an indoor 3D semantic map.

[0060] In one embodiment of the present invention, an octree structure can be used to store an indoor 3D point cloud map, and object category labels can be integrated into the indoor 3D point cloud map to construct an indoor 3D semantic map. An octree is a hierarchical data structure in 3D space, based on which space can be infinitely divided. The specific principle is as follows: each node in the octree represents a voxel, and its value is the size of the space occupied by that voxel. Each voxel can be divided into eight voxels of the same size, and this process is repeated until the smallest square is formed. The entire large square can be considered the root node, and the smallest square can be considered a "leaf node." In the octree, the map volume can be increased eightfold when moving from the lower level node to the upper level. If all nodes of a square are occupied or not occupied, there is no need to expand that node. In the octree space, the bottom-level leaves represent the lowest resolution of the map. Occupancy and unoccupiedness are expressed in probabilistic form. Let there be a node n, and its corresponding observed data z. The prior probability of a node being occupied is generally taken as 0.5, indicating that in an unknown environment, the prior probability of a node being occupied is equal to that of not being occupied. This is represented by P(n|z). 1:T Let t represent the observation probability of this node from the initial time to time t. Then, the probability of this node from the initial time to time t+1 is:

[0061]

[0062] The probability logarithm can be expressed as:

[0063] L(n|z 1:t+1 )=L(n|z 1:t )+L(n|z t+1 (2)

[0064] Wherein L(n|z 1:t L(n|z) represents the logarithmic probability of observation at that node from the initial time to time t. t+1 ) represents the logarithmic probability of the node from time t+1.

[0065] In one embodiment of the present invention, the role of the data acquisition robot is to obtain an environmental image sequence for constructing an indoor 3D semantic map, while the robot to be located is a robot that uses the environmental image sequence of the indoor 3D semantic map constructed in the above steps to achieve its own localization. The robot to be located and the data acquisition robot can be the same robot or different robots.

[0066] In one embodiment of the present invention, since the position of the identified object is fixed, the estimation module 70 can roughly estimate the position of the robot to be located indoors based on the position of the object in the room and in combination with the previously constructed three-dimensional semantic map.

[0067] In one embodiment of the present invention, the positioning module 80 can obtain the distance between the photographed object and the robot to be positioned by the time-of-flight method.

[0068] In one embodiment of the present invention, the positioning module 80 may employ time-of-flight (TOF) ranging technology in the Kinect V2 camera to calculate the distance between the camera and the object. TOF is an active multi-point depth information measurement method based on the time difference of flight of light. In the depth camera system, an infrared emitter continuously projects infrared measurement modulated light beams onto the object within its field of view. An internal timer calculates the emission time of each beam, and an infrared detector searches for the beams reflected back from the object in the incident direction. When the detector receives the returned beam, the timer stops. This method utilizes the known speed of light to obtain the distance information between the actual object and the depth camera plane corresponding to that pixel by measuring the depth value of a specified pixel in the depth image. The TOF measurement formula is:

[0069]

[0070] Where d is the distance between the object being measured and the depth camera, c is the flight speed of the emitted light beam in the measurement environment, f is the modulation frequency of the infrared light speed, and Δψ is the phase difference between the incident and reflected light beams. Since the approximate location of the robot to be located indoors is known, the planar distance between the robot and the photographed object is calculated using the time-of-flight method. Therefore, the precise location of the robot indoors can be calculated from the planar distances between the robot and any two objects indoors.

[0071] The indoor positioning device based on target detection and visual SLAM according to embodiments of the present invention constructs an indoor three-dimensional semantic map through SLAM algorithm, acquires environmental images of the location of the robot to be positioned using an RGB-D camera, and uses a target detection model to accurately position the robot, thereby improving the accuracy and real-time performance of robot positioning.

[0072] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. "A plurality of" means two or more, unless otherwise explicitly specified.

[0073] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0074] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0075] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0076] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0077] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0078] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0079] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0080] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0081] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An indoor localization method based on target detection and visual SLAM, characterized in that, Includes the following steps: The robot acquires a sequence of images of the entire indoor environment using an RGB-D camera. Based on the environmental image sequence, an indoor 3D point cloud map is constructed using the SLAM algorithm; Semantic segmentation is performed on keyframes in the environmental image sequence, and an object detection model is used to detect the keyframes to obtain the category information of objects in the keyframes; The category information of objects in the keyframes is fused into the indoor 3D point cloud map to construct an indoor 3D semantic map. The environmental image of the current location of the robot to be located is obtained by using an RGB-D camera carried by the robot to be located indoors. The target detection model is used to detect the environmental image of the current location of the robot to be located, so as to obtain the category information of objects in the environmental image of the current location of the robot to be located; Based on the category information of objects in the environmental image of the current location of the robot to be located and the indoor three-dimensional semantic map, the approximate location of the robot to be located in the indoor environment is estimated. The distance between the object being photographed and the robot being located is obtained by using the RGB-D camera carried by the robot to be located. Combined with the approximate position of the robot to be located in the room, the precise position of the robot to be located in the room is obtained.

2. The indoor positioning method based on target detection and visual SLAM according to claim 1, characterized in that, The Deeplab V3 plus algorithm is used to perform semantic segmentation on keyframes in the environmental image sequence, and the target detection model is the SSD model.

3. The indoor positioning method based on target detection and visual SLAM according to claim 2, characterized in that, The indoor 3D point cloud map is stored using an octree structure, and object category labels are incorporated into the indoor 3D point cloud map to construct the indoor 3D semantic map.

4. The indoor positioning method based on target detection and visual SLAM according to claim 3, characterized in that, The objects in the room are fixed objects, including one or more of the following: sofa, coffee table, dining table, bed, wardrobe, washing machine, and refrigerator.

5. The indoor positioning method based on target detection and visual SLAM according to claim 4, characterized in that, The distance between the photographed object and the robot to be located is obtained using the time-of-flight method.

6. An indoor positioning device based on target detection and visual SLAM, characterized in that, include: The first acquisition module is used to acquire a sequence of environmental images of the entire indoor environment by using an RGB-D camera carried by the robot. The first construction module is used to construct an indoor 3D point cloud map based on the environmental image sequence using the SLAM algorithm. The first detection module is used to perform semantic segmentation on keyframes in the environmental image sequence and to detect the keyframes using an object detection model to obtain the category information of objects in the keyframes. The second construction module is used to fuse the category information of objects in the keyframe into the indoor 3D point cloud map to construct an indoor 3D semantic map. The second acquisition module is used to acquire an environmental image of the current location of the robot to be located through an RGB-D camera carried by the robot to be located in the indoor environment. The second detection module is used to detect the environmental image of the current location of the robot to be located using the target detection model, so as to obtain the category information of objects in the environmental image of the current location of the robot to be located. An estimation module is used to estimate the approximate location of the robot to be located in the indoor environment based on the category information of objects in the environmental image of the current location of the robot to be located and the indoor three-dimensional semantic map. The positioning module is used to obtain the distance between the object photographed by the RGB-D camera carried by the robot to be positioned and the robot to be positioned, and combine it with the approximate position of the robot to be positioned in the room to obtain the precise position of the robot to be positioned in the room.

7. The indoor positioning device based on target detection and visual SLAM according to claim 6, characterized in that, The first detection module uses the Deeplab V3 plus algorithm to perform semantic segmentation on keyframes in the environmental image sequence, and the target detection model is the SSD model.

8. The indoor positioning device based on target detection and visual SLAM according to claim 7, characterized in that, The second construction module uses an octree structure to store the indoor 3D point cloud map and incorporates object category labels into the indoor 3D point cloud map to construct the indoor 3D semantic map.

9. The indoor positioning device based on target detection and visual SLAM according to claim 8, characterized in that, The objects in the room are fixed objects, including one or more of the following: sofa, coffee table, dining table, bed, wardrobe, washing machine, and refrigerator.

10. The indoor positioning device based on target detection and visual SLAM according to claim 9, characterized in that, The positioning module obtains the distance between the photographed object and the robot to be positioned using the time-of-flight method.

Images