Visual positioning method, device and equipment based on floor planar graph and robot

Abstract

The invention provides a visual positioning method, device and equipment based on a floor plan, and a robot. The visual positioning method comprises the following steps: acquiring an RGB-depth image corresponding to an indoor moving target at a current time point; extracting a semantic line in the RGB image by using a semantic line detection model; obtaining a three-dimensional point cloud of the semantic line in combination with the semantic line of the RGB image and the corresponding depth image; determining a local structure chart of the moving target at the current time point according to the three-dimensional point cloud of the semantic line; determining position coordinates of the moving target at the current time point based on the local structure chart at the current time point; the invention further provides a visual positioning device based on the floor planar graph, electronic equipment and an indoor mobile robot. The indoor scene does not need to be additionally mapped in advance, the floor plan is directly used for positioning and matching, and the cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of indoor positioning technology, and in particular to a visual positioning method, device, equipment and robot based on a floor plan. Background Art

[0002] Indoor positioning refers to the technology of determining and tracking the position of mobile targets (including moving objects or people) in indoor environments. The development of indoor positioning is of great significance for providing better user experience and realizing intelligent indoor navigation, security monitoring and other applications. At present, the main methods of indoor positioning include wireless signal positioning, inertial navigation, visual positioning and hybrid positioning methods.

[0003] Among them, visual positioning is an indoor positioning method based on camera images. Compared with other positioning methods, it has the advantages of low cost and is more in line with the idea of ​​human navigation. Visual positioning technology can determine the position of mobile devices relative to these feature points by identifying and matching feature points or landmarks in the scene. Visual positioning can use computer vision technology for feature extraction, matching and position estimation. Common methods include image matching based on feature descriptors (such as SIFT, SURF), SLAM (Simultaneous Localization and Mapping), etc. Visual positioning is widely used in indoor navigation, augmented reality and other fields. However, due to the influence of factors such as light and occlusion, visual positioning still has certain difficulties in complex environments. Moreover, visual positioning needs to map the scene before positioning, otherwise it cannot be positioned, but mapping will increase the workload and cost of the system. At the same time, since the room layout may change at any time, it is necessary to continuously update and maintain the established map, which brings additional costs.

[0004] Floor plans are ubiquitous in interior architectural environments, with floor plans available for nearly all modern buildings. They have a number of very useful properties: permanence, while furniture and objects in an environment may move or change, the structural information represented in a floor plan remains constant, so these documents are able to provide a stable description of an environment over years and decades; expressiveness, the structure depicted in a floor plan provides an excellent basis for understanding a particular interior environment; convenience, these blueprints express structure in a very lightweight format, making it easy to store and interpret in both physical and digital formats. Summary of the invention

[0005] The present invention aims to at least solve the technical problems of existing indoor visual positioning technology, which requires the establishment of an indoor scene map in advance and the need to update and maintain the map as the room layout changes, resulting in large workload and high maintenance costs, and provides a visual positioning method, device, equipment and robot based on a floor plan.

[0006] To achieve the purpose of the present invention, in a first aspect of the present invention, a visual positioning method based on a floor plan is provided, including: obtaining an RGB-depth image corresponding to a mobile target in a room at a current time point, the RGB-depth image including a mutually corresponding RGB image and a depth image; extracting semantic lines in the RGB image using a pre-trained semantic line detection model; obtaining a three-dimensional point cloud of the semantic lines in combination with the semantic lines of the RGB image and the corresponding depth image; determining a local structure diagram of the mobile target at the current time point according to the three-dimensional point cloud of the semantic lines; determining the position coordinates of the mobile target at the current time point based on the local structure diagram at the current time point, including: matching and positioning the local structure diagram at the current time point with the floor plan to obtain a transformation matrix of the mobile target from an initial position to a position at the current time point; calculating the position coordinates of the mobile target at the current time point through the transformation matrix and the initial position coordinates; or, matching and positioning the local structure diagram at the current time point with the local structure diagram at the previous time point to obtain a transformation matrix of the mobile target from a position at the previous time point to a position at the current time point; calculating the position coordinates of the mobile target at the current time point through the transformation matrix and the position coordinates of the mobile target at the previous time point.

[0007] Preferably, the RGB-depth image corresponding to the moving target at the current time point is formed by stitching together multiple RGB-depth sub-images synchronously shot by multiple RGB-D cameras.

[0008] Preferably, the RGB-depth image corresponding to the moving target at the current time point is a mutually corresponding RGB sub-image sequence and a depth sub-image sequence collected within a time neighborhood of the moving target at the current time point, the RGB image is a RGB sub-image sequence, and the depth image is a depth sub-image sequence;

[0009] The method of extracting semantic lines from an RGB image using a pre-trained semantic line detection model includes: extracting semantic lines from each RGB sub-image in an RGB sub-image sequence using the pre-trained semantic line detection model;

[0010] The method combines the semantic lines of the RGB image and the corresponding depth image to obtain the three-dimensional point cloud of the semantic lines, including: combining the semantic lines of each RGB sub-image and the depth sub-image corresponding to each RGB sub-image to obtain a subset of the three-dimensional point cloud of the semantic lines of each RGB sub-image, and splicing the three-dimensional point cloud subsets of the semantic lines of all RGB sub-images to obtain the three-dimensional point cloud of the semantic lines.

[0011] Preferably, the semantic line detection model includes:

[0012] Feature pyramid network, used to obtain multi-layer feature maps of RGB images input to the semantic line detection model;

[0013] A Hough transform unit is used to convert multiple layers of feature maps into parameter spaces respectively to obtain multiple parameter space features corresponding to the multiple layers of feature maps;

[0014] Multi-layer convolutional network uses multi-layer convolution to perform convolution processing on multiple parameter space features in parallel to obtain multiple convolution features;

[0015] A concatenation unit concatenates multiple convolutional features to obtain concatenated features;

[0016] Prediction convolution layer, using convolution to predict the splicing features point by point to obtain the probability that each point in the splicing features belongs to the semantic line;

[0017] A threshold screening unit, screening out points whose probability of belonging to the semantic line is greater than or equal to a preset probability threshold in the splicing features, to obtain screening features;

[0018] The inverse Hough transform unit performs inverse Hough transform on the filtered features to obtain the semantic lines of the RGB image input to the semantic line detection model.

[0019] Preferably, the loss function of the semantic line detection model in the training process is:

[0020] L=-∑ i {G i ·log(P i )+(1-G i )·log(1-P i )};

[0021] Among them, G i represents the true value of the i-th point on the semantic line after the calibrated semantic line in the RGB image sample is converted to the parameter space and Gaussian smoothed; P i It represents the probability that the i-th point belongs to the semantic line after the RGB image sample is input into the semantic line detection model.

[0022] Preferably,

[0023] The determining of the local structure diagram of the moving target at the current time point according to the three-dimensional point cloud of the semantic line includes:

[0024] According to the Harmanton hypothesis, the normal vector of the indoor ceiling or ground is taken as the gravity direction, and point clouds in two directions other than the gravity direction among the three main directions of the Harmanton hypothesis are extracted from the three-dimensional point cloud of the semantic line;

[0025] Based on the extracted point cloud, the local structure map of the moving target at the current time point is obtained.

[0026] Preferably, the conversion matrix includes a rotation matrix and a translation matrix of the mobile target from an initial position or a position at a previous time point to a position at a current time point.

[0027] To achieve the purpose of the present invention, according to a second aspect of the present invention, a visual positioning device based on a floor plan is provided, which is used to implement the visual positioning method based on a floor plan described in the first aspect of the present invention, comprising:

[0028] An image acquisition module is used to acquire an RGB-depth image corresponding to a moving target in the room at a current time point, wherein the RGB-depth image includes an RGB image and a depth image corresponding to each other;

[0029] The semantic line extraction module uses the pre-trained semantic line detection model to extract semantic lines in RGB images;

[0030] The semantic line 3D point cloud acquisition module combines the semantic line of the RGB image and the corresponding depth image to obtain the 3D point cloud of the semantic line;

[0031] A local structure map acquisition module determines the local structure map of the mobile target at the current time point based on the three-dimensional point cloud of the semantic line;

[0032] The mobile target position coordinate calculation module determines the position coordinates of the mobile target at the current time point based on the local structure diagram at the current time point, including:

[0033] The local structure diagram at the current time point and the floor plan are matched and positioned to obtain a transformation matrix of the mobile target from the initial position to the current time point; the position coordinates of the mobile target at the current time point are calculated using the transformation matrix and the initial position coordinates;

[0034] or,

[0035] The local structure diagram at the current time point is matched and located with the local structure diagram at the previous time point to obtain the transformation matrix of the mobile target from the position at the previous time point to the position at the current time point; the position coordinates of the mobile target at the current time point are calculated by the transformation matrix and the position coordinates of the mobile target at the previous time point.

[0036] To achieve the purpose of the present invention, according to the third aspect of the present invention, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the visual positioning method based on the floor plan described in the first aspect of the present invention.

[0037] To achieve the purpose of the present invention, according to the fourth aspect of the present invention, there is provided an indoor mobile robot, comprising a robot body, a processor installed on the robot body and an RGB-D camera, the processor reads an RGB-depth image sequence from the RGB-D camera, and performs indoor positioning of the mobile robot according to the visual positioning method based on the floor plan described in the first aspect of the present invention, wherein the mobile target is the mobile robot.

[0038] In the indoor positioning technical solution provided by the present invention, firstly, the semantic lines (i.e., the contour lines in the indoor environment, which may include the boundary lines between the wall and the ground or the roof surface, the corner lines of the wall, etc.) existing in the RGB image in the RGB-depth image at the current time point are obtained, and the semantic lines are mapped to the depth image corresponding to the RGB image to obtain the three-dimensional point cloud of the semantic lines. Based on the Harmanton hypothesis, the point clouds of three main directions are extracted from the three-dimensional point cloud of the semantic lines, and the local structure diagram of the mobile target within the viewing angle at the current time point is obtained by using the point clouds of the three main directions, thereby reducing the influence of the point clouds of non-main directions on the local structure diagram. The local structure diagram at the current time point and the floor plan or the local structure diagram at the previous time point are matched and positioned to obtain a conversion matrix, and finally the position coordinates of the mobile target at the current time point are calculated using the conversion matrix and the initial position coordinates or the position coordinates of the mobile target at the previous time point. It can be seen that the present invention does not need to build a map of the indoor scene in advance, and directly uses the long-term effective floor plan as the positioning matching reference, thereby reducing the cost. At the same time, in the local structure diagram, the point clouds of only three main wind directions are extracted using the Harmanton hypothesis to generate the local structure diagram, so that the expression style of the local structure diagram is close to the floor plan, thereby improving the positioning matching accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is an architecture diagram of a visual positioning method based on a floor plan in a preferred embodiment of the present invention;

[0040] Figure 2 is a step diagram of a visual positioning method based on a floor plan in a preferred embodiment of the present invention;

[0041] Figure 3 It is a logic diagram of a visual positioning method based on a floor plan in a preferred embodiment of the present invention;

[0042] Figure 4 is a structural schematic diagram of a semantic line detection model in a preferred embodiment of the present invention;

[0043] Figure 5 It is a schematic diagram of the Hough transform principle;

[0044] Figure 6 It is a schematic diagram of matching positioning in a visual positioning method based on a floor plan in a preferred embodiment of the present invention;

[0045] Figure 7 It is a schematic structural diagram of an electronic device in a preferred embodiment of the present invention.

[0046] Reference numerals:

[0047] 1 Feature pyramid network; 2 Hough transform unit; 3 Multi-layer convolutional network; 4 Splicing unit; 5 Prediction convolution layer; 6 Threshold screening unit; 7 Inverse Hough transform unit. DETAILED DESCRIPTION

[0048] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0049] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0050] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0051] The execution subject of the visual positioning method based on the floor plan includes but is not limited to at least one of the electronic devices such as the server and the terminal that can be configured to execute the method provided in the embodiment of the present application. In other words, the visual positioning method based on the floor plan can be executed by software or hardware installed on the terminal device or the server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0052] In a preferred embodiment, referring to Figures 1 to 3 The visual positioning method based on the floor plan provided by the present invention includes:

[0053] Step S1, obtaining an RGB-depth image corresponding to a moving target in a room at a current time point, wherein the RGB-depth image includes an RGB image and a depth image corresponding to each other.

[0054] In this embodiment, the mobile target is preferably, but not limited to, a mobile robot, a mobile vehicle, or a person carrying or installing an image acquisition device. The image acquisition device is preferably, but not limited to, an RGB-D camera that can synchronously acquire RGB images or depth images. The image acquisition can be located in front of or on top of the mobile target to acquire the front image and depth information.

[0055] Step S2, using the pre-trained semantic line detection model to extract semantic lines in the RGB image. Semantic lines are the contour lines in the room, including the boundary between the wall and the ground or roof, the corner line of the wall, etc. The semantic lines of the RGB image obtained at this time do not include depth information. Figure 4 As shown, it is not yet possible to match the floor plan.

[0056] Step S3, combining the semantic line of the RGB image and the corresponding depth image to obtain the three-dimensional point cloud of the semantic line. Specifically, the semantic line of the RGB image is mapped to the corresponding depth image, the depth value of the pixel point corresponding to the semantic line is obtained, and the depth value of the pixel point on the semantic line is subjected to coordinate transformation processing to obtain the three-dimensional point cloud of the semantic line, wherein the coordinate transformation processing method can refer to the existing method of converting the depth image into a three-dimensional point cloud, such as the depth image conversion into a three-dimensional point cloud technology disclosed in the Chinese patent with publication number CN111223053A, which will not be repeated here. The coordinates of the midpoint of the three-dimensional point cloud of the semantic line include coordinates in three axes: the floor height direction, the horizontal direction of the floor plane, and the vertical direction of the floor plane.

[0057] Step S4, determining the local structure diagram of the mobile target at the current time point based on the three-dimensional point cloud of the semantic line. Preferably, in order to make the expression style of the obtained local structure diagram consistent with the floor plan, step S4 includes: according to the Harmanton hypothesis, taking the normal vector of the indoor ceiling or ground as the gravity direction (vertical direction, such as the direction of the vertical structure of the indoor wall), extracting point clouds in two directions other than the gravity direction from the three main directions of the Harmanton hypothesis from the three-dimensional point cloud of the semantic line; obtaining the local structure diagram of the mobile target at the current time point based on the extracted point cloud, specifically using the extracted point cloud diagram as the local structure diagram.

[0058] Step S5, determining the position coordinates of the moving target at the current time point based on the local structure diagram at the current time point, includes:

[0059] Step S5A, matching and positioning the local structure diagram at the current time point with the floor plan to obtain a transformation matrix of the mobile target from the initial position to the current time point; calculating the position coordinates of the mobile target at the current time point through the transformation matrix and the initial position coordinates;

[0060] or,

[0061] Step S5B, match and locate the local structure diagram at the current time point with the local structure diagram at the previous time point to obtain the transformation matrix of the mobile target from the position at the previous time point to the position at the current time point; calculate the position coordinates of the mobile target at the current time point through the transformation matrix and the position coordinates of the mobile target at the previous time point.

[0062] In this embodiment, step S5A and step S5B implement two parallel schemes of step S5. Step S5A is global positioning based on the initial position, and the initial position is preferably but not limited to the center point of the indoor doorway. Step S5B is mobile sliding positioning based on the position at the last time point. The initial position coordinates and the position coordinates of the mobile target at the last time point are stored in advance. During the indoor activities of the mobile target, the above steps S1 to S5 can be executed regularly or in real time to achieve indoor positioning of the mobile target.

[0063] In a preferred embodiment, in order to avoid too little semantic line information contained in a single RGB-depth image, that is, too little room contour line information in the field of view, and inability to perform effective positioning and matching, the RGB-depth image corresponding to the moving target at the current time point is a splicing of multiple RGB-depth sub-images taken synchronously by multiple RGB-D cameras. Multiple RGB-D cameras can be placed in parallel and take images synchronously to expand the field of view and capture enough semantic line information.

[0064] In a preferred embodiment, in order to avoid too little semantic line information contained in a single RGB-depth image, that is, too little room contour line information in the field of view, and inability to perform effective positioning and matching, the RGB-depth image corresponding to the mobile target at the current time point is a mutually corresponding RGB sub-image sequence and a depth sub-image sequence collected in the time neighborhood of the mobile target at the current time point, the RGB image is an RGB sub-image sequence, the depth image is a depth sub-image sequence, and the images in the RGB sub-image sequence and the depth sub-image sequence have time continuity, which can reduce the hardware cost compared with the previous preferred embodiment. The time neighborhood at the current time point can be expressed as: t-Δt, t represents the current time, and Δt represents the neighborhood size. Δt can be determined according to the camera image acquisition frame rate, and can be obtained by dividing the preset number of RGB images of the RGB sub-image sequence by the camera image acquisition frame rate by rounding, and the preset number of RGB images can be negatively correlated with the moving speed of the mobile target, which can ensure that the acquired local structure map matches the camera field of view range of the mobile target at the current time point, thereby improving the positioning accuracy and real-time performance. The number of preset RGB images is greater than or equal to 2 and less than or equal to 20. Preferably, the number of preset RGB images is 5.

[0065] At this time, in step S2, the semantic lines in the RGB image are extracted using the pre-trained semantic line detection model, including: extracting the semantic lines of each RGB sub-image in the RGB sub-image sequence using the pre-trained semantic line detection model.

[0066] At this time, in step S3, the semantic lines of the RGB image and the corresponding depth image are combined to obtain a three-dimensional point cloud of the semantic lines, including: combining the semantic lines of each RGB sub-image and the depth sub-image corresponding to each RGB sub-image to obtain a three-dimensional point cloud subset of the semantic lines of each RGB sub-image, splicing the three-dimensional point cloud subsets of the semantic lines of all RGB sub-images to obtain a three-dimensional point cloud of the semantic lines, and during the splicing process, filtering out duplicate points to form a three-dimensional point cloud of semantic lines without duplicate coordinates.

[0067] In a preferred embodiment, in order to more accurately extract semantic lines from RGB images, refer to Figure 4 As shown, the semantic line detection model includes:

[0068] The feature pyramid network 1 is used to obtain a multi-layer feature map of the RGB image (which may be the RGB sub-image or the spliced ​​RGB image in the above preferred embodiment) input to the semantic line detection model. Figure 4As shown in the figure, the feature pyramid network 1 (FPN network) is a multi-layer structure. Considering the balance between performance and computational complexity, a four-layer FPN network is selected. The feature pyramid network 1 is used as an encoder. The resolutions of the four-layer FPN network are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input RGB image resolution, respectively. Each RGB image obtains 4 layers of feature maps with different resolutions.

[0069] Hough transform unit 2 is used to transform the multi-layer feature maps into parameter space respectively, and obtain multiple parameter space features corresponding to the multi-layer feature maps. Using Hough transform, the feature map of each layer is transformed into parameter space to obtain the parameter space features corresponding to the feature map of the layer. The principle of Hough transform is as follows Figure 5 As shown, Figure 5 The straight line in can be represented by the distance r from the coordinates of the origin l The angle θ between the origin and the perpendicular line to the horizontal X-axis l A straight line in a two-dimensional plane can be uniquely determined by these two parameters, while a point in a two-dimensional plane is determined by a series of r l ,θ l Therefore, the following mapping relationship can be achieved:

[0070] r l ,θ l ←P(l)

[0071] l←P -1 (r l ,θ l )

[0072] By moving the point of the two-dimensional plane to r l ,θ l plane (is a new plane in the parameter space, the horizontal and vertical coordinates of the plane become r,θ, a series of r l ,θ l The coordinates of the points on the new plane) can be mapped to get the same number of straight lines. The intersection of the straight lines is the parameter corresponding to the straight lines on the two-dimensional plane. The process of mapping the two-dimensional plane is called Hough transform, the space after Hough transform is called parameter space, and the process of mapping the parameter space back to the two-dimensional plane is called inverse Hough transform.

[0073] The multi-layer convolution network 3 uses multi-layer convolution to perform convolution processing on multiple parameter space features in parallel to obtain multiple convolution features. Each parameter space feature obtains a corresponding convolution feature. The convolution kernel size of the convolution layer in the multi-layer convolution network 3 can be 3×3. The convolution layer is used in the multi-layer convolution network 3 to aggregate context features.

[0074] The stitching unit 4 stitches multiple convolution features to obtain a stitching feature. To facilitate stitching, a bilinear interpolation method can be used to unify the resolution of multiple convolution features, and then stitching is performed after unification. The stitching method is preferably but not limited to summing or averaging the feature values ​​of each pixel.

[0075] Prediction convolution layer 5 uses convolution to predict the splicing features point by point to obtain the probability that each point in the splicing features belongs to the semantic line. The convolution kernel size in the prediction convolution layer 5 can be 1×1, which is used to predict the splicing features point by point to obtain the prediction probability. The prediction probability of a point represents the probability that the point belongs to the semantic line.

[0076] The threshold screening unit 6 screens out points whose probability of belonging to the semantic line is greater than or equal to a preset probability threshold in the splicing features to obtain screening features. The preset probability value can be manually set based on experience, and its value range can be 0.6 to 0.9. Figure 5 As shown, the screening features include sparse points whose probability of belonging to the semantic line is greater than or equal to a preset probability threshold.

[0077] The inverse Hough transform unit 7 performs an inverse Hough transform on the selected features to obtain semantic lines, such as Figure 5 As shown, the semantic lines include the contour lines in the room, specifically the boundary lines between the wall and the ground and the ceiling, the bending boundary lines of the wall, etc.

[0078] In a preferred embodiment, in order to improve the accuracy of the semantic line detection model in capturing the semantic line, the training process of the semantic line detection model includes:

[0079] Step 1, obtain an RGB image sample set; calibrate semantic lines in each RGB image in the RGB image sample set, convert the calibrated semantic lines into parameter space using Hough transform, and then perform Gaussian smoothing on the semantic lines in the parameter space. Number the points on the calibrated semantic lines in each RGB image, and the number index is i, where i is a positive integer. The values ​​of the points on the semantic lines after Gaussian smoothing are used as supervisory data for RGB image sample training.

[0080] Step 2, construct the network structure of the semantic line detection model and initialize the network parameters;

[0081] Step 3, divide the RGB image sample set into a training set, a validation set, and a test set;

[0082] Step 4, use the training set to train the network structure of the semantic line detection model, and during the training process, take the minimum value of the loss function as the goal, and use the gradient descent method to continuously update the network parameters of the network structure of the semantic line detection model until the training stop condition is reached, such as reaching the maximum number of training times or the loss function value is minimized. After the training stop condition is reached, the final network parameters of the network structure of the semantic line detection model are output.

[0083] Step 5: Use the validation set to validate the trained semantic line detection model. If the validation passes, the training ends and the process goes to step 6. If the validation fails, the hyperparameters (learning rate, number of network layers, etc.) are adjusted and the process goes back to step 4. Step 6: Use the test set to validate the trained semantic line detection model and output the network parameters of the semantic line detection model obtained in step 4.

[0084] In a preferred embodiment, the loss function of the semantic line detection model during the training process is:

[0085] L=-∑ i {G i ·log(P i )+(1-G i )·log(1-P i )};

[0086] Among them, G i It represents the true value of the original i-th point on the semantic line after the semantic line marked in the RGB image sample is converted to the parameter space and Gaussian smoothing is performed. When the i-th point exists after Gaussian smoothing (that is, it is not a background pixel), the true value is 1; when the i-th point does not exist after Gaussian smoothing (that is, it is a background pixel), the true value is 0; P i It represents the probability that the i-th point belongs to the semantic line after the RGB image sample is input into the semantic line detection model.

[0087] In a preferred embodiment, the transformation matrix includes a rotation matrix R and a translation matrix t of the mobile target from the initial position or the position at the previous time point to the current time point. Further preferably, the position coordinate A of the mobile target at the current time point is calculated according to the following formula:

[0088]

[0089] Where B represents the position coordinates of the moving target at the previous time node or the position coordinates of the moving target at the initial position; R 12 Represents the first row and second column element of the rotation matrix R, t 12 Represents the first row and second column element of the translation matrix t.

[0090] In this embodiment, in order to achieve correct alignment, matching and positioning of the local structure diagram at the current time point with the floor plan or the local structure diagram at the previous time point, in step S5, the local structure diagram at the current time point and the floor plan or the local structure diagram at the previous time point are matched and positioned to obtain the transformation matrix of the moving target from the previous time point to the current time point, and the transformation matrix of the moving target from the previous time point to the current time point is obtained by referring to the local structure diagram at the current time point. Figure 6 As shown, specifically including:

[0091] Use the existing ICP algorithm (pose estimation algorithm for multiple 3D-3D points, the Z axis coordinate value can be fixed) to perform pose estimation and positioning. The floor plan can be obtained from the construction documents of the building or public maps. Define the error term of the i-th point in space, e i =p i -(Rp' i +t), construct the least squares problem and find R,t with the smallest sum of squared errors. That is:

[0092]

[0093] In the formula, p i , p' i They represent the spatial three-dimensional coordinates of the i-th point in the floor plan / local structure diagram at the previous time point and the local structure diagram at the current time point, respectively. R, t represent the rotation matrix and translation matrix, respectively.

[0094] The present invention also discloses a visual positioning device based on a floor plan, which is used to implement the visual positioning method based on a floor plan. In a preferred embodiment, the device includes:

[0095] The image acquisition module acquires the RGB-depth image corresponding to the indoor moving target at the current time point, and the RGB-depth image includes the corresponding RGB image and depth image.

[0096] The semantic line extraction module uses the pre-trained semantic line detection model to extract semantic lines in RGB images.

[0097] The semantic line 3D point cloud acquisition module combines the semantic lines of the RGB image and the corresponding depth image to obtain the 3D point cloud of the semantic line.

[0098] A local structure map acquisition module determines the local structure map of the mobile target at the current time point based on the three-dimensional point cloud of the semantic line;

[0099] The mobile target position coordinate calculation module determines the position coordinates of the mobile target at the current time point based on the local structure diagram at the current time point, including:

[0100] The local structure diagram at the current time point and the floor plan are matched and positioned to obtain a transformation matrix of the mobile target from the initial position to the current time point; the position coordinates of the mobile target at the current time point are calculated using the transformation matrix and the initial position coordinates;

[0101] or,

[0102] The local structure diagram at the current time point is matched and located with the local structure diagram at the previous time point to obtain the transformation matrix of the mobile target from the position at the previous time point to the position at the current time point; the position coordinates of the mobile target at the current time point are calculated by the transformation matrix and the position coordinates of the mobile target at the previous time point.

[0103] In this embodiment, the image acquisition module, semantic line extraction module, semantic line three-dimensional point cloud acquisition module, local structure map acquisition module and mobile target position coordinate calculation module respectively correspond to steps S1 to S5 of the above-mentioned visual positioning method based on floor plan, and will not be repeated here.

[0104] The present invention also discloses an indoor mobile robot. In a preferred embodiment, the indoor mobile robot includes a robot body, a processor installed on the robot body, and an RGB-D camera. The processor reads an RGB-depth image sequence from the RGB-D camera and performs indoor positioning of the mobile robot according to the visual positioning method based on the floor plan. The mobile target is the mobile robot. The RGB-D camera can simultaneously collect corresponding RGB image sequences and depth image sequences.

[0105] In this embodiment, the robot body is preferably, but not limited to, a wheeled robot or a tracked robot. The processor pre-stores the floor plan of the mobile robot's activity space, the initial position coordinates, and the position coordinates of the mobile target at each historical time point. During the movement of the mobile robot, the visual positioning method based on the floor plan is executed periodically or in real time to obtain the current position coordinates of the mobile robot.

[0106] The present invention also discloses a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the above-mentioned floor plan-based visual positioning method provided by the present invention. The computer program product should be understood as a software product that mainly implements its solution through a computer program, such as a program product integrated in the cloud or a software library.

[0107] The present invention also discloses an electronic device. In one embodiment, the electronic device includes at least one processor; and a memory connected to the at least one processor in communication;

[0108] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the visual positioning method based on the floor plan provided by the present invention.

[0109] like Figure 7 , which is a schematic diagram of the structure of an electronic device for a visual positioning method based on a floor plan provided by an embodiment of the present invention. The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a visual positioning method program based on a floor plan.

[0110] In some embodiments, the processor 10 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 11 (for example, executing a visual positioning method based on a floor plan, etc.), and calls data stored in the memory 11 to execute various functions of the electronic device and process data.

[0111] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code of the visual positioning method program based on the floor plan, but also can be used to temporarily store data that has been output or is to be output.

[0112] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0113] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0114] Figure 7 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 7 The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0115] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, and power status indicators. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0116] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited by this structure.

[0117] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0118] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", "an implementation", "a preferred implementation" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0119] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A visual positioning method based on floor plan, characterized in that: include: Acquire an RGB-depth image corresponding to a moving target in the room at a current time point, wherein the RGB-depth image includes an RGB image and a depth image corresponding to each other; Extract semantic lines from RGB images using a pre-trained semantic line detection model; Combine the semantic lines of the RGB image and the corresponding depth image to obtain the 3D point cloud of the semantic lines; Determine the local structure diagram of the moving target at the current time point according to the three-dimensional point cloud of the semantic line; Determining the position coordinates of the moving target at the current time point based on the local structure diagram at the current time point includes: The local structure diagram at the current time point and the floor plan are matched and positioned to obtain the transformation matrix of the moving target from the initial position to the position at the current time point; Calculate the position coordinates of the moving target at the current time point through the transformation matrix and the initial position coordinates; or, The local structure diagram at the current time point is matched and located with the local structure diagram at the previous time point to obtain the transformation matrix of the mobile target from the position at the previous time point to the position at the current time point; the position coordinates of the mobile target at the current time point are calculated by the transformation matrix and the position coordinates of the mobile target at the previous time point.

2. The visual positioning method based on floor plan as claimed in claim 1, characterized in that: The RGB-depth image corresponding to the moving target at the current time point is formed by stitching together multiple RGB-depth sub-images synchronously taken by multiple RGB-D cameras.

3. The visual positioning method based on floor plan as claimed in claim 1, characterized in that: The RGB-depth image corresponding to the mobile target at the current time point is a mutually corresponding RGB sub-image sequence and a depth sub-image sequence collected within the time neighborhood of the mobile target at the current time point, the RGB image is a RGB sub-image sequence, and the depth image is a depth sub-image sequence; The method of extracting semantic lines from an RGB image using a pre-trained semantic line detection model includes: extracting semantic lines from each RGB sub-image in an RGB sub-image sequence using the pre-trained semantic line detection model; The method combines the semantic lines of the RGB image and the corresponding depth image to obtain the three-dimensional point cloud of the semantic lines, including: combining the semantic lines of each RGB sub-image and the depth sub-image corresponding to each RGB sub-image to obtain a subset of the three-dimensional point cloud of the semantic lines of each RGB sub-image, and splicing the three-dimensional point cloud subsets of the semantic lines of all RGB sub-images to obtain the three-dimensional point cloud of the semantic lines.

4. The visual positioning method based on floor plan according to any one of claims 1 to 3, characterized in that: The semantic line detection model includes: Feature pyramid network, used to obtain multi-layer feature maps of RGB images input to the semantic line detection model; A Hough transform unit is used to convert multiple layers of feature maps into parameter spaces respectively to obtain multiple parameter space features corresponding to the multiple layers of feature maps; Multi-layer convolutional network uses multi-layer convolution to perform convolution processing on multiple parameter space features in parallel to obtain multiple convolution features; A concatenation unit concatenates multiple convolutional features to obtain concatenated features; Prediction convolution layer, using convolution to predict the splicing features point by point to obtain the probability that each point in the splicing features belongs to the semantic line; A threshold screening unit, screening out points whose probability of belonging to the semantic line is greater than or equal to a preset probability threshold in the splicing features, to obtain screening features; The inverse Hough transform unit performs inverse Hough transform on the filtered features to obtain the semantic lines of the RGB image input to the semantic line detection model.

5. The visual positioning method based on floor plan as claimed in claim 4, characterized in that: The loss function of the semantic line detection model in the training process is: L=-∑ i {G i ·log(P i )+(1-G i )·log(1-P i )}; Among them, G i represents the true value of the original i-th point on the semantic line after the semantic line marked in the RGB image sample is converted to the parameter space and Gaussian smoothing is performed; P i It represents the probability that the i-th point belongs to the semantic line after the RGB image sample is input into the semantic line detection model.

6. The visual positioning method based on floor plan as claimed in any one of claims 1 to 3 and 5, characterized in that: The determining of the local structure diagram of the moving target at the current time point according to the three-dimensional point cloud of the semantic line includes: According to the Harmanton hypothesis, the normal vector of the indoor ceiling or ground is taken as the gravity direction, and point clouds in two directions other than the gravity direction among the three main directions of the Harmanton hypothesis are extracted from the three-dimensional point cloud of the semantic line; Based on the extracted point cloud, the local structure map of the moving target at the current time point is obtained.

7. The visual positioning method based on floor plan as claimed in claim 1, characterized in that: The conversion matrix includes a rotation matrix and a translation matrix of the mobile target from an initial position or a position at a previous time point to a position at a current time point.

8. A visual positioning device based on a floor plan, characterized in that: The method for visual positioning based on a floor plan according to any one of claims 1 to 7 comprises: An image acquisition module is used to acquire an RGB-depth image corresponding to a moving target in the room at a current time point, wherein the RGB-depth image includes an RGB image and a depth image corresponding to each other; The semantic line extraction module uses the pre-trained semantic line detection model to extract semantic lines in RGB images; The semantic line 3D point cloud acquisition module combines the semantic line of the RGB image and the corresponding depth image to obtain the 3D point cloud of the semantic line; A local structure map acquisition module determines the local structure map of the mobile target at the current time point based on the three-dimensional point cloud of the semantic line; The mobile target position coordinate calculation module determines the position coordinates of the mobile target at the current time point based on the local structure diagram at the current time point, including: The local structure diagram at the current time point and the floor plan are matched and positioned to obtain a transformation matrix of the mobile target from the initial position to the current time point; the position coordinates of the mobile target at the current time point are calculated using the transformation matrix and the initial position coordinates; or, The local structure diagram at the current time point is matched and located with the local structure diagram at the previous time point to obtain the transformation matrix of the mobile target from the position at the previous time point to the position at the current time point; the position coordinates of the mobile target at the current time point are calculated by the transformation matrix and the position coordinates of the mobile target at the previous time point.

9. An electronic device, characterized in that: The electronic device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor so that the at least one processor can execute the floor plan-based visual positioning method described in any one of claims 1 to 7.

10. An indoor mobile robot, characterized in that: The invention comprises a robot body, a processor installed on the robot body and an RGB-D camera, wherein the processor reads an RGB-depth image sequence from the RGB-D camera and performs indoor positioning of the mobile robot according to the visual positioning method based on the floor plan as described in any one of claims 1 to 7, wherein the mobile target is a mobile robot.

Images