Method and device for obtaining position of indoor object

Abstract

The present invention relates to a method and device for obtaining the position of indoor objects, which belongs to the field of indoor space positioning technology, and solves the problem that the existing indoor object positioning position error is large and cannot effectively reflect the recording and change of building information. Multi-sensors are calibrated in high-precision space to obtain the external parameters from the local coordinate system of the camera to the center of the basic coordinate system; the laser point set and image point set of the indoor space are obtained by multi-sensors to locate the multi-line laser radar; the laser point set and image point set of the object to be identified located in the indoor space are obtained by multi-sensors, and the position of the object to be identified in the camera coordinate system is detected by the target recognition model, an image feature extractor for extracting visual semantic features and a fusion module for effectively fusing laser radar features and visual semantic features; and the absolute position of the object to be identified in the world coordinate system is obtained by coordinate transformation. It can reduce the positioning position error of indoor objects.

Description

Technical Field

[0001] The present invention relates to the technical field of indoor space positioning, and in particular to a method and device for obtaining the position of an indoor object. Background Art

[0002] LiDAR is a new type of measuring instrument that uses the principle of laser ranging to determine the space to be scanned. The straight-line distance from the laser to the target is obtained by measuring the phase difference (time difference) between the laser emission signal and the reflected signal of the target point by point. At the same time, the spatial position of the target point is obtained based on the direction of the emitted laser signal and the spatial position of the laser. By densely scanning the surface of the object to be scanned with a laser, a three-dimensional surface model of the object can be obtained.

[0003] BIM technology is also one of the hottest topics in science and technology. Its three-dimensional images with additional information make the visualization of the entire life cycle of buildings a reality. From this, we can derive an idea: Can we develop a system that has both the accuracy and measurability of laser point clouds and the informativeness and flexibility of BIM models? At present, the indoor space positioning technology on the market, whether using GPS or Bluetooth, has errors beyond the ideal range, which makes it impossible to effectively reflect the recording and changes of building information. Summary of the invention

[0004] In view of the above analysis, the embodiments of the present invention aim to provide a method and device for obtaining the position of an indoor object, so as to solve the problem that the positioning position error of the existing indoor objects is large and thus cannot effectively reflect the recording and change of building information.

[0005] On the one hand, an embodiment of the present invention provides a method for obtaining the position of an indoor object, comprising: performing high-precision spatial calibration on multiple sensors to obtain external parameters from a local camera coordinate system to a base coordinate center, wherein the multiple sensors include a multi-line laser radar, a fixed laser radar, and a camera; obtaining a laser point set and an image point set of an indoor space through the multiple sensors to locate the multi-line laser radar; obtaining a laser point set and an image point set of an object to be identified located in the indoor space through the multiple sensors, and detecting the position of the object to be identified in the camera coordinate system using a target recognition model, wherein the target recognition model includes a point cloud feature extractor for extracting laser radar features, an image feature extractor for extracting visual semantic features, and a fusion module for effectively fusing the laser radar features and the visual semantic features; and obtaining the absolute position of the object to be identified in the world coordinate system through coordinate transformation.

[0006] The beneficial effects of the above technical solution are as follows: This application can significantly reduce the error of indoor object positioning by high-precision spatial calibration of multiple sensors, positioning the multi-line laser radar, and using the target recognition model to detect the position of the object to be identified in the camera coordinate system. In addition, the absolute position of the object to be identified in the world coordinate system is obtained through coordinate transformation, so that the target to be identified can be added to the building information model BIM, so as to effectively reflect the recording and changes of building information.

[0007] Based on the further improvement of the above method, the lidar features include the geometric structure and depth information of the point cloud; the visual semantic features include the color and texture information of the image; and the fusion module is used to fuse the geometric structure and depth information of the point cloud with the color and texture information of the image to generate a 3D model of the object to be identified.

[0008] Based on the further improvement of the above method, the point cloud feature extractor includes a multi-scale point cloud feature extractor, which is used to extract the geometric structure and depth information of the multi-scale point cloud; the image feature extractor includes a multi-scale image feature extractor, which is used to extract the color and texture information of the multi-scale image; and the fusion module includes a multi-scale fusion module, which is used to fuse the geometric structure and depth information of point clouds of different scales with the color and texture information of images of corresponding scales layer by layer.

[0009] Based on the further improvement of the above method, acquiring the laser point set and image point set of the object to be identified located in the indoor space through the multi-sensor, and using the target recognition model to detect the position of the object to be identified in the camera coordinate system further includes: establishing a deep learning neural network, and training the deep learning neural network with the marked visible light image and the marked laser point cloud map to obtain the target recognition model; using the camera to shoot the visible light image of the target to be identified and using the multi-line laser radar and the fixed laser radar to scan the laser point cloud map of the target to be identified, and inputting the visible light image of the target to be identified and the laser point cloud map of the target to be identified into the target recognition model to obtain the type of the target to be identified and the position in the camera coordinate system.

[0010] Based on the further improvement of the above method, the following loss function is constructed, and the deep learning neural network is trained using the marked visible light image and the marked laser point cloud image based on the loss function:

[0011]

[0012] Among them, D and G represent the predicted bounding box and the real bounding box respectively, c represents the confidence of the classification of D; the total residual function L total The definition is as follows:

[0013] L total =L rpn +L rcnn ,

[0014] Among them, L rpn and L rcnn They represent the residual functions of the two sub-networks, RPN and RCNN, respectively. The RPN network is used to generate candidate boxes, and the RCNN optimization network is used to optimize the target detection box.

[0015] Based on the further improvement of the above method, acquiring the laser point set and the image point set of the indoor space by the multi-sensor to locate the multi-line laser radar further includes: performing high-precision positioning based on feature matching, wherein the high-precision positioning based on feature matching further includes: calculating the inclination angle ω of the laser beam of the laser point compared to the horizontal plane of the laser radar according to the laser point coordinates (x, y, z):

[0016]

[0017] The relative pose between the k+1th frame and the kth frame is:

[0018]

[0019] The points in the k+1th frame are transferred to the kth frame coordinate system:

[0020]

[0021] Among them, p i is a line feature, is the prediction line feature;

[0022] Construct the residual function of line features and surface features to solve the pose vector Twl in the world coordinate system:

[0023]

[0024]

[0025] Among them, |p a -p b | is the length of the line feature, when p i When it is a line feature, search for the nearest line feature point p in the previous frame a , and find another line feature point p on the adjacent line b , forming a straight line; when it is a surface feature, search for the nearest surface feature point p in the previous frame m , and find two surface feature points p on the adjacent lines j and p l , forming a plane.

[0026] The beneficial effects of the above technical solution are as follows: the multi-line laser radar is positioned using the laser point set and image point set obtained by the portable laser scanner to obtain the position and posture of the multi-line laser radar in the world coordinate system, which can avoid adding the positioning of objects inside the wall to the next room outside the wall due to insufficient system accuracy.

[0027] Based on the further improvement of the above method, high-precision spatial calibration of multiple sensors to obtain external parameters from the local coordinate system of the camera to the base coordinate center further includes: jointly calibrating the multi-line laser radar and the camera to obtain the rotation and translation of the multi-line laser radar relative to the camera; and jointly calibrating the multi-line laser radar and the solid laser radar to calculate the external parameters between the multi-line laser radar and the fixed laser radar, wherein the joint calibration of the multi-line laser radar and the camera further includes:

[0028]

[0029] Where Zc is the scale parameter; (X w ,Y w ,Z w ) is the world coordinate system; (u, v) is the pixel coordinate; the camera coordinate system uses the optical axis of the camera as the z-axis, and the center of the optical system of the camera is the origin Oc. The camera coordinate system Xc, Yc are parallel to the X and Y axes of the image coordinate system respectively. The distance f between the origin of the camera coordinate system and the origin of the image coordinate system is the focal length. is a camera intrinsic parameter matrix, and the camera intrinsic parameter matrix is ​​solved by using Zhang Zhengyou calibration method; is the camera extrinsic parameter matrix; and the multi-line laser radar and the camera are jointly coarsely calibrated with external parameters; and the multi-line laser radar and the camera are jointly finely calibrated with external parameters.

[0030] Based on the further improvement of the above method, the multi-line laser radar and the solid laser radar are jointly calibrated to calculate the external parameters between the multi-line laser radar and the fixed laser radar, further including: collecting point cloud data of two laser radars in a standard indoor space; providing plane features from the point cloud data; matching the plane features; after completing the plane feature matching, using singular value decomposition to solve the initial values ​​of R and t; and establishing an optimization function according to the square of the distance from the point to the plane as the objective function.

[0031] Based on the further improvement of the above method, obtaining the position Pw of the target to be identified in the world coordinate system by the following formula further includes:

[0032] Pw=Twl*Tlc*Pc

[0033] Among them, the position of the target to be identified in the camera coordinate system is Pc; the posture of the multi-line laser radar in the world coordinate system is Twl; and the external parameters from the local coordinate system of the camera to the base coordinate center are Tlc.

[0034] On the other hand, an embodiment of the present invention provides a device for obtaining the position of an indoor object, comprising: a calibration module, used to perform high-precision spatial calibration on multiple sensors to obtain external parameters from the local coordinate system of the camera to the base coordinate center, wherein the multiple sensors include a multi-line laser radar, a fixed laser radar and a camera; a positioning module, used to obtain a laser point set and an image point set of the indoor space through the multiple sensors to position the multi-line laser radar; a target recognition model, used to obtain a laser point set and an image point set of an object to be identified located in the indoor space through the multiple sensors, and use the target recognition model to detect the position of the object to be identified in the camera coordinate system, wherein the target recognition model includes a point cloud feature extractor for extracting laser radar features, an image feature extractor for extracting visual semantic features and a fusion module for effectively fusing the laser radar features and the visual semantic features; and a coordinate transformation module, used to obtain the absolute position of the object to be identified in the world coordinate system through coordinate transformation.

[0035] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0036] 1. This application can significantly reduce the error of indoor object positioning by high-precision spatial calibration of multiple sensors, positioning the multi-line laser radar, and using the target recognition model to detect the position of the object to be identified in the camera coordinate system. In addition, the absolute position of the object to be identified in the world coordinate system is obtained through coordinate transformation, so that the target to be identified can be added to the building information model BIM, so as to effectively reflect the record and change of building information.

[0037] 2. Use the laser point set and image point set obtained by the portable laser scanner to locate the multi-line laser radar to obtain the position and posture of the multi-line laser radar in the world coordinate system, which can avoid adding the positioning of objects inside the wall to the next room outside the wall due to insufficient system accuracy.

[0038] 3. By connecting multiple fusion module networks between the point cloud branch network and the image branch network, lidar features (geometry) and visual semantic features (texture) are fused at multiple levels.

[0039] 4. By designing a new Loss function, the consistency of target classification and positioning confidence level can be further improved.

[0040] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be achieved and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0042] Figure 1 is a flow chart of a method for obtaining the position of an indoor object according to an embodiment of the present invention;

[0043] Figure 2 Schematic diagram of the rotation and translation of the lidar relative to the camera;

[0044] Figure 3 An illustration of a target image acquired by a camera and a laser point cloud image acquired by a lidar.

[0045] Figure 4 This is a diagram for extracting the center of a circle for a target given in advance.

[0046] Figure 5 Establish a local coordinate system for multi-line lidar and solid-state lidar.

[0047] Figure 6 Flowchart of the high-precision positioning method based on direct matching of multi-line lidar.

[0048] Figure 7 is the coordinate system of the multi-line lidar.

[0049] Figure 8 It is a laser point cloud image with the distance from the laser point to the radar and the curvature.

[0050] Fig. 9 Schematic diagram of the fusion of a point cloud branch and an image branch according to an embodiment of the present invention.

[0051] Fig.10 is a diagram of a deep network structure according to an embodiment of the present invention.

[0052] Fig.11 4 is a structural diagram of a fusion module network structure according to an embodiment of the present invention.

[0053] Fig.12 Schematic diagram of the spatial transformation between the laser radar and the camera according to an embodiment of the present invention.

[0054] Fig.13 It is a block diagram of an automatic identification and positioning device for indoor objects according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0056] Indoor space positioning technology, which requires extremely high advantages of both laser point cloud and BIM model, happens to be a place where it can be put into practice.

[0057] A specific embodiment of the present invention discloses a method for obtaining the position of an indoor object. Figure 1 , the method for obtaining the position of an indoor object includes: in step S102, high-precision spatial calibration is performed on multiple sensors to obtain external parameters from the local coordinate system of the camera to the base coordinate center, wherein the multiple sensors include a multi-line laser radar, a fixed laser radar and a camera; in step S104, a laser point set and an image point set of the indoor space are obtained by the multiple sensors to locate the multi-line laser radar; in step S106, a laser point set and an image point set of an object to be identified located in the indoor space are obtained by the multiple sensors, and a target recognition model is used to detect the position of the object to be identified in the camera coordinate system, wherein the target recognition model includes a point cloud feature extractor for extracting laser radar features, an image feature extractor for extracting visual semantic features and a fusion module for effectively fusing laser radar features and visual semantic features; and in step S108, the absolute position of the object to be identified in the world coordinate system is obtained by coordinate transformation.

[0058] Compared with the prior art, the method for obtaining the position of indoor objects provided in this embodiment can greatly reduce the error of the indoor object positioning position by performing high-precision spatial calibration on multiple sensors, positioning the multi-line laser radar, and using the target recognition model to detect the position of the object to be identified in the camera coordinate system. In addition, the absolute position of the object to be identified in the world coordinate system is obtained through coordinate transformation, so that the target to be identified is added to the building information model BIM, so as to effectively reflect the recording and change of building information.

[0059] In the following, reference will be made to Figure 1 , each step of the method for obtaining the position of an indoor object according to an embodiment of the present invention is described in detail.

[0060] In step S102, high-precision spatial calibration is performed on the multi-sensor to obtain external parameters from the local coordinate system of the camera to the base coordinate center, wherein the multi-sensor includes a multi-line laser radar, a fixed laser radar and a camera. High-precision spatial calibration of the multi-sensor to obtain external parameters from the local coordinate system of the camera to the base coordinate center further includes: jointly calibrating the multi-line laser radar and the camera to obtain the rotation and translation of the multi-line laser radar relative to the camera; and jointly calibrating the multi-line laser radar and the solid laser radar to calculate the external parameters between the multi-line laser radar and the fixed laser radar.

[0061] Specifically, the joint calibration of the multi-line laser radar and the camera further includes:

[0062]

[0063] Where Zc is the scale parameter; (X w ,Y w ,Z w ) is the world coordinate system; (u, v) is the pixel coordinate; the camera coordinate system uses the optical axis of the camera as the z-axis, and the origin Oc is the center of the optical system of the camera. The camera coordinate system Xc and Yc are parallel to the image coordinate system X and Y axes respectively. The distance f between the origin of the camera coordinate system and the origin of the image coordinate system is the focal length. is the camera intrinsic parameter matrix, and the camera intrinsic parameter matrix is ​​solved using Zhang Zhengyou calibration method; is the camera extrinsic parameter matrix; and the extrinsic parameters of the multi-line laser radar and the camera are jointly coarsely calibrated; and the extrinsic parameters of the multi-line laser radar and the camera are jointly finely calibrated.

[0064] Specifically, the joint calibration of the multi-line laser radar and the solid laser radar to calculate the external parameters between the multi-line laser radar and the fixed laser radar further includes: collecting point cloud data of the two laser radars in a standard indoor space; providing plane features from the point cloud data; matching the plane features; after completing the plane feature matching, using singular value decomposition to solve the initial values ​​of R and t; and establishing an optimization function according to the square of the distance from the point to the plane as the objective function.

[0065] In step S104, the laser point set and the image point set of the indoor space are obtained by multi-sensors to locate the multi-line laser radar. The laser point set and the image point set of the indoor space are obtained by multi-sensors to locate the multi-line laser radar further includes: high-precision positioning based on feature matching. High-precision positioning based on feature matching further includes: according to the laser point coordinates (x, y, z), calculating the inclination angle ω of the laser beam of the laser point compared to the horizontal plane of the laser radar:

[0066]

[0067] The relative pose between the k+1th frame and the kth frame is:

[0068]

[0069] The points in the k+1th frame are transferred to the kth frame coordinate system:

[0070]

[0071] Among them, p i is a line feature, is the prediction line feature;

[0072] Construct the residual function of line features and surface features to solve the pose vector Twl in the world coordinate system:

[0073]

[0074]

[0075] Among them, |p a -p b | is the length of the line feature, when p i When it is a line feature, search for the nearest line feature point p in the previous frame a , and find another line feature point p on the adjacent line b , forming a straight line; when it is a surface feature, search for the nearest surface feature point p in the previous frame m , and find two surface feature points p on the adjacent lines j and p l , forming a plane.

[0076] In step S106, a laser point set and an image point set of an object to be identified in an indoor space are acquired through multiple sensors, and a target recognition model is used to detect the position of the object to be identified in a camera coordinate system, wherein the target recognition model includes a point cloud feature extractor for extracting lidar features, an image feature extractor for extracting visual semantic features, and a fusion module for effectively fusing lidar features and visual semantic features.

[0077] The laser radar features include the geometric structure and depth information of the point cloud. The visual semantic features include the color and texture information of the image. The fusion module is used to fuse the geometric structure and depth information of the point cloud with the color and texture information of the image to generate a 3D model of the object to be identified. Specifically, the point cloud feature extractor includes a multi-scale point cloud feature extractor for extracting the geometric structure and depth information of the multi-scale point cloud. The image feature extractor includes a multi-scale image feature extractor for extracting the color and texture information of the multi-scale image. The fusion module includes a multi-scale fusion module for fusing the geometric structure and depth information of point clouds of different scales with the color and texture information of images of corresponding scales layer by layer.

[0078] Specifically, the laser point set and image point set of the object to be identified in the indoor space are obtained by multiple sensors, and the position of the object to be identified in the camera coordinate system is detected by using the target recognition model, which further includes: establishing a deep learning neural network, training the deep learning neural network with the marked visible light image and the marked laser point cloud map to obtain the target recognition model; using a camera to take a visible light image of the target to be identified and using a multi-line laser radar and a fixed laser radar to scan the laser point cloud map of the target to be identified, and inputting the visible light image of the target to be identified and the laser point cloud map of the target to be identified into the target recognition model to obtain the type of the target to be identified and the position in the camera coordinate system. Construct the following loss function, and train the deep learning neural network with the marked visible light image and the marked laser point cloud map based on the loss function:

[0079]

[0080] Where D and G represent the predicted bounding box and the true bounding box respectively, and c represents the confidence of the classification of D;

[0081] The total residual function L total The definition is as follows:

[0082] L total =L rpn +L rcnn ,

[0083] Among them, L rpn and L rcnn They represent the residual functions of the two sub-networks, RPN and RCNN. The RPN network is used to generate candidate boxes, and the RCNN optimization network is used to optimize the target detection box.

[0084] In step S108, the absolute position of the object to be identified in the world coordinate system is obtained by coordinate transformation. Obtaining the position Pw of the object to be identified in the world coordinate system by the following formula further includes:

[0085] Pw=Twl*Tlc*Pc

[0086] Among them, the position of the target to be identified in the camera coordinate system is Pc; the posture of the multi-line laser radar in the world coordinate system is Twl; and the external parameters from the camera local coordinate system to the base coordinate center are Tlc.

[0087] In step S110, the target to be identified is added to the Building Information Model (BIM) according to its position in the world coordinate system. Specifically, BIM uses digital means to build a virtual building in a computer, which provides a single, complete, and logically related building information library. The connotation of "information" is not only the visual information described by geometric shapes, but also includes a large amount of non-geometric information, such as the fire resistance level and heat transfer coefficient of materials, the cost and procurement information of components, etc.

[0088] A specific embodiment of the present invention discloses a device for obtaining the position of an indoor object. Fig.13 According to an embodiment of the present invention, the position acquisition device of an indoor object includes: a calibration module 1302, which is used to perform high-precision spatial calibration on multiple sensors to obtain external parameters from the local coordinate system of the camera to the base coordinate center, wherein the multiple sensors include a multi-line laser radar, a fixed laser radar and a camera; a positioning module 1304, which is used to obtain a laser point set and an image point set of the indoor space through the multiple sensors to locate the multi-line laser radar; a target recognition model 1306, which is used to obtain a laser point set and an image point set of an object to be identified located in the indoor space through the multiple sensors, and use the target recognition model to detect the position of the object to be identified in the camera coordinate system, wherein the target recognition model includes a point cloud feature extractor for extracting laser radar features, an image feature extractor for extracting visual semantic features and a fusion module for effectively fusing laser radar features and visual semantic features; and a coordinate transformation module 1308, which is used to obtain the absolute position of the object to be identified in the world coordinate system through coordinate transformation.

[0089] In the following, reference will be made to Figures 2 to 12 , the method for obtaining the position of an indoor object according to an embodiment of the present invention is described in detail by way of a specific example.

[0090] The technology of handheld intelligent monitoring equipment involves appearance and structure design, hardware system design, software system design (embedded software, acquisition software, intelligent processing software), core algorithm design, etc. The core algorithm is the key to this application, and the rationality of its design directly affects the overall performance of the equipment. This article mainly explains this part in detail.

[0091] The core algorithms of handheld intelligent monitoring devices mainly include the following four parts: multi-sensor high-precision spatial calibration algorithm, high-precision positioning algorithm based on lidar, multi-modal target recognition algorithm based on deep learning, and target position acquisition algorithm based on multi-sensor fusion. These four parts will be explained in detail below.

[0092] 1. Multi-sensor high-precision spatial calibration algorithm

[0093] The core sensors in this system are 1 multi-line laser radar, 1 solid-state laser radar and 1 visible light camera. Therefore, sensor calibration mainly refers to the study of a set of rigorous algorithms and operating procedures through precise design, and finally obtaining high-precision extrinsic parameters between the three. The extrinsic parameters are represented by a 4*4 spatial transformation matrix or a combination of a position vector and an Euler angle vector (quaternion).

[0094] The relationship between Euler angle and rotation matrix transformation:

[0095] Assume that the rotation is in the order of ZYX (2,1,0), and the rotation angles are φ, θ, and ψ respectively. The rotation matrix is ​​expressed as follows:

[0096]

[0097] This embodiment uses multi-line laser as a basic coordinate reference to calibrate the transformation relationship from visible light camera and solid-state laser to multi-line laser.

[0098] 1.1 Multi-line laser and camera joint calibration

[0099] like Figure 2 As shown, the result of the joint calibration of the lidar and the camera is the rotation and translation of the lidar relative to the camera.

[0100] The joint calibration is carried out in two steps: first calibrate the camera intrinsic parameters and then jointly calibrate the extrinsic parameters.

[0101] 1.1.1 Camera internal parameter calibration

[0102] Concept of intrinsic parameter calibration: In the image measurement process and computer vision, in order to determine the three-dimensional geometric relationship position of a point on a spatial object and the relationship between its corresponding point in the image, a geometric model of camera imaging must be established. The parameters of the model are the parameters of the camera, and the process of solving the parameters is called camera calibration.

[0103] The experimental process of intrinsic parameter calibration is as follows:

[0104] (1) Print a piece of A4 paper with a checkerboard pattern (the black and white spacing is known) and stick it on a flat plate.

[0105] (2) Take several pictures of the chessboard (usually 10-20 pictures).

[0106] (3) Detect feature points (Harris features) in the image.

[0107] (4) Five internal parameters and six external parameters are calculated using the analytical solution estimation method.

[0108] (5) Based on the maximum likelihood estimation strategy, design the optimization objective and achieve parameter refinement.

[0109] The imaging process of the camera can be expressed as:

[0110]

[0111] Among them, the world coordinate system (Xw, Yw, Zw), the camera coordinate system (Xc, Yc, Zc), the pixel coordinate system (u, v), the camera coordinate system uses the optical axis of the camera as the z-axis, the center of the light in the camera's optical system is the origin Oc, the camera coordinate system Xc, Yc are parallel to the image coordinate system X, Y axis respectively. The distance f between the origin of the camera coordinate system and the origin of the image coordinate system is the focal length is a camera intrinsic parameter matrix, and the camera intrinsic parameter matrix is ​​solved by using Zhang Zhengyou calibration method; is the camera extrinsic parameter matrix.

[0112] 1.1.2 Joint calibration of external parameters

[0113] Rough calibration:

[0114] (1) Description of landmark points

[0115] This paper uses markers for calibration because the edges of markers are easier to detect in the two sensor data. In the calibration process, laser and camera are used to collect one frame of data respectively as basic data.

[0116] (2) Detection under point cloud

[0117] like Figure 3 As shown, since the depth information is discontinuous, edge detection can be achieved by detecting the depth difference between adjacent points on the same line. Figure 3 (a) and (c) are the original images for detection, and (b) and (d) are the laser point cloud images corresponding to (a) and (c) respectively.

[0118] The size, quantity, and position of the targets on the actual calibration board are known, and detection, verification, and point cloud pruning are then required.

[0119] a. Keep the inner points of the plane and remove the outer points. The inner points will be processed as follows: Figure 4 As shown in (a) in .

[0120] b. Use the random sampling consensus algorithm RANSAC (full name Random sample consensus) method to extract the center of the circle for the target given in advance (four circles forming a square, radius, point distance), such as Figure 4 As shown in (b) in .

[0121] c. Verify the test results. If passed, the test ends. If not, proceed to the next step.

[0122] d. If the points cannot be extracted, the point cloud needs to be processed and trimmed. Set rules to extract the points you want to extract.

[0123] (3) Detection in the camera

[0124] The Sobel operator is used to extract the image edge, and then the Hough transform is used to extract the circular boundary.

[0125] (4) Calculate the initial translation vector (t x t y t z )

[0126] Calculate t by the following formula z , r 3D Represents the radius of the point cloud detection, r 2D represents the radius detected by the image, Z represents the depth coordinate of the center of the circle detected by the image, and f represents the focal length:

[0127]

[0128] X, Y, and Z represent the coordinates of the point under the laser radar, x and y represent the coordinates in the image coordinate system, and o x , o y Indicates the principal point deviation.

[0129]

[0130]

[0131] Precision calibration:

[0132] The process of fine calibration is to search for the optimal solution of parameters in a small parameter search space. Based on the premise that the edges detected by the camera and the edges detected by the lidar match each other, the loss function is designed as follows:

[0133] a: Project the target point in the laser radar coordinate system onto the camera plane to construct a two-dimensional image plane;

[0134] b: After being processed by the Sobel operator, an image containing edge information is generated;

[0135] c: Use the Inverse Distance Transform (IDT) method to perform L1 regularization on the edge information of the image.

[0136] d: Construct error function S E , use nonlinear optimization methods to optimize external parameters so as to minimize the value of the loss function.

[0137]

[0138] The above formula represents the setting of Loss function, I C Represents the data in the processed image coordinate system, I V It represents the data in the radar coordinate system after processing and projection. Therefore, optimization is performed on the basis of rough calibration to improve the calibration accuracy.

[0139] 1.2 Joint calibration of multi-line laser and solid-state laser

[0140] This project uses surface feature matching to calculate the calibration method of external parameters between lidars, and collects point cloud data of two lasers as basic data in a standard calibration room.

[0141] (1) Surface feature extraction

[0142] a. Point cloud preprocessing: Due to the influence of noise, there are many points that are not on the plane, so point cloud preprocessing is performed first to remove the points that are not on the plane.

[0143] b. Use RANSAC to fit a plane between two point clouds, which may fit many planes.

[0144] c. Since the plane normal vector is equivalent to a coordinate axis, only the matching of three coordinate axes is needed to solve the rotation of the two coordinate systems. When using RANSAC to fit two point clouds, there may be more than three planes. According to the number of plane points, the three largest planes are retained.

[0145] d. Solve the parameter coefficients of the plane. The plane equation is usually expressed as:

[0146] β (i,0) x n +β (i,1) y n +β (i,2) z n +β (i,3) =0

[0147] Among them, β (i,0) , β (i,1) , β (i,2) and β(i,3) are parameters respectively. Therefore, the distance from a point to a plane can be defined as:

[0148] f i (P n )=|β (i,0) x n +β (i,1) y n +β (i,2) z n +β (i,3) |

[0149] e. Construct the least squares problem. Select N points on the plane. The plane coefficients should satisfy the following least squares problem:

[0150]

[0151] f. After solving the plane coefficients, consider the three planes as the three planes of the XYZ coordinate axis. The origin of the local coordinate system composed of these three planes is the intersection of the three planes:

[0152]

[0153] (2) Face feature matching

[0154] like Figure 5 As shown, the ground is usually selected as a feature surface during the calibration process, and the installation of the laser radar is generally approximately parallel to the ground. Therefore, after setting the normal vector direction, use:

[0155] n 地面 =max([0,0,1]n i )

[0156] Determine which plane in the point cloud is the ground, and after setting the local right-handed coordinate system, the three normal vectors n1, n2 and n3 satisfy the following relationship:

[0157] (n2×n1)n3>0

[0158] By traversing the combination, each plane can be assigned a corresponding number to complete the matching of plane features.

[0159] (3) Closed-form solution of initial value of external parameter

[0160] After completing the matching of the plane features, the three normal vectors are regarded as three points, and the initial value of R can be solved by SVD decomposition, because the three normal vectors have the following corresponding relationship:

[0161] Rn1=n′1, Rn2=n′2, Rn3=n′3

[0162] R[n1,n2,n3]=[n′1,n′2,n′3]

[0163] P=[n1,n2,n3],Q=[n′1,n′2,n′3]

[0164] H=PQ T

[0165] Performing SVD decomposition on the H matrix can obtain the intermediate variables V and U. The initial values ​​of R and t can be solved using the following formula, where O and O′ are the origin coordinates of the local coordinate systems of the two lidars.

[0166] R=VU T ,t=O′-RO

[0167] (4) Nonlinear Optimization

[0168] Finally, we use the square of the distance from the point to the plane as the objective function, establish an optimization function, and perform iterative optimization. The optimization function is to sum the error values ​​of the three matching planes respectively, where the error of each matching plane consists of two parts: the first part f i 2 (Rp′+t) is the distance from the point on the solid-state laser plane to the corresponding plane of the multi-line laser, and p′ is the coordinate of the point on the solid-state laser plane; the second part f i 2 (R′p+t) is the distance from a point on the multi-line laser plane to the corresponding plane of the solid-state laser, and p is the coordinate of the point on the multi-line laser plane.

[0169]

[0170] 2. High-precision positioning algorithm based on LiDAR

[0171] This module mainly adopts a high-precision positioning algorithm based on direct matching and feature fusion.

[0172] 2.1 High-precision positioning algorithm based on direct matching

[0173] For two point sets:

[0174] X={x1,x2,…,x Nx}

[0175] Y={y1,y2,…,y Ny}

[0176] refer to Figure 6First, preprocess the point cloud data; divide the space into grids, count the points that fall in each grid; calculate the mean and covariance of each grid and construct a Gaussian distribution based on the points in each grid; calculate the joint probability based on the predicted posture; solve R,t, and determine whether the solution is completed. If not, return to calculating the joint probability based on the predicted posture. If it is completed, output R,t.

[0177] Find the objective function such that:

[0178]

[0179] in:

[0180]

[0181] The objective function is defined as:

[0182]

[0183] y′ i =T(p,y i )=Ry i +t

[0184] The objective function is defined as follows, μ is the centroid of the point set X, y′ i To use the predicted pose for point y i The coordinates of the point obtained after the transformation.

[0185] According to the process of Gauss-Newton method, iterative optimization can be performed by simply calculating the Jacobian of the residual function with respect to the parameters to be determined.

[0186] 2.2 High-precision positioning algorithm based on feature matching

[0187] (1) Line and surface feature extraction

[0188] refer to Figure 7 , according to the laser point coordinates (x, y, z), the inclination angle ω of the laser beam compared to the radar horizontal plane can be calculated:

[0189]

[0190] refer to Figure 8 , based on the inclination angle and the radar internal parameters (the designed inclination angle of each scanning line), it can be known to which laser beam the radar belongs.

[0191] The curvature is calculated based on the length X between the adjacent points and the current point (the length refers to the distance from the laser point to the radar), and the feature is judged based on the curvature.

[0192]

[0193] (2) Line-surface feature association

[0194] The relative pose between the k+1th frame and the kth frame is:

[0195]

[0196] The points in the k+1th frame are transferred to the kth frame coordinate system:

[0197]

[0198] p i is a line feature, is the prediction line feature.

[0199] (3) Posture optimization

[0200] Construct the residual functions of line features and surface features respectively:

[0201]

[0202]

[0203] in, is the prediction line feature; |p a -p b | is the length of the line feature, when p i When it is a line feature, search for the nearest line feature point p in the previous frame a , and find another line feature point p on the adjacent line b , forming a straight line. When it is a surface feature, search for the nearest surface feature point p in the previous frame m , and find two surface feature points p on the adjacent lines j and p l , forming a plane.

[0204] According to the convex optimization foundation, we only need to obtain the Jacobian of the residual with respect to the variable to be determined, and then we can use Gauss-Newton and other optimization methods to solve the pose vector.

[0205] 3. Multimodal object recognition based on deep learning

[0206] Traditional deep learning-based image target recognition technology has become stable and mature, but it is prone to false detection and missed detection in small targets and complex scenes, so it can only be applied in target recognition scenes with relatively simple environments. Deep learning is essentially a data-driven pattern expression learning method. Its poor performance in complex situations is mainly limited by the limitations of single sensor data. Multimodal data can effectively express the environment in different feature dimensions, and its huge advantages have gradually attracted great attention from researchers. This project intends to adopt a new deep neural network structure. The network input is multimodal sensor data, including visible light images and laser point clouds. The feature expression learning of different input data is realized through network parameters, and the output is the type and location of the target. A certain amount of labeled data is used for training and learning to achieve automatic tuning of parameters, thereby greatly improving the accuracy of target recognition and environmental adaptability.

[0207] Theoretically, image information is dense and regular, and contains rich color and texture information, but the disadvantage is that it is two-dimensional information. There is a scale problem due to distance. Compared with images, point clouds are sparse and irregular, which makes it impossible to directly process point clouds using traditional CNN perception. However, point clouds contain three-dimensional geometric structures and depth information, which is more beneficial for 3D target detection. Therefore, the information of the two is theoretically complementary. In addition, in the current two-dimensional image detection, deep learning methods are all designed based on CNN, while in point cloud target detection, there are networks designed with multiple basic structures such as MLP, CNN, and GCN. In the fusion process, it is also necessary to study which network to fuse with.

[0208] This project mainly adopts the method based on feature fusion, such as Fig. 9 As shown in the figure, this fusion requires some interaction in the feature layer. The main fusion method is to use feature extractors for both the point cloud and image branches, and fuse the networks of the image branch and the point cloud branch at the semantic level in the feedforward layer to achieve semantic fusion of multi-scale information.

[0209] The network structure used in this project is as follows Fig.10 As shown in the figure, the point cloud branch is a point encoder-decoder structure, and the image branch is a step-by-step encoding network, and feature fusion is performed layer by layer.

[0210] The network consists of a two-stream RPN network and an optimization network. The RPN network is mainly used to generate candidate boxes, and the optimization network is mainly used to optimize target detection boxes. The network can be directly trained end-to-end. Through the L1-Fusion module, the network can effectively fuse lidar features and visual semantic features.

[0211] L1-Fusion module network structure, such as Fig.11 As shown:

[0212] In addition, the network designs a new loss function (CE loss) to further improve the consistency of target classification and localization confidence level.

[0213] CE loss is defined as follows:

[0214]

[0215] Where D and G represent the predicted and true bounding boxes respectively, and c represents the confidence of D’s classification.

[0216] The total residual function L total The definition is as follows:

[0217] L total =L rpn +L rcnn ,

[0218] Where L rpn and L rcnn Represent the residual functions of the two sub-networks of rpn and rcnn respectively.

[0219] 4. Target position acquisition based on multi-sensor fusion

[0220] refer to Fig.12 After the three steps of sensor calibration, high-precision positioning, and target detection of multimodal data, the results of the above three steps can be used for fusion calculation to solve the absolute position of the target of interest in the world coordinate system. The entire calculation process is summarized as follows:

[0221] (1) At time t, the target of interest is detected based on the multi-mode fusion results, and its position in the camera coordinate system is Pc;

[0222] (2) The position of the base coordinate center (multi-line laser) in the world coordinate system at this moment has been solved as Twl;

[0223] (3) The external parameters of each sensor have been calibrated with high precision, where the external parameter from the camera local coordinate system to the base coordinate center is Tlc;

[0224] (4) The position Pw of the target of interest in the world coordinate system can be solved by spatial transformation, and its calculation formula is:

[0225] Pw=Twl*Tlc*Pc

[0226] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0227] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for obtaining the position of an indoor object, characterized in that: include: Performing high-precision spatial calibration on multiple sensors to obtain external parameters from the local coordinate system of the camera to the base coordinate center, wherein the multiple sensors include a multi-line laser radar, a fixed laser radar and a camera; Acquire a laser point set and an image point set of an indoor space by using the multi-sensor to locate the multi-line laser radar; Acquire a laser point set and an image point set of an object to be identified in the indoor space through the multi-sensor, and detect the position of the object to be identified in the camera coordinate system using a target recognition model, wherein the target recognition model includes a point cloud feature extractor for extracting laser radar features, an image feature extractor for extracting visual semantic features, and a fusion module for effectively fusing the laser radar features and the visual semantic features; and Obtaining the absolute position of the object to be identified in the world coordinate system through coordinate transformation; The point cloud feature extractor includes a multi-scale point cloud feature extractor for extracting the geometric structure and depth information of the multi-scale point cloud; the image feature extractor includes a multi-scale image feature extractor for extracting the color and texture information of the multi-scale image; and the fusion module includes a multi-scale fusion module for fusing the geometric structure and depth information of the point clouds of different scales with the color and texture information of the images of the corresponding scales layer by layer; Wherein, acquiring the laser point set and the image point set of the object to be identified in the indoor space through the multi-sensor, and detecting the position of the object to be identified in the camera coordinate system by using the target recognition model further comprises: establishing a deep learning neural network, and training the deep learning neural network by using the marked visible light image and the marked laser point cloud map to obtain the target recognition model; using the camera to shoot the visible light image of the target to be identified and using the multi-line laser radar and the fixed laser radar to scan the laser point cloud map of the target to be identified, and inputting the visible light image of the target to be identified and the laser point cloud map of the target to be identified into the target recognition model to obtain the type of the target to be identified and the position in the camera coordinate system; The following loss function is constructed, and the deep learning neural network is trained using the marked visible light image and the marked laser point cloud image based on the loss function: Among them, D and G represent the predicted bounding box and the real bounding box respectively, c represents the confidence of the classification of D; the total residual function L total The definition is as follows: L total =L rpn +L rcnn , Among them, L rpn and L rcnn They represent the residual functions of the two sub-networks, RPN and RCNN, respectively. The RPN network is used to generate candidate boxes, and the RCNN optimization network is used to optimize the target detection box.

2. The method for obtaining the position of an indoor object according to claim 1, characterized in that: The laser radar features include the geometric structure and depth information of the point cloud; The visual semantic features include color and texture information of the image; as well as The fusion module is used to fuse the geometric structure and depth information of the point cloud with the color and texture information of the image to generate a 3D model of the object to be identified.

3. The method for obtaining the position of an indoor object according to claim 1, characterized in that: Acquiring the laser point set and the image point set of the indoor space by the multi-sensor to locate the multi-line laser radar further includes: performing high-precision positioning based on feature matching, wherein the high-precision positioning based on feature matching further includes: According to the laser point coordinates (x, y, z), the inclination angle ω of the laser beam of the laser point compared to the laser radar horizontal plane is calculated: The relative pose between the k+1th frame and the kth frame is: The points in the k+1th frame are transferred to the kth frame coordinate system: Among them, pi is the line feature, is the prediction line feature; Construct the residual function of line features and surface features to solve the pose vector Twl in the world coordinate system: Among them, |p a -p b | is the length of the line feature, when p i When it is a line feature, search for the nearest line feature point p in the previous frame a , and find another line feature point p on the adjacent line b , forming a straight line; when it is a surface feature, search for the nearest surface feature point p in the previous frame m , and find two surface feature points p on the adjacent lines j and p l , forming a plane.

4. The method for obtaining the position of an indoor object according to claim 1, characterized in that: High-precision spatial calibration of multiple sensors to obtain external parameters from the camera local coordinate system to the base coordinate center further includes: Jointly calibrating the multi-line laser radar and the camera to obtain a rotation and a translation of the multi-line laser radar relative to the camera; and The multi-line laser radar and the solid laser radar are jointly calibrated to calculate the external parameters between the multi-line laser radar and the fixed laser radar. The joint calibration of the multi-line laser radar and the camera further includes: Where Zc is the scale parameter; (X w ,Y w ,Z w ) is the world coordinate system; (u, v) is the pixel coordinate; the camera coordinate system uses the optical axis of the camera as the z-axis, and the center of the optical system of the camera is the origin Oc. The camera coordinate system Xc, Yc are parallel to the X and Y axes of the image coordinate system respectively. The distance f between the origin of the camera coordinate system and the origin of the image coordinate system is the focal length. is a camera intrinsic parameter matrix, and the camera intrinsic parameter matrix is ​​solved by using Zhang Zhengyou calibration method; is the camera extrinsic parameter matrix; and Performing a joint rough calibration of external parameters of the multi-line laser radar and the camera; and The multi-line laser radar and the camera are jointly calibrated for external parameters.

5. The method for obtaining the position of an indoor object according to claim 4, characterized in that: The method of jointly calibrating the multi-line laser radar and the solid laser radar to calculate the external parameters between the multi-line laser radar and the fixed laser radar further includes: Collect point cloud data from two LiDARs in a standard indoor space; providing planar features from the point cloud data; matching the planar features; After completing the plane feature matching, the initial values ​​of R and t are solved using singular value decomposition; and The square of the distance from the point to the plane is used as the objective function to establish the optimization function.

6. The method for obtaining the position of an indoor object according to claim 5, characterized in that: Obtaining the position Pw of the target to be identified in the world coordinate system by the following formula further includes: Pw=TTwl*Tlc*PC Among them, the position of the target to be identified in the camera coordinate system is Pc; the posture of the multi-line laser radar in the world coordinate system is Twl; and the external parameters from the local coordinate system of the camera to the base coordinate center are Tlc.

7. A device for obtaining the position of an indoor object, used to execute the method for obtaining the position of an indoor object according to any one of claims 1 to 6, characterized in that: include: A calibration module, used for performing high-precision spatial calibration on multiple sensors to obtain external parameters from the local coordinate system of the camera to the base coordinate center, wherein the multiple sensors include a multi-line laser radar, a fixed laser radar and a camera; A positioning module, used for acquiring a laser point set and an image point set of an indoor space through the multi-sensor to position the multi-line laser radar; a target recognition model, used for acquiring a laser point set and an image point set of an object to be recognized located in the indoor space through the multi-sensor, and detecting a position of the object to be recognized in a camera coordinate system using the target recognition model, wherein the target recognition model includes a point cloud feature extractor for extracting laser radar features, an image feature extractor for extracting visual semantic features, and a fusion module for effectively fusing the laser radar features and the visual semantic features; and The coordinate transformation module is used to obtain the absolute position of the object to be identified in the world coordinate system through coordinate transformation.

Images