Method and device for automatically identifying and positioning indoor objects

Abstract

The present invention relates to a method and device for automatically identifying and positioning indoor objects, belonging to the field of indoor space positioning technology. The method solves the problem that existing indoor object positioning has large errors and cannot effectively reflect the recording and change of building information. The method includes: calibrating the transformation relationship between a fixed laser radar and a camera to a multi-line laser radar to obtain extrinsic parameters from the camera's local coordinate system to the base coordinate center; positioning the multi-line laser radar to obtain the multi-line laser radar's position in the world coordinate system; obtaining a visible light image and a laser point cloud image and inputting them into a multimodal target recognition model to obtain the position of the target to be identified in the camera coordinate system; obtaining the position of the target to be identified in the world coordinate system based on the extrinsic parameters, the position of the multi-line laser radar in the world coordinate system, and the position of the target to be identified in the camera coordinate system; and adding the target to be identified to the building information model. Adding the target to the BIM effectively reflects the recording and change of building information.

Description

Technical Field

[0001] The present invention relates to the technical field of indoor space positioning, and in particular to a method and device for automatically identifying and positioning indoor objects. Background Art

[0002] LiDAR is a new type of measuring instrument that uses the principle of laser ranging to determine the space to be scanned. The straight-line distance from the laser to the target is determined by measuring the phase difference (time difference) between the laser's transmitted signal and the reflected signal from the target, point by point. The spatial position of the target point is also determined based on the direction of the transmitted laser signal and the spatial position of the laser. By intensively scanning the surface of an object with the laser, a three-dimensional surface model of the object can be obtained.

[0003] BIM technology is one of the hottest topics in technology right now. Its ability to attach information to three-dimensional images makes visualization of the entire building lifecycle a reality. This raises the question of whether it would be possible to develop a system that combines the accuracy and measurable range of laser point clouds with the informativeness and flexibility of BIM models. Current indoor positioning technologies on the market, whether using GPS or Bluetooth, have errors exceeding ideal limits and are unable to effectively record and modify building information. Summary of the Invention

[0004] In view of the above analysis, the embodiments of the present invention aim to provide a method and apparatus for automatically identifying and positioning indoor objects, so as to solve the problem that the existing indoor object positioning has large errors and cannot effectively reflect the recording and change of building information.

[0005] On the one hand, an embodiment of the present invention provides a method for automatic identification and positioning of indoor objects, including: calibrating the transformation relationship from a fixed laser radar and a camera to a multi-line laser radar to obtain external parameters from the camera's local coordinate system to the base coordinate center, wherein the portable laser scanner includes the multi-line laser radar, the fixed laser radar and the camera; using the laser point set and the image point set obtained by the portable laser scanner to position the multi-line laser radar to obtain the posture of the multi-line laser radar in the world coordinate system; obtaining a visible light image and a laser point cloud map of the target to be identified and inputting them into a multimodal target recognition model to obtain the position of the target to be identified in the camera coordinate system; based on the external parameters from the camera's local coordinate system to the base coordinate center, the posture of the multi-line laser radar in the world coordinate system and the position of the target to be identified in the camera coordinate system, obtaining the position of the target to be identified in the world coordinate system; and adding the target to be identified to the building information model according to the position of the target to be identified in the world coordinate system.

[0006] The beneficial effects of the above technical solution are as follows: This application can obtain the position of the target to be identified in the world coordinate system based on the extrinsic parameters from the camera's local coordinate system to the base coordinate center, the multi-line lidar's pose in the world coordinate system, and the position of the target to be identified in the camera coordinate system, significantly reducing the positioning error of indoor objects. In addition, based on the position of the target to be identified in the world coordinate system, the target to be identified can be added to the Building Information Model (BIM), effectively reflecting the recording and changes of building information.

[0007] Based on further improvements of the above method, obtaining the visible light image and laser point cloud of the target to be identified and inputting them into the multimodal target recognition model to obtain the position of the target to be identified in the camera coordinate system further includes: establishing a deep learning neural network, and training the deep learning neural network using the marked visible light image and the marked laser point cloud map to obtain the multimodal target recognition model; using the camera to shoot the visible light image of the target to be identified and using the laser radar to scan the laser point cloud map of the target to be identified, and inputting the visible light image of the target to be identified and the laser point cloud map of the target to be identified into the multimodal target recognition model to obtain the type of the target to be identified and the position in the camera coordinate system.

[0008] Based on further improvements of the above method, establishing a deep learning neural network includes: establishing a point cloud branch network, which adopts a first feature extractor; establishing an image branch network, which adopts a second feature extractor, wherein multiple fusion module networks are connected between the point cloud branch network and the image branch network to fuse lidar features and visual semantic features at multiple levels.

[0009] Based on the further improvement of the above method, calibrating the transformation relationship from the fixed laser radar and the camera to the multi-line laser radar to obtain the external parameters from the local coordinate system of the camera to the base coordinate center further includes: jointly calibrating the multi-line laser radar and the camera to obtain the rotation and translation of the multi-line laser radar relative to the camera; and jointly calibrating the multi-line laser radar and the solid-state laser radar to calculate the external parameters between the multi-line laser radar and the fixed laser radar.

[0010] Based on a further improvement of the above method, jointly calibrating the multi-line laser radar and the camera to obtain the rotation and translation of the multi-line laser radar relative to the camera further includes calibrating the camera intrinsic parameters by the following formula:

[0011]

[0012] Among them, (X w ,Y w ,Zw ) is the world coordinate system; (u, v) is the pixel coordinate; the camera coordinate system uses the optical axis of the camera as the z-axis, and the center of the optical system of the camera is the origin Oc. The camera coordinate system Xc and Yc are parallel to the X and Y axes of the image coordinate system respectively. The distance f between the origin of the camera coordinate system and the origin of the image coordinate system is the focal length. is the camera intrinsic parameter matrix, and the camera intrinsic parameter matrix is ​​solved using Zhang Zhengyou calibration method; is the camera extrinsic parameter matrix; and the multi-line laser radar and the camera are jointly coarsely calibrated with extrinsic parameters; and the multi-line laser radar and the camera are jointly finely calibrated with extrinsic parameters.

[0013] Based on a further improvement of the above method, the multi-line laser radar and the solid-state laser radar are jointly calibrated to calculate the external parameters between the multi-line laser radar and the fixed laser radar, further including: collecting point cloud data of the two laser radars in a standard indoor space; providing plane features from the point cloud data; matching the plane features; after completing the plane feature matching, using singular value decomposition to solve the initial values ​​of R and t; and establishing an optimization function according to the square of the distance from the point to the plane as the objective function.

[0014] Based on a further improvement of the above method, obtaining the position Pw of the target to be identified in the world coordinate system by the following formula further includes:

[0015] Pw=Twl*Tlc*Pc

[0016] Among them, the position of the target to be identified in the camera coordinate system is Pc; the posture of the multi-line laser radar in the world coordinate system is Twl; and the external parameter from the local coordinate system of the camera to the base coordinate center is Tlc.

[0017] Based on a further improvement of the above method, positioning the multi-line laser radar using the laser point set and image point set acquired by the portable laser scanner to obtain the position and posture of the multi-line laser radar in the world coordinate system further includes: performing high-precision positioning based on direct matching or performing high-precision positioning based on feature matching, wherein the high-precision positioning based on feature matching further includes: calculating the inclination angle ω of the laser beam compared to the horizontal plane of the laser radar according to the laser point coordinates (x, y, z):

[0018]

[0019] The relative pose of the k+1th frame and the kth frame is:

[0020]

[0021] Construct residual functions of line features and surface features to solve the pose vector.

[0022] On the other hand, an embodiment of the present invention provides an automatic identification and positioning device for indoor objects, comprising: a calibration module for calibrating the transformation relationship from a fixed laser radar and a camera to a multi-line laser radar to obtain external parameters from the camera's local coordinate system to the base coordinate center, wherein the portable laser scanner includes the multi-line laser radar, the fixed laser radar and the camera; a positioning module for positioning the multi-line laser radar using the laser point set and image point set obtained by the portable laser scanner to obtain the pose of the multi-line laser radar in the world coordinate system; a multimodal target recognition module for obtaining a visible light image and a laser point cloud image of a target to be identified and inputting them into a multimodal target recognition model to obtain the position of the target to be identified in the camera coordinate system; a target position acquisition module for acquiring the position of the target to be identified in the world coordinate system based on the external parameters from the camera's local coordinate system to the base coordinate center, the pose of the multi-line laser radar in the world coordinate system and the position of the target to be identified in the camera coordinate system; and an adding module for adding the target to be identified to a building information model according to the position of the target to be identified in the world coordinate system.

[0023] Based on further improvements of the above-mentioned device, the multimodal target recognition module further includes: a network establishment submodule for establishing a deep learning neural network; a training submodule for training the deep learning neural network using the marked visible light image and the marked laser point cloud map to obtain the multimodal target recognition model; a portable laser scanner for using the camera to capture the visible light image of the target to be identified and using the laser radar to scan the laser point cloud map of the target to be identified; and a position acquisition submodule for inputting the visible light image of the target to be identified and the laser point cloud map of the target to be identified into the multimodal target recognition model to obtain the type of the target to be identified and the position in the camera coordinate system.

[0024] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0025] 1. This application can determine the position of the target in the world coordinate system based on the extrinsic parameters from the camera's local coordinate system to the base coordinate center, the multi-line lidar's pose in the world coordinate system, and the target's position in the camera coordinate system, significantly reducing the error in indoor object positioning. Furthermore, based on the target's position in the world coordinate system, the target can be added to the building information model, effectively reflecting the recording and modification of building information.

[0026] 2. Use the laser point set and image point set obtained by the portable laser scanner to locate the multi-line laser radar to obtain the position and posture of the multi-line laser radar in the world coordinate system. This can avoid the positioning of objects inside the wall being added to the next room outside the wall due to insufficient system accuracy.

[0027] 3. By connecting multiple fusion module networks between the point cloud branch network and the image branch network, lidar features (geometry) and visual semantic features (texture) are fused at multiple levels.

[0028] 4. By designing a new Loss function, the consistency of target classification and positioning confidence level can be further improved.

[0029] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like parts throughout the drawings.

[0031] Figure 1 Flowchart of a method for automatically identifying and positioning indoor objects according to an embodiment of the present invention;

[0032] Figure 2 Schematic diagram of the rotation and translation of the lidar relative to the camera;

[0033] Figure 3 An illustration of the target image acquired by the camera and the laser point cloud image acquired by the lidar.

[0034] Figure 4 This is a diagram showing the extraction of the center of a circle for a predetermined target.

[0035] Figure 5 Establish a local coordinate system for multi-line lidar and solid-state lidar.

[0036] Figure 6 Flowchart of the high-precision positioning method based on direct matching of multi-line lidar.

[0037] Figure 7 is the coordinate system of the multi-line lidar.

[0038] Figure 8 It is a laser point cloud image with the distance from the laser point to the radar and the curvature.

[0039] Figure 9 Schematic diagram of the fusion of the point cloud branch and the image branch according to an embodiment of the present invention.

[0040] Figure 10 2 is a diagram of a deep network structure according to an embodiment of the present invention.

[0041] Figure 11 2 is a structural diagram of a fusion module network structure according to an embodiment of the present invention.

[0042] Figure 12 Schematic diagram of spatial transformation between a laser radar and a camera according to an embodiment of the present invention.

[0043] Figure 13 FIG. 4 is a block diagram of an apparatus for automatically identifying and positioning indoor objects according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0045] Indoor space positioning technology, which requires extremely high standards for the advantages of both laser point clouds and BIM models, has become a place where it can be put into practical use.

[0046] A specific embodiment of the present invention discloses a method for automatically identifying and locating indoor objects. Figure 1 The method for automatic identification and positioning of indoor objects includes: in step S102, calibrating the transformation relationship from the fixed laser radar and the camera to the multi-line laser radar to obtain the extrinsic parameters of the camera local coordinate system to the base coordinate center, wherein the portable laser scanner includes the multi-line laser radar, the fixed laser radar and the camera; in step S104, positioning the multi-line laser radar using the laser point set and the image point set obtained by the portable laser scanner to obtain the posture of the multi-line laser radar in the world coordinate system; in step S106, obtaining the visible light image and the laser point cloud map of the target to be identified and inputting them into the multimodal target recognition model to obtain the position of the target to be identified in the camera coordinate system; in step S108, obtaining the position of the target to be identified in the world coordinate system based on the extrinsic parameters from the camera local coordinate system to the base coordinate center, the posture of the multi-line laser radar in the world coordinate system and the position of the target to be identified in the camera coordinate system; and in step S110, adding the target to be identified to the building information model according to the position of the target to be identified in the world coordinate system.

[0047] Compared to existing technologies, the automatic indoor object identification and positioning method provided in this embodiment uses the extrinsic parameters from the camera's local coordinate system to the base coordinate center, the multi-line lidar's pose in the world coordinate system, and the target's position in the camera coordinate system to determine the target's position in the world coordinate system. This significantly reduces indoor object positioning errors. Furthermore, based on the target's position in the world coordinate system, the target can be added to the building information model, effectively reflecting the recording and changes of building information.

[0048] In the following, reference will be made to Figure 1 , each step of the method for automatic identification and positioning of indoor objects according to an embodiment of the present invention is described in detail.

[0049] In step S102, the transformation relationship from the fixed laser radar and camera to the multi-line laser radar is calibrated to obtain extrinsic parameters from the camera's local coordinate system to the base coordinate center, wherein the portable laser scanner includes a multi-line laser radar, a fixed laser radar, and a camera. Calibrating the transformation relationship from the fixed laser radar and camera to the multi-line laser radar to obtain extrinsic parameters from the camera's local coordinate system to the base coordinate center further includes: jointly calibrating the multi-line laser radar and camera to obtain the rotation and translation of the multi-line laser radar relative to the camera; and jointly calibrating the multi-line laser radar and the solid-state laser radar to calculate extrinsic parameters between the multi-line laser radar and the fixed laser radar.

[0050] Specifically, jointly calibrating the multi-line laser radar and the camera to obtain the rotation and translation of the multi-line laser radar relative to the camera further includes calibrating the camera intrinsic parameters through the following formula:

[0051]

[0052] Among them, (X w ,Y w ,Z w ) is the world coordinate system; (u, v) is the pixel coordinate; the camera coordinate system uses the optical axis of the camera as the z-axis, and the center of the optical system of the camera is the origin Oc. The camera coordinate system Xc, Yc are parallel to the X and Y axes of the image coordinate system respectively. The distance f between the origin of the camera coordinate system and the origin of the image coordinate system is the focal length. is the camera intrinsic parameter matrix, and the camera intrinsic parameter matrix is ​​solved using Zhang Zhengyou calibration method; is the camera extrinsic parameter matrix; and the extrinsic parameters of the multi-line lidar and camera are jointly coarsely calibrated; and the extrinsic parameters of the multi-line lidar and camera are jointly finely calibrated.

[0053] Specifically, the joint calibration of a multi-line laser radar and a solid-state laser radar to calculate extrinsic parameters between the two radars further includes: collecting point cloud data from the two laser radars in a standard indoor space; providing plane features from the point cloud data; matching the plane features; after completing the plane feature matching, using singular value decomposition to solve for the initial values ​​of R and t; and establishing an optimization function using the squared distance from the point to the plane as the objective function. This optimization function can improve the accuracy of the extrinsic parameter calibration based on the initial values.

[0054] In step S104, the multi-line laser radar is positioned using the laser point set and the image point set obtained by the portable laser scanner to obtain the position and posture of the multi-line laser radar in the world coordinate system. Specifically, the multi-line laser radar is positioned using the laser point set and the image point set obtained by the portable laser scanner to obtain the position and posture of the multi-line laser radar in the world coordinate system further includes: performing high-precision positioning based on direct matching or performing high-precision positioning based on feature matching, wherein the high-precision positioning based on feature matching further includes: calculating the inclination angle ω of the laser beam relative to the horizontal plane of the laser radar according to the laser point coordinates (x, y, z):

[0055]

[0056] The relative pose of the k+1th frame and the kth frame is:

[0057]

[0058] Construct residual functions of line features and surface features to solve the pose vector.

[0059] In step S106, a visible light image and a laser point cloud of the target to be identified are obtained and input into a multimodal target recognition model to obtain the position of the target to be identified in the camera coordinate system. Specifically, obtaining the visible light image and the laser point cloud of the target to be identified and inputting them into the multimodal target recognition model to obtain the position of the target to be identified in the camera coordinate system further includes: establishing a deep learning neural network. Specifically, establishing the deep learning neural network includes: first, establishing a point cloud branch network, which utilizes a first feature extractor; and establishing an image branch network, which utilizes a second feature extractor. Multiple fusion module networks are connected between the point cloud branch network and the image branch network to fuse lidar features and visual semantic features at multiple levels. Then, the deep learning neural network is trained using the labeled visible light image and the labeled laser point cloud to obtain a multimodal target recognition model. A visible light image of the target to be identified is then captured using a camera, and a laser point cloud of the target to be identified is scanned using a lidar. The visible light image and the laser point cloud of the target to be identified are then input into the multimodal target recognition model to determine the type of the target to be identified and its position in the camera coordinate system.

[0060] In step S108, the position of the target to be identified in the world coordinate system is obtained based on the extrinsic parameters from the camera local coordinate system to the base coordinate center, the pose of the multi-line laser radar in the world coordinate system, and the position of the target to be identified in the camera coordinate system. Specifically, obtaining the position Pw of the target to be identified in the world coordinate system by the following formula further includes:

[0061] Pw=Twl*Tlc*Pc

[0062] Among them, the position of the target to be identified in the camera coordinate system is Pc; the pose of the multi-line laser radar in the world coordinate system is Twl; and the external parameters from the camera local coordinate system to the base coordinate center are Tlc.

[0063] In step S110, the target to be identified is added to the Building Information Model (BIM) based on its position in the world coordinate system. Specifically, BIM uses digital means to create a virtual building in a computer. This virtual building provides a single, complete, and logically related building information library. This "information" encompasses not only visual information describing geometric shapes but also a wealth of non-geometric information, such as the fire resistance rating and heat transfer coefficient of materials, component costs, and procurement information.

[0064] Another specific embodiment of the present invention discloses an automatic identification and positioning device for indoor objects. Figure 13The indoor object automatic identification and positioning device includes: a calibration module 1302 , a positioning module 1304 , a multimodal target recognition module 1306 , a target position acquisition module 1308 and an adding module 1310 .

[0065] Calibration module 1302 is used to calibrate the transformation relationship from a fixed laser radar and camera to a multi-line laser radar to obtain the extrinsic parameters from the camera's local coordinate system to the base coordinate center. The portable laser scanner includes a multi-line laser radar, a fixed laser radar, and a camera. Positioning module 1304 is used to locate the multi-line laser radar using the laser point set and image point set acquired by the portable laser scanner to obtain the multi-line laser radar's position in the world coordinate system. Multimodal target recognition module 1306 is used to acquire a visible light image and laser point cloud of the target to be identified and input them into a multimodal target recognition model to obtain the target's position in the camera coordinate system. Specifically, the multimodal target recognition module 1306 further includes: a network establishment submodule for establishing a deep learning neural network; a training submodule for training the deep learning neural network using labeled visible light images and labeled laser point cloud images to obtain a multimodal target recognition model; a portable laser scanner for capturing visible light images of the target to be recognized using a camera and scanning the laser point cloud image of the target to be recognized using a laser radar; and a position acquisition submodule for inputting the visible light image and the laser point cloud image of the target to be recognized into the multimodal target recognition model to obtain the type of the target to be recognized and its position in the camera coordinate system. The target position acquisition module 1308 is used to acquire the position of the target to be recognized in the world coordinate system based on the extrinsic parameters from the camera local coordinate system to the base coordinate center, the pose of the multi-line laser radar in the world coordinate system, and the position of the target to be recognized in the camera coordinate system. The addition module 1310 is used to add the target to be recognized to the building information model based on its position in the world coordinate system.

[0066] In the following, reference will be made to Figures 2 to 12 , the method for automatic identification and positioning of indoor objects according to an embodiment of the present invention is described in detail by way of specific examples.

[0067] The technology behind handheld intelligent monitoring devices involves several components, including exterior and structural design, hardware system design, software system design (embedded software, data acquisition software, and intelligent processing software), and core algorithm design. The core algorithm is crucial to this application, and its rationality directly impacts the device's overall performance. This article primarily provides a detailed and in-depth explanation of this principle.

[0068] The core algorithms of handheld intelligent monitoring devices primarily consist of four components: a multi-sensor high-precision spatial calibration algorithm, a lidar-based high-precision positioning algorithm, a deep learning-based multimodal target recognition algorithm, and a multi-sensor fusion-based target position acquisition algorithm. Each of these components will be explained in detail below.

[0069] 1. Multi-sensor high-precision spatial calibration algorithm

[0070] The core sensors in this system are a multi-line laser radar, a solid-state laser radar, and a visible light camera. Therefore, sensor calibration mainly refers to the development of a set of rigorous algorithms and operating procedures through precise design, and ultimately obtaining high-precision extrinsic parameters between these three sensors. The extrinsic parameters are represented by a 4*4 spatial transformation matrix or a combination of a position vector and an Euler angle vector (quaternion).

[0071] The relationship between Euler angle and rotation matrix transformation:

[0072] Assume that the rotation is in the order of ZYX (2, 1, 0), and the rotation angles are φ, θ, and ψ respectively. The rotation matrix is ​​expressed as follows:

[0073]

[0074] This embodiment uses multi-line laser as the basic coordinate reference to calibrate the visible light camera and the transformation relationship from solid-state laser to multi-line laser.

[0075] 1.1 Multi-line laser and camera joint calibration

[0076] like Figure 2 As shown in Figure 2, the result of the joint calibration of the lidar and camera is the rotation and translation of the lidar relative to the camera.

[0077] The joint calibration is carried out in two steps: first calibrating the camera intrinsic parameters and then jointly calibrating the extrinsic parameters.

[0078] 1.1.1 Camera internal parameter calibration

[0079] Concept of intrinsic parameter calibration: In image measurement and computer vision, in order to determine the three-dimensional geometric position of a point on a spatial object and the relationship between its corresponding point in the image, a geometric model of camera imaging must be established. The parameters of the model are the parameters of the camera, and the process of solving the parameters is called camera calibration.

[0080] The experimental process of intrinsic parameter calibration is as follows:

[0081] (1) Print a piece of A4 paper with a checkerboard pattern (the black and white spacing is known) and stick it on a flat plate.

[0082] (2) Take several pictures of the chessboard (usually 10-20 pictures).

[0083] (3) Detect feature points (Harris features) in the image.

[0084] (4) Calculate five internal parameters and six external parameters using analytical solution estimation methods.

[0085] (5) Based on the maximum likelihood estimation strategy, design the optimization objective and implement parameter refinement.

[0086] The imaging process of the camera can be expressed as:

[0087]

[0088] Among them, the world coordinate system (Xw, Yw, Zw), camera coordinates (Xc, Yc, Zc), pixel coordinates (u, v), the camera coordinate system uses the optical axis of the camera as the z-axis, the center position of the light in the camera's optical system is the origin Oc, the camera coordinate system Xc, Yc are parallel to the image coordinate system X, Y axis respectively. The distance f between the origin of the camera coordinate system and the origin of the image coordinate system is the focal length is the camera intrinsic parameter matrix, and the camera intrinsic parameter matrix is ​​solved using Zhang Zhengyou calibration method; is the camera extrinsic parameter matrix.

[0089] 1.1.2 Joint calibration of external parameters

[0090] Rough calibration:

[0091] (1) Description of landmark points

[0092] This paper uses markers for calibration because the edges of markers are easier to detect in the two sensor data. During the calibration process, a laser and camera are used to collect one frame of data respectively as the basic data.

[0093] (2) Detection under point cloud

[0094] like Figure 3 As shown in Figure 2, since the depth information is discontinuous, edge detection can be achieved by detecting the depth difference between adjacent points on the same line. Figure 3 (a) and (c) are the original detection images, and (b) and (d) are the laser point cloud images corresponding to (a) and (c) respectively.

[0095] The size, number, and position of the targets on the actual calibration board are known, and detection, verification, and point cloud pruning are then required.

[0096] a. Keep the inner points of the plane and remove the outer points. The inner points will be processed as follows: Figure 4 As shown in (a) in .

[0097] b. Use the random sampling consensus algorithm RANSAC (full name Random sample consensus) method to extract the center of the circle for the given target (four circles forming a square, radius, and point distance), such as Figure 4 As shown in (b) in .

[0098] c. Verify the test results. If passed, the test ends; if not, proceed to the next step.

[0099] d. If the points cannot be extracted, the point cloud needs to be processed and trimmed. Set rules to extract the points you want to extract.

[0100] (3) Detection in the camera

[0101] The Sobel operator is used to extract the image edge, and then the Hough transform is used to extract the circular boundary.

[0102] (4) Calculate the initial translation vector (t x t y t z )

[0103] Calculate t by the following formula z , r 3D Represents the radius of the point cloud detection, r 2D Represents the radius detected by the image, Z represents the depth coordinate of the center of the circle detected by the image, and f represents the focal length:

[0104]

[0105] X, Y, and Z represent the coordinates of the point under the laser radar, x and y represent the coordinates in the image coordinate system, and o x 、o y Indicates the principal point deviation.

[0106]

[0107]

[0108] Precision calibration:

[0109] The process of fine calibration is to search for the optimal solution of parameters in a small parameter search space. Based on the premise that the edges detected by the camera and the edges detected by the lidar match each other, the loss function is designed as follows:

[0110] a: Project the target point in the LiDAR coordinate system onto the camera plane to construct a two-dimensional image plane;

[0111] b: After being processed by the Sobel operator, an image containing edge information is generated;

[0112] c: Use the Inverse Distance Transform (IDT) method to perform L1 regularization on the edge information of the image.

[0113] d: Construct error function S E , use nonlinear optimization methods to optimize external parameters to minimize the value of the loss function.

[0114]

[0115] The above formula represents the setting of Loss function, I C Represents the data in the processed image coordinate system, I V It represents the data in the radar coordinate system after processing and projection. Therefore, optimization is performed on the basis of rough calibration to improve the calibration accuracy.

[0116] 1.2 Joint calibration of multi-line laser and solid-state laser

[0117] This project uses a calibration method to calculate the external parameters between lidars by surface feature matching, and collects point cloud data of two lasers as basic data in a standard calibration room.

[0118] (1) Face feature extraction

[0119] a. Point cloud preprocessing: Due to the influence of noise, there are many points that are not on the plane. Therefore, point cloud preprocessing is performed first to remove the points that are not on the plane.

[0120] b. Use RANSAC to fit a plane between two point clouds, which may result in many planes being fitted.

[0121] c. Since a plane normal vector is equivalent to a coordinate axis, only three coordinate axes need to be matched to solve the rotation of the two coordinate systems. When using RANSAC to fit two point clouds, there may be more than three planes. Based on the number of plane points, the three largest planes are retained.

[0122] d. Solve for the parametric coefficients of the plane. The plane equation is usually expressed as:

[0123] β (i,0) x n +β (i,1) y n +β (i,2) z n +β(i,3) =0

[0124] Among them, β (i,0) , β (i,1) , β (i,2) and β (i,3) are parameters respectively. Therefore, the distance from a point to a plane can be defined as:

[0125] f i (P n )=|β (i,0) x n +β (i,1) y n +β (i,2) z n +β (i,3) |

[0126] e. Construct a least squares problem. Select N points on the plane. The plane coefficients should satisfy the following least squares problem:

[0127]

[0128] f. After solving the plane coefficients, consider the three planes as three planes of the XYZ coordinate axis. The origin of the local coordinate system composed of these three planes is the intersection of the three planes:

[0129]

[0130] (2) Face feature matching

[0131] like Figure 5 As shown in the figure, the ground is usually selected as a feature surface during the calibration process, and the installation of the lidar is generally approximately parallel to the ground. Therefore, after setting the normal vector direction, use:

[0132] n 地面 =max([0,0,1]n i )

[0133] Determine which plane in the point cloud is the ground. After setting the local right-handed coordinate system, the three normal vectors n1, n2, and n3 satisfy the following relationship:

[0134] (n2×n1)n3>0

[0135] By traversing the combination, each plane can be assigned a corresponding number to complete the matching of plane features.

[0136] (3) Closed-form solution of initial value of external parameter

[0137] After completing the plane feature matching, the three normal vectors are treated as three points and the initial value of R can be solved by SVD decomposition. Since the three normal vectors have the following corresponding relationship:

[0138] Rn1=n′1, Rn2=n′2, Rn3=n′3

[0139] R[n1,n2,n3]=[n′1,n′2,n′3]

[0140] P=[n1,n2,n3],Q=[n′1,n′2,n′3]

[0141] H=PQ T

[0142] Performing SVD decomposition on the H matrix can obtain the intermediate variables V and U. The initial values ​​of R and t can be solved using the following formula, where O and O′ are the origin coordinates of the local coordinate systems of the two lidars, respectively.

[0143] R=VU T ,t=O′-RO

[0144] (4) Nonlinear optimization

[0145] Finally, we use the square of the distance from the point to the plane as the objective function, establish an optimization function, and perform iterative optimization. The optimization function is to sum the error values ​​of the three matching planes respectively, where the error of each matching plane consists of two parts: the first part is the distance from the point on the solid-state laser plane to the corresponding plane of the multi-line laser, and p′ is the coordinate of the point on the solid-state laser plane; Part II is the distance from a point on the multi-line laser plane to the corresponding plane of the solid-state laser, and p is the coordinate of the point on the multi-line laser plane.

[0146]

[0147] 2. High-precision positioning algorithm based on lidar

[0148] This module mainly adopts a high-precision positioning algorithm based on direct matching and feature fusion.

[0149] 2.1 High-precision positioning algorithm based on direct matching

[0150] For two point sets:

[0151] X={x1,x2,...,x Nx}

[0152] Y={y1,y2,...,y Ny}

[0153] refer to Figure 6First, preprocess the point cloud data; divide the space into grids and count the points that fall within each grid; calculate the mean and covariance of each grid based on the points in each grid, and construct a Gaussian distribution; calculate the joint probability based on the predicted pose; solve R,t and determine whether the solution is complete. If not, return to calculating the joint probability based on the predicted pose. If it is complete, output R,t.

[0154] Find the objective function such that:

[0155]

[0156] in:

[0157]

[0158] The objective function is defined as:

[0159]

[0160] y′ i =T(p,y i )=Py i +t

[0161] The objective function is defined as follows, μ is the center of mass of the point set X, y′ i To use the predicted pose for point y i The coordinates of the point obtained after the transformation.

[0162] According to the process of Gauss-Newton method, iterative optimization can be performed by simply calculating the Jacobian of the residual function with respect to the parameters to be determined.

[0163] 2.2 High-precision positioning algorithm based on feature matching

[0164] (1) Line and surface feature extraction

[0165] refer to Figure 7 , according to the laser point coordinates (x, y, z), the inclination angle ω of the laser beam compared to the radar horizontal plane can be calculated:

[0166]

[0167] refer to Figure 8 ,According to the inclination angle and the radar internal parameters (the designed inclination angle of each scanning line), it can be known to which laser beam the radar belongs.

[0168] The curvature is calculated based on the length X between the adjacent points and the current point (the length refers to the distance from the laser point to the radar). The feature is then judged based on the curvature.

[0169]

[0170] (2) Line-surface feature association

[0171] The relative pose of the k+1th frame and the kth frame is:

[0172]

[0173] The points in the k+1th frame are transferred to the kth frame coordinate system:

[0174]

[0175] p i is a line feature, is the predicted line feature.

[0176] (3) Posture optimization

[0177] Construct the residual functions of line features and surface features respectively:

[0178]

[0179]

[0180] in, is the predicted line feature; |p a -p b | is the length of the line feature, when p i When it is a line feature, search for the nearest line feature point p in the previous frame a , and find another line feature point p on the adjacent line b , forming a straight line. When it is a surface feature, search for the nearest surface feature point p in the previous frame m , and find two surface feature points p on adjacent lines j and p l , forming a plane.

[0181] According to the convex optimization foundation, we only need to obtain the Jacobian of the residual with respect to the variable to be determined, and then we can use Gauss-Newton optimization to solve the pose vector.

[0182] 3. Multimodal object recognition based on deep learning

[0183] Traditional deep learning-based image target recognition technology has become stable and mature, but it is prone to false detection and missed detection in small targets and complex scenes, which means that it can currently only be applied to target recognition scenarios with relatively simple environments. Deep learning is essentially a data-driven pattern expression learning method. Its current poor performance in complex situations is mainly limited by the limitations of single sensor data. Multimodal data can effectively express the environment in different feature dimensions, and its huge advantages are gradually attracting the attention of researchers. This project proposes to adopt a new deep neural network structure. The network input is multimodal sensor data, including visible light images and laser point clouds. The network parameters are used to learn the feature expression of different input data, and the output is the type and location of the target. Using a certain amount of labeled data for training and learning, the parameters can be automatically tuned, thereby significantly improving the accuracy of target recognition and environmental adaptability.

[0184] Theoretically, image information is dense and regular, containing rich color and texture information. However, its disadvantage is that it is two-dimensional. This leads to scaling issues due to distance. Compared to images, point clouds are sparse and irregular, making direct processing on point clouds using traditional CNN perception impractical. However, point clouds contain three-dimensional geometric structure and depth information, which is more beneficial for 3D object detection. Therefore, the two types of information are theoretically complementary. Furthermore, current deep learning methods for 2D image detection are all based on CNNs, while point cloud object detection uses a variety of networks with basic structures, including MLPs, CNNs, and GCNs. Therefore, the specific network to be integrated during the fusion process is a matter of considerable research.

[0185] This project mainly adopts the method based on feature fusion, such as Figure 9 This fusion requires some interaction at the feature level. The primary fusion approach is to use feature extractors for both the point cloud and image branches, fusing the image and point cloud networks at semantic levels in the feedforward hierarchy to achieve semantic fusion of multi-scale information.

[0186] The network structure used in this project is as follows Figure 10 As shown in the figure, the point cloud branch is a point encoder-decoder structure, and the image branch is a step-by-step encoding network, and feature fusion is performed layer by layer.

[0187] The network consists of a two-stream RPN network and an optimization network. The RPN network is mainly used for candidate box generation, while the optimization network is mainly used for object detection box optimization. The network can be directly trained end-to-end. Through the L1-Fusion module, the network can effectively integrate LiDAR features and visual semantic features.

[0188] L1-Fusion module network structure, such as Figure 11 As shown:

[0189] In addition, the network designs a new loss function (CE loss) to further improve the consistency of target classification and localization confidence level.

[0190] CE loss is defined as follows:

[0191]

[0192] Where D and G represent the predicted and true bounding boxes respectively, and c represents the confidence of D's classification.

[0193] The total residual function L total The definition is as follows:

[0194] L total =L rpn +L rcnn ,

[0195] Among them L rpn and L rcnn Represent the residual functions of the two sub-networks rpn and rcnn respectively.

[0196] 4. Target position acquisition based on multi-sensor fusion

[0197] refer to Figure 12 After the three steps of sensor calibration, high-precision positioning, and target detection of multimodal data, the results of the above three steps can be used for fusion calculation to solve the absolute position of the target of interest in the world coordinate system. The entire calculation process is summarized as follows:

[0198] (1) At time t, the target of interest is detected based on the multimodal fusion results, and its position in the camera coordinate system is Pc;

[0199] (2) The pose of the base coordinate center (multi-line laser) in the world coordinate system at this moment has been solved as Twl;

[0200] (3) The external parameters of each sensor have been calibrated with high precision, where the external parameter from the camera local coordinate system to the base coordinate center is Tlc;

[0201] (4) The position Pw of the target of interest in the world coordinate system can be solved by spatial transformation, and its calculation formula is:

[0202] Pw=Twl*Tlc*Pc

[0203] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0204] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A method for automatically identifying and locating indoor objects, characterized in that: include: Calibrate the transformation relationship between the fixed laser radar and the camera and the multi-line laser radar to obtain the extrinsic parameters of the camera local coordinate system to the base coordinate center, wherein the portable laser scanner includes the multi-line laser radar, the fixed laser radar and the camera; The multi-line laser radar is positioned using the laser point set and the image point set obtained by the portable laser scanner to obtain the position and posture of the multi-line laser radar in the world coordinate system, wherein the multi-line laser radar is positioned using the laser point set and the image point set obtained by the portable laser scanner to obtain the position and posture of the multi-line laser radar in the world coordinate system further includes: performing high-precision positioning based on direct matching or performing high-precision positioning based on feature matching, wherein the high-precision positioning based on feature matching further includes: calculating, according to the laser point coordinates (x, y, z), the inclination angle ω of the laser at the laser point coordinates compared to the horizontal plane of the laser radar: The relative pose of the k+1th frame and the kth frame is: Construct the residual function of line features and surface features to solve the pose vector; Obtaining a visible light image and a laser point cloud image of a target to be identified and inputting them into a multimodal target recognition model to obtain the position of the target to be identified in a camera coordinate system; Based on the extrinsic parameters from the camera local coordinate system to the base coordinate center, the pose of the multi-line laser radar in the world coordinate system, and the position of the target to be identified in the camera coordinate system, the position of the target to be identified in the world coordinate system is obtained, wherein obtaining the position Pw of the target to be identified in the world coordinate system by the following formula further includes: Pw=Twl*Tlc*Pc; Wherein, the position of the target to be identified in the camera coordinate system is Pc; the pose of the multi-line laser radar in the world coordinate system is Twl; and the external parameter of the camera local coordinate system to the base coordinate center is Tlc; and According to the position of the target to be identified in the world coordinate system, the target to be identified is added to the building information model to reflect the recording and change of building information.

2. The method for automatically identifying and locating indoor objects according to claim 1, characterized in that: Acquiring a visible light image and a laser point cloud of a target to be identified and inputting them into a multimodal target recognition model to obtain a position of the target to be identified in a camera coordinate system further includes: Establishing a deep learning neural network, and training the deep learning neural network using the labeled visible light image and the labeled laser point cloud image to obtain the multimodal target recognition model; The camera is used to capture a visible light image of the target to be identified and a laser radar is used to scan a laser point cloud map of the target to be identified, and the visible light image of the target to be identified and the laser point cloud map of the target to be identified are input into the multimodal target recognition model to obtain the type of the target to be identified and the position in the camera coordinate system.

3. The method for automatically identifying and locating indoor objects according to claim 2, characterized in that: Building a deep learning neural network involves: Establishing a point cloud branch network, wherein the point cloud branch network adopts a first feature extractor; An image branch network is established, wherein the image branch network adopts a second feature extractor, wherein a plurality of fusion module networks are connected between the point cloud branch network and the image branch network to fuse lidar features and visual semantic features at multiple levels.

4. The method for automatically identifying and locating indoor objects according to claim 1, characterized in that: Calibrating the transformation relationship between the fixed laser radar and the camera and the multi-line laser radar to obtain the extrinsic parameters of the camera local coordinate system to the base coordinate center further includes: Jointly calibrating the multi-line laser radar and the camera to obtain a rotation and a translation of the multi-line laser radar relative to the camera; and The multi-line laser radar and the solid-state laser radar are jointly calibrated to calculate the external parameters between the multi-line laser radar and the fixed laser radar.

5. The method for automatic identification and positioning of indoor objects according to claim 2, characterized in that: Jointly calibrating the multi-line laser radar and the camera to obtain the rotation and translation of the multi-line laser radar relative to the camera further includes: The following formula is used to calibrate the camera internal parameters: Among them, Zc is the scale parameter; (X w ,Y w ,Z w ) is the world coordinate system; (u, v) is the pixel coordinate; the camera coordinate system uses the optical axis of the camera as the z-axis, and the center of the optical system of the camera is the origin Oc. The camera coordinate system Xc and Yc are parallel to the X and Y axes of the image coordinate system respectively. The distance f between the origin of the camera coordinate system and the origin of the image coordinate system is the focal length. is the camera intrinsic parameter matrix, and the camera intrinsic parameter matrix is ​​solved using Zhang Zhengyou calibration method; is the camera extrinsic parameter matrix; and Performing a joint rough calibration of the extrinsic parameters of the multi-line laser radar and the camera; and Perform joint precise calibration of the external parameters of the multi-line laser radar and the camera.

6. The method for automatic identification and positioning of indoor objects according to claim 4, characterized in that: Jointly calibrating the multi-line laser radar and the solid-state laser radar to calculate external parameters between the multi-line laser radar and the fixed laser radar further includes: Collect point cloud data from two lidars in a standard indoor space; providing planar features from the point cloud data; matching the planar features; After completing the plane feature matching, the initial values ​​of R and t are solved using singular value decomposition; and The optimization function is established by taking the square of the distance from the point to the plane as the objective function.

7. An automatic identification and positioning device for indoor objects, characterized in that: include: a calibration module for calibrating the transformation relationship between the fixed laser radar and the camera and the multi-line laser radar to obtain the extrinsic parameters of the camera local coordinate system to the base coordinate center, wherein the portable laser scanner includes the multi-line laser radar, the fixed laser radar and the camera; A positioning module, configured to use the laser point set and the image point set acquired by the portable laser scanner to position the multi-line laser radar to obtain the position and posture of the multi-line laser radar in a world coordinate system, wherein using the laser point set and the image point set acquired by the portable laser scanner to position the multi-line laser radar to obtain the position and posture of the multi-line laser radar in a world coordinate system further includes: performing high-precision positioning based on direct matching or performing high-precision positioning based on feature matching, wherein performing high-precision positioning based on feature matching further includes: calculating, according to the laser point coordinates (x, y, z), the inclination angle ω of the laser at the laser point coordinates compared to the horizontal plane of the laser radar: The relative pose of the k+1th frame and the kth frame is: Construct the residual function of line features and surface features to solve the pose vector; A multimodal target recognition module is used to obtain a visible light image and a laser point cloud image of a target to be recognized and input them into a multimodal target recognition model to obtain the position of the target to be recognized in the camera coordinate system; A target position acquisition module is configured to acquire the position of the target to be identified in the world coordinate system based on the extrinsic parameters from the camera local coordinate system to the base coordinate center, the pose of the multi-line laser radar in the world coordinate system, and the position of the target to be identified in the camera coordinate system. The position Pw of the target to be identified in the world coordinate system is further acquired by the following formula: Pw=Twl*Tlc*Pc; Wherein, the position of the target to be identified in the camera coordinate system is Pc; the pose of the multi-line laser radar in the world coordinate system is Twl; and the external parameter of the camera local coordinate system to the base coordinate center is Tlc; and The adding module is used to add the target to be identified to the building information model according to the position of the target to be identified in the world coordinate system to reflect the recording and change of building information.

8. The automatic identification and positioning device for indoor objects according to claim 7, characterized in that: The multimodal target recognition module further comprises: Network building submodule, used to build deep learning neural networks; A training submodule, configured to train the deep learning neural network using the labeled visible light image and the labeled laser point cloud image to obtain the multimodal target recognition model; a portable laser scanner, which uses the camera to capture a visible light image of the target to be identified and uses a laser radar to scan a laser point cloud image of the target to be identified; and The position acquisition submodule is used to input the visible light image of the target to be identified and the laser point cloud image of the target to be identified into the multimodal target recognition model to obtain the type of the target to be identified and the position in the camera coordinate system.

Images