Robot scene positioning method and construction operation method

Abstract

The present invention discloses a robot scene positioning method and a construction operation method. It relates to the field of robot control, wherein the robot scene positioning method generates a completion state grid map of a building by extracting structural information according to BIM data, obtains scanning data of the current scene by a laser radar installed on the robot, and uses an adaptive Monte Carlo algorithm to locate the robot in the current scene according to the scanning data and the completion state grid map to obtain the current positioning information of the robot. Based on BIM model information as prior information, information about the structural dimensions of the building is extracted from it and converted into a completion state grid map that can be used for robot positioning. The completion map positioning method with redundant information can be used throughout the entire engineering cycle, which simplifies the positioning process and improves the positioning efficiency of the robot in changing scenes.

Description

Technical Field

[0001] The present invention relates to the field of robot control, and in particular to a robot scene positioning method and a construction operation method. Background Art

[0002] The main principle of simultaneous localization and mapping (SLAM) is to detect the surrounding environment through the sensors on the robot and build an environment map while estimating the posture (posture includes position and direction). Taking visual SLAM, which is currently widely used for indoor robot positioning, as an example, after arriving at the construction site, the robot needs to be manually controlled to build a visual feature map of the scene. Especially when the site is large and the structure is complex, it will consume a lot of working time and manpower and material resources, reduce the autonomy of the robot, and lose the meaning of using robots to improve decoration automation. In addition, due to the particularity of building decoration work, there is a large-scale interaction between the construction environment and the robot. As the construction process proceeds, the actual environment in which the robot is located will continue to change, such as gradually building a wall and finally matching the finished drawing. In the face of changing scenes, traditionally, it is necessary to constantly re-build the map to ensure the positioning accuracy of the robot in the changing scene, resulting in reduced positioning efficiency. Therefore, it is necessary to propose a robot scene positioning method that can improve the positioning efficiency of the robot in the changing scene. Summary of the invention

[0003] The present invention aims to solve at least one of the technical problems existing in the related art. To this end, the present invention proposes a robot scene positioning method, which can improve the positioning efficiency of the robot in a changing scene, realize a positioning method that does not require prior mapping and can be used throughout the entire engineering cycle.

[0004] In a first aspect, an embodiment of the present invention provides: a robot scene positioning method, comprising:

[0005] Extract structural information from BIM data to generate a grid diagram of the building's completed status;

[0006] Obtaining scanning data of the current scene from a laser radar installed on the robot;

[0007] The robot is positioned in the current scene according to the scan data and the completion state grid diagram using an adaptive Monte Carlo algorithm to obtain current positioning information of the robot.

[0008] Furthermore, the using of the adaptive Monte Carlo algorithm to locate the robot in the current scene according to the scan data and the completion state grid diagram to obtain the current positioning information of the robot includes:

[0009] Acquiring engineering structure information contained in the scan data;

[0010] updating the likelihood domain of each particle in the adaptive Monte Carlo algorithm in real time according to the engineering structure information;

[0011] Using the likelihood domain as a weight to screen particles;

[0012] The robot is positioned in the current scene according to the screening result and the completion state grid diagram to obtain the current positioning information of the robot.

[0013] The embodiments of the present invention have at least the following beneficial effects: simplifying the positioning process and improving the positioning efficiency of the robot in changing scenes

[0014] In a second aspect, an embodiment of the present invention provides: a construction operation method, comprising:

[0015] Obtaining the current positioning information of the robot using the robot scene positioning method as described in any one of the first aspects;

[0016] Using the current positioning information to perform inter-frame pose tracking to obtain an estimated pose to be optimized;

[0017] Performing joint optimization on the estimated pose to be optimized to obtain a joint optimized pose;

[0018] The construction operation is performed according to the joint optimized posture and the three-dimensional position information obtained by feature extraction.

[0019] Furthermore, the step of using the current positioning information as an initial pose to perform inter-frame pose tracking to obtain an estimated pose to be optimized includes:

[0020] Obtaining current camera positioning information according to the transformation matrix between the robot and the camera and the current positioning information;

[0021] The camera positioning information is used as the initial position and posture to obtain the camera frame positioning difference, and the camera projection error and the camera frame positioning difference are used to obtain the estimated position and posture to be optimized.

[0022] Furthermore, the joint optimization is performed using a nonlinear optimization bundle adjustment method.

[0023] Furthermore, the camera projection error is obtained by feature matching, including:

[0024] Using a laser projector to project the environment to obtain a number of projection spots;

[0025] Acquire the projection light spot by using a camera device to perform ORB feature matching;

[0026] The camera projection error is obtained according to the feature matching result.

[0027] Furthermore, it also includes using a neural network to perform operation recognition, determining the three-dimensional position information of the operation target according to the operation recognition result, and the robot performs construction operations according to the joint optimized posture and the three-dimensional position information.

[0028] The embodiments of the present invention have at least the following beneficial effects: using the current positioning information obtained from the BIM model information, performing inter-frame pose tracking of the front-end fused BIM information and joint optimization of the back-end fused BIM information, thereby improving the accuracy of traditional visual SLAM.

[0029] In a third aspect, an embodiment of the present invention provides: a construction operation system, comprising:

[0030] A current positioning information acquisition unit: used to obtain the current positioning information of the robot using the robot scene positioning method as described in any one of the first aspects;

[0031] An inter-frame pose tracking unit: used to use the current positioning information to perform inter-frame pose tracking to obtain an estimated pose to be optimized;

[0032] A joint optimization unit: used for jointly optimizing the estimated pose to be optimized to obtain a joint optimized pose;

[0033] Construction operation unit: used for performing construction operations according to the joint optimized posture and the operation position information obtained by feature extraction.

[0034] In a fourth aspect, an embodiment of the present invention provides: an electronic device, characterized in that it includes:

[0035] at least one processor; and a memory communicatively coupled to the at least one processor;

[0036] The processor is used to execute the method as described in any one of the first aspects, or to execute the method as described in any one of the second aspects, by calling the computer program stored in the memory.

[0037] In a fifth aspect, an embodiment of the present invention provides: a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, wherein the computer-executable instructions are used to enable a computer to execute a method as described in any one of the first aspect, or to execute a method as described in any one of the second aspect.

[0038] The beneficial effects of the embodiments of the present invention are:

[0039] The embodiment of the present invention generates a completion status grid map of the building by extracting structural information from BIM data, obtains scanning data of the current scene from the laser radar installed on the robot, and uses an adaptive Monte Carlo algorithm to locate the robot in the current scene based on the scanning data and the completion status grid map to obtain the current positioning information of the robot. There is no need to remotely control the robot to construct a visual feature map of the scene in advance at the beginning of the project, which avoids consuming a lot of working time and manpower and material resources when the site area is large and the structure is complex, and reduces the autonomy of the robot. Based on the BIM model information as a priori information, the information about the structural size of the house building is extracted from it and converted into a completion status grid map that can be used for robot positioning. The completion map positioning method with redundant information can be used throughout the entire project cycle, which simplifies the positioning process and improves the positioning efficiency of the robot in changing scenes.

[0040] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0042] Figure 1 It is a schematic flow chart of a specific embodiment of the robot scene positioning method in an embodiment of the present invention;

[0043] Figure 2 It is a schematic diagram of a BIM model diagram and a completion state grid diagram of a specific embodiment of the robot scene positioning method in an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of laser scanning of a specific embodiment of the robot scene positioning method in an embodiment of the present invention;

[0045] Figure 4 It is a schematic flow chart of a specific embodiment of the construction operation method in an embodiment of the present invention;

[0046] Figure 5 It is a schematic diagram of inter-frame pose tracking of a specific embodiment of the construction operation method in an embodiment of the present invention;

[0047] Figure 6-Figure 9 This is a schematic diagram of visual enhancement effect of a specific embodiment of the construction operation method in the embodiment of the present invention;

[0048] Fig.10 is a schematic diagram of reprojection error in the related art;

[0049] Fig.11 It is a schematic diagram of inter-frame constrained reprojection error based on BIM information in a specific embodiment of the construction operation method in an embodiment of the present invention;

[0050] Fig.12 It is a schematic diagram of two types of wall defects;

[0051] Fig.13 It is a schematic diagram of the detection effect of a YOLO convolutional neural network in a specific embodiment of the construction operation method in an embodiment of the present invention;

[0052] Fig.14 It is a structural block diagram of a specific embodiment of the construction operating system in the embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the specific implementation methods of the present invention will be described below with reference to the accompanying drawings. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings and other implementation methods can be obtained based on these drawings without creative work.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0055] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0056] Example:

[0057] The working environment of construction robots is different from that of general indoor mobile robots. It has the following three characteristics: 1. It can obtain prior information of the working environment, BIM (Building Information Modeling, referred to as BIM); 2. The working environment is mostly indoor rough walls with weak textures; 3. The working environment will undergo structural changes as the construction progresses. The mapping and positioning process framework of traditional visual SLAM is inefficient and cannot work in weak texture environments. In addition, when the environmental structure changes, the feature map established by traditional visual SLAM will not be usable and the map needs to be rebuilt, which is irreconcilable with the working scene of construction robots.

[0058] This embodiment obtains prior information of the building environment (BIM) in advance and combines binocular vision with 2D laser radar to perform repeatable and high-precision positioning.

[0059] An embodiment of the present invention provides a robot scene positioning method. Figure 1 A schematic diagram of a flow chart of a robot scene positioning method provided by an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0060] S11: Extract structural information based on BIM data to generate a grid diagram of the building's completion status.

[0061] The full name of BIM is Building Information Modeling. BIM integrates various information from the design, construction, operation to the end of the entire life cycle of a building into a three-dimensional model information database. By establishing a virtual three-dimensional model of the building project and using digital technology, a complete building project information database that is consistent with the actual situation is provided for this model. The database not only contains geometric information, professional attributes and status information describing the building components, but also contains status information of non-component objects (such as space and motion behavior). Since BIM includes information storage solutions in the field of construction and has rich building information, this embodiment extracts the available information in BIM according to the positioning requirements of the construction robot and uses it as prior information on the building environment.

[0062] In one implementation, BIM is saved in the standard exchange format IFC (Industry Foundation Class), and the information description of IFC is divided into four levels: resource layer (IFC-Resource Layer), core layer (IFC-CoreLayer), shared layer (IFC-Interoperability Layer) and domain layer (IFC-Domain Layer).

[0063] The resource layer is the basic layer of the entire system. Any IFC layer can reference entities in the resource layer. This layer mainly defines the general information of the project. This information is independent of the specific building and has no overall structure. It is scattered basic information. The core content of this layer mainly includes attribute resources, expression resources, and structural resources. These entity resources are mainly used to define the upper-level entity resources to display the attributes of the upper-level entities.

[0064] The core layer defines the overall framework of the information model, such as product, process, control and other related information. Its main function is to organize the scattered basic information in the lower layer to form the basic structure of the IFC model, and then use it to describe the physical objects and abstract processes in the real world. It plays a connecting role in the entire system. This layer refines and defines abstract concepts applicable to the entire construction industry. For example, the IFCProduct entity can describe the construction site, construction space, and construction components of a construction project.

[0065] The shared layer mainly serves the domain layer, enabling information between domains to interact and refining the components of the system. Specific building components such as plates (IFCSlab), columns (IFCColumn), beams (IFCBeam), walls (IfcWall), etc. are all defined at this layer. If a building has 100 walls, there will be 100 IfcWall instances in the neutral file.

[0066] The domain layer is the top layer of the IFC system architecture, which mainly defines entity types for various professional fields. These entities have specific concepts for various professional fields, such as boilers and pipes in the HVAC field.

[0067] like Figure 2 As shown, it is a schematic diagram of a BIM model diagram and a completion status grid diagram in an embodiment. It can be seen from the figure that the structure and size information in the project are extracted according to the inheritance relationship between the IFC entities in the BIM model diagram, and then the completion status grid diagram of the building is generated according to the preset resolution.

[0068] S12: Obtain scanning data of the current scene from the laser radar installed on the robot.

[0069] In one embodiment, a 2D laser radar is installed on the robot to scan the construction scene in real time to obtain on-site scanning data, and the scanning data includes engineering structure information, such as load-bearing walls, partition walls, etc.

[0070] S13: Using an adaptive Monte Carlo algorithm to locate the robot in the current scene according to the scan data and the completion state grid map to obtain the current positioning information of the robot.

[0071] In one implementation, the implementation process of step S13 is as follows:

[0072] S131: Acquire engineering structure information contained in the scanned data. According to the characteristics of environmental changes during the construction process, initially there are only load-bearing walls. As the construction progresses, partition walls are built. When all partition walls are built, the effect of the finished drawing is achieved.

[0073] S132: updating the likelihood domain of each particle in the adaptive Monte Carlo algorithm in real time according to the obstacle information.

[0074] S133: Using the likelihood domain as a weight to screen particles.

[0075] S134: Positioning the robot in the current scene according to the screening results and the completion status grid map to obtain the current positioning information of the robot.

[0076] For example, the scan_layer is updated, the grid map of the completed state is loaded in the potential layer through the potential_map_server, and all the engineering structure information such as partition walls and other obstacles in the laser radar scan data at the current position are obtained in the scan_layer. The particles are screened by calculating the likelihood domain (i.e. the above probability) of each particle as a weight. The smaller the weight of the particle, the less consistent the observation data obtained with it as the benchmark is with the actual map, so the particle is more likely to be screened out.

[0077] like Figure 3 As shown in FIG. 1 , a schematic diagram of laser scanning in one embodiment is shown. For all particles, if there is an unobservable wall in the map (as shown by the dotted line in the figure), the laser will pass through the wall. If the end can illuminate other walls, the weight can be calculated based on the likelihood domain of other walls (as shown in the left figure). If the end does not illuminate any wall, it will be discarded as an invalid value (as shown in the right figure). When an object appears in both the scan layer scan_layer and the potential layer potential_map_server, it is considered to be a wall.

[0078] In one embodiment, positioning is achieved using a particle filter of an adaptive Monte Carlo algorithm. Monte Carlo particle filter positioning can be viewed as the following process: for example, a group of particles are evenly distributed in the scene at the beginning, and the particles are moved by the movement of the robot. For example, when the robot moves forward one meter, all the particles also move forward one meter. Then, the position of each particle is used to simulate a sensor position information and compare it with the scanning data obtained by the robot through 2D lidar scanning in this embodiment, thereby assigning a probability to each particle, that is, a likelihood domain, and regenerating the particles according to the size of the generated probability. Generally speaking, the higher the probability, the greater the generation probability, that is, the likelihood domain is weighted and screened to leave particles with large weights. After multiple iterations, all the particles will slowly converge, and the robot is positioned according to the results of the particle convergence to obtain the current positioning information of the robot.

[0079] In one embodiment, there are many ways to calculate the weight. The weight can be simply considered as the difference between the actual sensor measurement value. For example, if the difference between the predicted position of a particle and the distance before the measurement target is large, the larger the difference, the smaller the weight. A small weight means that the distance is far from the actual position.

[0080] For example, the robot is initially located near a wall, but the robot does not know where it is. At this time, all particles are initialized, and the weights of all particles are the same. At this time, the robot knows through laser scanning data that there is a wall in front of it. At this time, the weights of particles close to the wall are increased. After resampling, the robot moves, and it can be seen that the sampled particles are concentrated near the particle points that were originally close to the wall. At this time, the robot re-perceives and knows that there is still a wall in front of it. At this time, the weights of the particles close to the wall that were originally sampled become larger. After screening, the particles with large weights are retained. At this time, the main sampling points are concentrated at the position of the wall. The robot knows through the screening results that its position is next to the wall, realizing the positioning of the robot.

[0081] Since any construction phase map is a subset of the map at the completion state, and the particle likelihood domain of this embodiment can be updated in real time, the current positioning information of the robot can be obtained in real time through weight screening. Therefore, the completion state grid map used in this embodiment can be used for navigation in different construction stages, and there is no need to remotely control the robot to build a visual feature map of the scene at the beginning of the project, so as to avoid consuming a lot of working time and manpower and material resources when the site area is large and the structure is complex, and reduce the autonomy of the robot. At the same time, this embodiment uses BIM model information as prior information, extracts information about the size of the building structure, and converts it into a completion state grid map that can be used for robot positioning. The completion map positioning method with redundant information can be used throughout the entire project cycle, which simplifies the positioning process. In the face of changing scenes, positioning can be done directly, and the positioning efficiency of the robot in changing scenes is improved.

[0082] In another embodiment, a construction operation method is provided, which is improved based on the traditional visual SLAM method. The problem that the traditional visual SLAM cannot meet the requirements of efficient and high-precision positioning and mapping in the case of changing environment and weak texture indoor decoration is solved, and at the same time, the constructed SLAM map has semantic information through deep learning.

[0083] The advantages and disadvantages of traditional laser SLAM and visual SLAM are very obvious: in the face of weak texture environment, the stability and accuracy of laser SLAM are better than traditional visual SLAM; and in terms of perceiving high-dimensional environmental information (such as identifying wall defect points, three-dimensional reconstruction, etc.), visual SLAM is better. However, visual SLAM has poor robustness to environmental changes and cannot be directly applied to construction scenes. It cannot work well in the decoration environment of indoor walls with weak texture. Therefore, this embodiment effectively combines the respective advantages of traditional laser SLAM and traditional visual SLAM through the construction operation method of fusion SLAM of vision combined with 2D laser radar.

[0084] like Figure 4 As shown, it is a schematic diagram of the process flow of the construction operation method provided by the embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0085] S21: obtaining the current positioning information of the robot using any one of the robot scene positioning methods described in the above embodiments, that is, obtaining the current positioning information of the robot through the above embodiments.

[0086] S22: Using the current positioning information to perform inter-frame pose tracking to obtain the estimated pose to be optimized. The inter-frame pose tracking of this embodiment can be regarded as the inter-frame pose tracking of the front-end fused BIM information.

[0087] The front end of traditional visual SLAM uses a motion uniform speed model, which assumes that the camera is always in a constant speed state, so the transformation matrix of the k+1th frame relative to the kth frame is the same as the transformation matrix of the kth frame relative to the k-1th frame. However, such an assumption obviously does not meet the working requirements of the construction robot. Therefore, in this embodiment, the uniform speed model is improved by integrating BIM information.

[0088] S23: Jointly optimize the estimated pose to be optimized to obtain a joint optimized pose. For example, in one implementation, a nonlinear optimization bundle adjustment method is used for joint optimization, and the nonlinear optimization bundle adjustment method (BundleAdjustment) is also called BA method.

[0089] S24: Construction operations are performed based on the joint optimized posture and the three-dimensional position information obtained by feature extraction, wherein the three-dimensional position information is a kind of semantic information. Maps are constructed based on the semantic information to identify the operated target. For example, when repairing a wall, the three-dimensional position information of the wall defect is used. The construction operation is to repair the wall defect by the robot, etc., thereby expanding the application scope of the robot.

[0090] In one implementation, the specific process of the inter-frame pose tracking of the front-end fusion BIM information in step S22 is as follows, including:

[0091] S221: Obtain the current camera positioning information according to the transformation matrix between the robot and the camera and the current positioning information, expressed as:

[0092] T 相机 =T 当前机器人定位 *T 机器人-相机

[0093] Among them, T 相机 Indicates the current camera positioning information, T 当前机器人定位 Indicates the current positioning information of the robot, T 机器人-相机 Represents the transformation matrix between the robot and the camera, which can be obtained as a priori parameters.

[0094] S222: Using the camera positioning information as the initial pose to obtain the camera frame positioning difference, and using the camera projection error and the camera frame positioning difference to obtain the estimated pose to be optimized.

[0095] In one embodiment, the current camera positioning information is used as the initial pose to replace the initial pose of the uniform motion model in the related art, and then based on the optimized camera frame positioning difference from the previous frame to the current frame, the three-dimensional feature points observed in the current frame are combined with the current camera projection error.

[0096] like Figure 5 As shown in FIG. 1 , it is a schematic diagram of inter-frame pose tracking of this embodiment, where P is a three-dimensional space point, P1 and P2 are observation matching point pairs of the three-dimensional space point in the camera's kth frame (the previous frame that has been optimized) and the k+1th frame (the current frame to be optimized), that is, the observation matching point pairs are obtained by feature matching, P2' is the projection point of the three-dimensional space point P in the camera's k+1th frame, and e 投影 represents the camera plane distance error between the observation point P2 and the projection point P2' in the current frame of the camera, that is, the camera projection error mentioned above, e BIM represents the camera frame positioning difference, that is, the error between the estimated pose to be optimized in the current frame of the camera and the initial pose of the camera obtained through the BIM information (the camera positioning information obtained in step S221).

[0097] The estimated pose to be optimized belongs to inter-frame tracking, and the current frame is optimized by observation, which is expressed as:

[0098]

[0099] Where ζ* represents the estimated pose to be optimized, i.e., the Lie algebra of the camera pose, u represents the pixel coordinates of the observation point P2 in the k+1th frame of the camera, Kexp(ζ^)P represents the projection of the three-dimensional space point P on the frame plane of the k+1th frame of the camera under the camera parameter K, and its projection value is expressed as P2',T w_curr represents the estimated pose of the camera in the k+1th frame to be optimized, Tw_BIM represents the camera positioning information of the k+1th frame obtained through BIM information, that is, the initial pose, e 投影 represents the camera plane distance error between the observation point P2 and the projection point P2' of the camera's k+1th frame, that is, the camera projection error mentioned above, e BIM It represents the positioning difference between camera frames, that is, the error between the estimated pose to be optimized in the current frame of the camera and the initial pose of the camera obtained through BIM information.

[0100] In one implementation, obtaining the camera projection error through feature matching includes:

[0101] S2221: Using a laser projector to project the environment to obtain a plurality of projection spots;

[0102] S2222: Using a camera to obtain a projection spot for ORB feature matching;

[0103] S2223: Obtain camera projection error according to the feature matching result.

[0104] Since the wall may lack texture features in the decoration environment, it will be impossible to extract visual features for feature matching. Therefore, this embodiment enhances the visual features of artificial projection for this situation. The specific method is to use a laser projector to form a dot matrix to project the environment onto the wall lacking visual features to obtain a number of projection spots, and use a camera device, such as a binocular camera, to capture these artificially projected projection spots to extract ORB features for ORB feature matching, and obtain the camera projection error according to the above method based on the feature matching results to calculate the estimated pose to be optimized.

[0105] In one embodiment, the purpose of setting up a laser projector dot matrix is ​​to meet the requirements of visual feature projection of the entire site when some construction scenes have large walls and large scene ranges. Therefore, an omnidirectional laser dot matrix projection device is designed. For example, the device is set up with 8 laser dot matrix projectors and evenly distributed on a 360-degree circular base. Each laser projector is equipped with a rotating base with 2 degrees of freedom, which can respectively realize rotation and pitch perpendicular to the horizontal plane, so as to meet the projection requirements of different house structures. At the same time, a tripod can be installed at the bottom to increase the height of the projection point and prevent debris from blocking the light path.

[0106] like Figure 6-Figure 9 The figure is a schematic diagram of the visual enhancement effect in one implementation of this embodiment. Figure 6 The visual features extracted from the indoor wall. Figure 7 It is the feature that is successfully matched between the two images. It can be found that due to the flatness of the wall and the sparse texture features, there are few feature points that can be successfully matched between adjacent frames. When the wall that needs to be operated on in the indoor painting process is a white wall, the visual features that can be extracted will be even sparser. Figure 8 This is a schematic diagram of the laser projector spot. Fig. 9 This is the feature matching after visual feature enhancement using a laser projector. It can be seen that after laser projection, the number of ORB features that can be extracted increases significantly, and feature matching can be achieved even on completely blank walls, realizing positioning and mapping in weak texture decoration environments. The camera projection error can be obtained according to the above method based on the feature matching results to calculate the estimated pose to be optimized, ultimately meeting the construction robot construction operating system requirements.

[0107] The above process of this embodiment completes the inter-frame pose tracking of the front-end fusion BIM information, and can estimate the movement of the camera through the detection images between adjacent frames, and connect the movements of adjacent moments in series to form the movement trajectory of the robot, thereby achieving positioning. On the other hand, according to the camera pose at each moment, the position of the three-dimensional space point corresponding to each pixel can be calculated, and the scene map can be obtained.

[0108] Since any measurement data has noise, the inter-frame pose tracking of the front-end fused with BIM information can estimate the movement of the camera from the image. However, it is necessary to reduce the impact of noise in the estimation process for back-end optimization, otherwise cumulative drift will inevitably occur, and drift will make it impossible to establish a consistent map. Therefore, the front-end estimated pose to be optimized and the initial values ​​of these data are sent to the back-end for overall optimization. Generally speaking, the optimization method is filtering or nonlinear optimization algorithm.

[0109] In this embodiment, step S23 performs joint optimization on the estimated pose to be optimized to obtain the joint optimization in the joint optimized pose, i.e., the above-mentioned back-end optimization process. In one implementation, the joint optimization is performed using a nonlinear optimization bundle adjustment method, which is also called the BA method and is a nonlinear optimization method.

[0110] In related technologies, back-end optimization mostly uses the reprojection error method, which projects the three-dimensional feature point coordinates corresponding to the pixel coordinates to the adjacent frame according to the estimated posture, and calculates the error between the pixel coordinates obtained on the adjacent frame and the pixel coordinates obtained by feature matching.

[0111] like Fig.10 As shown in the figure, it is a schematic diagram of reprojection error in the related art, where P is a three-dimensional space point, P1 and P2 are the observation matching point pairs of the three-dimensional space point in the camera's kth frame and k+1 frame respectively, P2' is the projection point of the three-dimensional space point P in the camera's k+1th frame, and e is the camera plane distance error between the camera's k+1th frame observation point P2 and the projection point P2'. The nonlinear optimization bundle adjustment method BA is used for solution.

[0112] However, the optimization of the existing nonlinear optimization bundle adjustment method BA is overly dependent on the initial pose of the camera. Due to the inaccurate initial pose, the optimization process of the nonlinear optimization bundle adjustment method BA requires multiple iterations to achieve sufficient accuracy, and the required iteration cycle is long, and good optimization results cannot be obtained. Therefore, this embodiment improves the reprojection error method, uses the fused BIM information, and pre-positions the camera positioning information through the 2D laser radar as the initial pose of the current frame of the camera, and realizes the inter-frame constraint, so that more accurate optimization results can be obtained in fewer optimization iteration cycles.

[0113] like Fig.11 As shown in FIG. 1 , it is a schematic diagram of the inter-frame constrained reprojection error based on BIM information in this embodiment. Where P is a three-dimensional space point, P1 and P2 are the observation matching point pairs of the three-dimensional space point in the camera's kth frame and k+1th frame, respectively, that is, the observation matching point pairs are obtained by feature matching, P2' is the projection point of the three-dimensional space point P in the camera's k+1th frame, e 投影 represents the camera plane distance error between the observation point P2 and the projection point P2' of the camera k+1 frame, that is, the camera projection error mentioned above, e BIM It represents the positioning difference between camera frames, that is, the error between the estimated pose to be optimized in the current frame of the camera and the initial pose of the camera obtained through BIM information.

[0114] The jointly optimized pose after joint optimization belongs to inter-frame optimization, so it is necessary to traverse several co-viewing frames of the current frame for joint optimization, which can be expressed as:

[0115]

[0116] Among them, ζ* 1 represents the joint optimization pose, which is a Lie algebra. represents the frames before the k+1th frame, for example, 20 common view frames are taken, that is, k is traversed from 1 to 20, u represents the pixel coordinates of the observation point P2 in the k+1th frame of the camera, Kexp(ζ ^ )P represents the projection of the three-dimensional space point P on the frame plane of the camera k+1 frame under the camera parameter K, and its projection value is expressed as P2',T w_k+1 represents the estimated pose of the camera in the k+1th frame to be optimized, T w_BIM1 represents the camera positioning information of the k+1th frame obtained through BIM information, that is, the initial pose, e 投影 represents the camera plane distance error between the observation point P2 and the projection point P2' of the camera's k+1th frame, that is, the camera projection error mentioned above, e BIM It represents the positioning difference between camera frames, that is, the error between the estimated pose to be optimized in the k+1th frame of the camera and the initial pose of the camera obtained through BIM information.

[0117] In the above-mentioned process of obtaining the jointly optimized pose, in the graph optimization nonlinear optimization bundle adjustment method BA, the constraint edge between the camera k-th frame and the camera k+1-th frame is added by injecting BIM information. Compared with the related technology in which the nonlinear optimization only has the edge constraints of the camera and the landmark three-dimensional space points, the optimization effect is improved and the number of optimization iterations is reduced.

[0118] In one embodiment, step S24 performs construction operations based on the joint optimized posture and the three-dimensional position information obtained by feature extraction, wherein the three-dimensional position information is a kind of semantic information, and a map is constructed based on the semantic information to identify the operated target, such as the three-dimensional position information of wall defects during wall repair, thereby expanding the application scope of the robot.

[0119] For example, neural networks are used to perform operation recognition, and the three-dimensional position information of the operation target is determined based on the operation recognition results. The robot performs construction operations based on the three-dimensional position information. Operation recognition is to identify the operation object based on the different operation types. This is because during the construction process, the construction robot not only needs to obtain its own position and the position of objects in the environment, but also needs to identify the operated object, such as the convex bumps on the wall that need to be polished, etc. It even needs to identify some common objects on the construction site, such as ladders, buckets, sandbags, etc., and locate these operation objects, and perform corresponding construction operations according to the operation objects.

[0120] Taking wall repair as an example, repairing wall defects is a major process in decoration construction. This task requires the robot to identify wall defects and determine their locations, so as to perform operations such as polishing and smoothing the defects. In this implementation, target detection is performed through the YOLO convolutional neural network to realize the recognition of wall defects, and the ORB feature corresponding to the defect is determined according to the defect recognition result to determine the three-dimensional position information of the defect in the world coordinates, and the three-dimensional position information of the operation target for repairing the defect is represented on the map, and it is registered in the visual feature map, which facilitates the robot to interact with the environment for construction operations, thereby meeting the work needs of the construction robot.

[0121] like Fig.12 The figure shows two schematic diagrams of wall defects, namely pit defects and convex pit defects, which are common wall defects. In one embodiment, a number of photos are collected at the decoration site, and a data set is manually annotated to perform model training. For example, 500 photos are collected as samples, of which 420 photos are used for training data sets and 80 photos are used for testing data sets. The model is trained using the software environment: Ubuntu+Darknet+Cuda+Cudnn to obtain appropriate model parameters.

[0122] like Fig.13As shown, it is a schematic diagram of the detection effect of the YOLO convolutional neural network of this embodiment. The upper figure is a schematic diagram of the collected wall defects, which includes two convex pit defects and one concave pit defect. The lower figure is a schematic diagram of the detection results after detection by the YOLO convolutional neural network of this embodiment. It can be seen from the figure that the three defects in the left figure are identified after detection.

[0123] The above-mentioned YOLO convolutional neural network based on deep learning establishes semantic information about defects, identifies the operated targets, and marks the operated targets into the feature map created by visual SLAM to realize an operable semantic map.

[0124] In one embodiment, it also includes performing loop detection on the joint optimized posture of the back-end to solve the problem of position estimation drifting over time. Loop detection is also called closed-loop detection. It is essentially an algorithm for detecting the similarity of observed data. It refers to the ability of the robot to recognize scenes that have been reached. If the loop detection is successful, the detection information is provided to the back-end for processing. The back-end adjusts the trajectory and map to conform to the loop detection results based on this information, optimizes the correction of accumulated errors, eliminates redundant map points, optimizes the essential map, and updates all map points. It can significantly reduce the accumulated errors, achieve global posture consistency, and obtain globally consistent trajectories and maps.

[0125] This embodiment uses BIM model information as prior information, extracts information about the size of the building structure, and converts it into a completed state grid map that can be used for robot positioning. The completed map positioning method with redundant information can be used throughout the entire engineering cycle, which simplifies the positioning process. Facing changing scenes, it can be directly positioned, and the robot's positioning efficiency in changing scenes is improved. Then, during the construction operation, in view of the problem that traditional visual SLAM cannot build maps and locate in a weak texture environment, the laser projector dot matrix is ​​set to increase the visual features of the wall for artificial projection, which solves the problem that the visual SLAM front end cannot work in a weak texture environment, and uses the YOLO convolutional neural network to detect operable targets on the wall, integrates the operable targets into the visual SLAM feature map, and constructs a semantic map containing the three-dimensional position information of the operation. At the same time, the current positioning information obtained by the BIM model information is used to perform inter-frame pose tracking of the front-end fusion BIM information and joint optimization of the back-end fusion BIM information to improve the accuracy of traditional visual SLAM.

[0126] In another embodiment of the present disclosure, a construction operation system is provided for executing the above construction operation method, such as Fig.14 As shown, it is a structural block diagram of the construction operating system of this embodiment, including:

[0127] The current positioning information acquisition unit 100 is used to obtain the current positioning information of the robot using the robot scene positioning method as described above;

[0128] Inter-frame pose tracking unit 200: used to use current positioning information to perform inter-frame pose tracking to obtain an estimated pose to be optimized;

[0129] Joint optimization unit 300: used to perform joint optimization on the estimated pose to be optimized to obtain a joint optimized pose;

[0130] Construction operation unit 400: used to perform construction operations according to the joint optimized posture and the operation position information obtained by feature extraction.

[0131] The specific details of each unit module in the above-mentioned construction operating system have been described in detail in the construction operation method corresponding to the above-mentioned embodiment, so they will not be repeated here.

[0132] In addition, an embodiment of the present invention further provides an electronic device, including:

[0133] at least one processor, and a memory communicatively coupled to the at least one processor;

[0134] The processor is used to execute the method described in Embodiment 1 by calling the computer program stored in the memory. The computer program is a program code. When the program code is run on the device, the program code is used to enable the device to execute the steps of the robot scene positioning method or the construction operation method described in the above embodiment of this specification.

[0135] In addition, the present invention also provides a computer-readable storage medium, which stores computer-executable instructions, wherein the computer-executable instructions are used to enable a computer to execute the steps of the robot scene positioning method or construction operation method described in the above embodiment part of this specification.

[0136] Without loss of generality, the computer readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storage technologies, CD-ROM, DVD or other optical storage, cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media is not limited to the above.

[0137] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of this specification are described. Other embodiments are within the scope of the attached claims. In some cases, the recorded actions or steps can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0138] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, storage medium and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0139] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A robot scene positioning method, characterized in that: include: Extract structural information from BIM data to generate a grid diagram of the building's completed status; During the construction process, the laser radar installed on the robot obtains the scanning data of the current construction scene in real time; wherein, the scanning data includes the engineering structure information, and the engineering structure information includes the load-bearing walls and partition walls. In the initial construction state, the engineering structure information only includes the load-bearing walls. As the construction progresses, when all the partition walls are built, the completion state is reached, and each construction stage grid map is a subset of the completion state grid map; The likelihood domain of each particle in the adaptive Monte Carlo algorithm is updated in real time according to the engineering structure information, and the likelihood domain of each particle is used as a weight to screen the particles; wherein the weight of the particle represents the degree of conformity between the observed map data obtained based on the particle and the actual map data, and the smaller the degree of conformity, the smaller the weight; The robot is positioned in the current construction scene according to the screening result and the completion status grid diagram to obtain the current positioning information of the robot.

2. A construction operation method, characterized in that: include: Obtaining the current positioning information of the robot using the robot scene positioning method as claimed in claim 1; Using the current positioning information to perform inter-frame pose tracking to obtain an estimated pose to be optimized; Performing joint optimization on the estimated pose to be optimized to obtain a joint optimized pose; The construction operation is performed according to the joint optimized posture and the three-dimensional position information obtained by feature extraction.

3. A construction operation method according to claim 2, characterized in that: The using the current positioning information to perform inter-frame pose tracking to obtain an estimated pose to be optimized includes: Obtaining current camera positioning information according to the transformation matrix between the robot and the camera and the current positioning information; The camera positioning information is used as the initial position and posture to obtain the camera frame positioning difference, and the camera projection error and the camera frame positioning difference are used to obtain the estimated position and posture to be optimized.

4. A construction operation method according to claim 2, characterized in that: The joint optimization is performed using a nonlinear optimization bundle adjustment method.

5. A construction operation method according to claim 3, characterized in that: The camera projection error is obtained by feature matching, including: Using a laser projector to project the environment to obtain a number of projection spots; Acquire the projection light spot by using a camera device to perform ORB feature matching; The camera projection error is obtained according to the feature matching result.

6. A construction operation method according to claim 2, characterized in that: It also includes using a neural network to perform operation recognition, determining the three-dimensional position information of the operation target according to the operation recognition result, and the robot performs construction operations according to the joint optimized posture and the three-dimensional position information.

7. A construction operating system, characterized in that: include: A current positioning information acquisition unit: used to obtain the current positioning information of the robot using the robot scene positioning method as claimed in claim 1; An inter-frame pose tracking unit: used to use the current positioning information to perform inter-frame pose tracking to obtain an estimated pose to be optimized; A joint optimization unit: used for jointly optimizing the estimated pose to be optimized to obtain a joint optimized pose; Construction operation unit: used for performing construction operations according to the joint optimized posture and the operation position information obtained by feature extraction.

8. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; The processor is configured to execute the method according to claim 1 or any one of claims 2 to 6 by calling the computer program stored in the memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to perform the method according to claim 1, or to perform the method according to any one of claims 2 to 6.

Images