A wall feature analysis system based on an RGBD depth camera

Abstract

The application discloses a wall feature analysis system based on an RGBD depth camera and belongs to the field of robot vision. The system comprises a construction area wall type confirmation module, a foreign matter identification module and a wall construction intelligent robot. The upper and lower limits of the construction area are determined through the construction area and wall type confirmation module, the construction position is accurately controlled, and the problems of fuzzy edge limit of wall construction, great construction difficulty and poor effect are solved. Through the foreign matter identification module, the problem that an operator needs to control the behavior of the wall construction intelligent robot in the wall construction process and avoid the collision and damage of the construction module with foreign matters is solved. Through the construction area and wall type confirmation module, the type of the wall to be constructed is identified, the intelligence degree of the wall construction intelligent robot is further enhanced, the wall construction intelligent robot can adopt different types of construction strategies for different types of walls, and therefore the construction efficiency and quality are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of robot vision, and particularly relates to a wall feature analysis system based on an RGBD depth camera. BACKGROUND

[0002] In the process of building decoration, for the built rough house, it is often necessary to perform plastering, whitewashing, and tiling on the wall, which not only makes the wall look beautiful, but also protects the wall to a certain extent.

[0003] Traditional wall construction work is often completed by manual labor, and the quality and speed of construction are related to the proficiency of the construction personnel. If the construction personnel operate improperly, it will lead to poor plastering quality, even rework, and affect the construction progress. In view of the above problems, many researchers have begun to develop intelligent robots for wall construction, such as wall plastering intelligent robots for wall plastering, building wall painting machines for wall painting, and automatic tiling robots for wall tiling. Although the construction of the wall in each direction is different, but in general, they are all for wall construction, and there are some common defects, mainly reflected in the following points:

[0004] 1. The existing wall construction intelligent robot has less research on the visual part, and even still relies on manual control by the operator to perform wall construction, and the intelligence of the wall construction equipment is low.

[0005] 2. The existing wall construction intelligent robot often uses horizontal or vertical operation as a unit for wall construction, but due to the lack of research on the visual part, the wall construction equipment has a fuzzy boundary for the wall construction area, and the construction under the premise of uncertainty whether it is a construction area may cause damage to the intelligent robot.

[0006] 3. When the construction wall is not a regular plane, or contains door and window hollows, junction boxes, building hanging nets and other foreign matters, most wall construction intelligent robots cannot operate normally and need to be manually taken over by the operator.

[0007] 4. The existing wall construction intelligent robot sets the construction method by the operator for different types of walls, and even takes the same construction strategy for different types of walls, lacking judgment on the type information of the wall. SUMMARY

[0008] The purpose of the present application is to provide a wall feature analysis system based on an RGBD depth camera to improve the problem of low intelligence of the existing wall construction intelligent robot when performing wall construction.

[0009] To achieve the above purpose, the present application adopts the following technical solutions:

[0010] A wall surface feature analysis system based on an RGBD depth camera, comprising: a construction area and wall surface category confirmation module, a foreign matter identification module and a wall surface construction intelligent robot;

[0011] The construction area and wall surface category confirmation module is used to confirm the wall surface construction area range and the category thereof, and send the confirmation result to the wall surface construction intelligent robot;

[0012] The foreign matter identification module is used to identify the foreign matter features of the construction area, and send the identified foreign matter features to the wall surface construction intelligent robot;

[0013] The wall surface construction intelligent robot is composed of a depth camera, a control module and a construction module, the depth camera is used to acquire wall surface image information, and send the acquired wall surface image information to the construction area and wall surface category confirmation module and the foreign matter identification module; the control module is used to receive the construction area and wall surface category confirmation result and the identified foreign matter features, and control the construction module to move according to the construction area and wall surface category confirmation result and the identified foreign matter features, so as to complete the construction operation of the construction area.

[0014] Further, the construction area range in the construction area and wall surface category confirmation module is confirmed according to the following steps:

[0015] a1, adjust the position relationship between the construction module, the depth camera and the wall surface, so as to avoid the depth camera from collecting the depth information of the ground or the ceiling;

[0016] a2, acquire wall surface image information by using the depth camera, the wall surface image information including depth image information and RGB image information;

[0017] a3, in order to improve the operation speed of the algorithm, grid sampling is performed on the depth image obtained in a2, to obtain a series of sampling points on the depth image, each sampling point containing the longitudinal and transverse coordinates under the image coordinates and the corresponding depth value information;

[0018] a4, after converting the sampling points on the depth image obtained in a3 into points in the camera coordinate system by using the intrinsic parameters of the depth camera, using the RANSAC algorithm to fit the wall surface, to obtain the wall surface plane equation;

[0019] a5, according to the preset fitting accuracy, judge whether the wall surface plane equation obtained in a4 meets the requirements, if yes, go to a6, if not, move the construction module and return to a1, until the obtained wall surface plane equation meets the preset fitting accuracy or reaches the upper limit of the fitting number; for the case of reaching the upper limit of the fitting number, it is indicated that the overall situation of the wall surface is poor, and the upper and lower limits of the wall surface construction area need to be manually set;

[0020] a6, grab the bottom depth image, use the wall plane equation obtained by a5 to detect the depth value points in the depth camera field of view range; if the same row in the depth image does not satisfy the depth value points of the plane equation obtained by a5 greater than or equal to the preset number, it is considered that the bottom end of the wall surface is in the depth camera field of view range, that is, the bottom end position of the wall surface can be determined;

[0021] a7, grab the top depth image, and determine the top end position of the construction area by the same method as determining the bottom end position of the wall surface;

[0022] a8, determine the construction area according to the bottom end position of the construction area determined by a6 and the top end position of the construction area determined by a7.

[0023] Further, the wall surface category in the construction area and wall surface category identification module is identified by a deep learning network model.

[0024] Further, the wall surface foreign matter identification module identifies foreign matter features according to the following steps:

[0025] b1, according to the brightness information and foreign matter contour information in the obtained RGB image, judge whether the construction area exists building hanging net, if yes, directly send the control module; if not, enter b2 to judge the foreign matter category;

[0026] b2, the foreign matter category judgment process is:

[0027] b2.1, for the depth image of the construction area obtained by the depth camera, the morphological operation is used to remove the connected domain area with length or height less than the preset value, so as to remove the texture interference in the construction area;

[0028] b2.2, on the basis of b2.1, the edge extraction algorithm is used to extract the contour features of the foreign matter;

[0029] b2.3, judge whether there is a connected domain exceeding the preset threshold value in the contour features extracted by b2.2, if there is a connected domain exceeding the preset threshold value, it means that the contour features are foreign matter contour, then according to the foreign matter contour information, the foreign matter category is identified, and the vertical distance between the construction module and the foreign matter is calculated, and they are sent to the control module together; if there is no connected domain exceeding the preset threshold value, it means that there is no foreign matter, so no operation is needed.

[0030] Further, the process of judging whether the wall surface foreign matter identification module exists in the construction area is as follows:

[0031] b1.1, histogram equalization and median filtering are performed on the depth image of the construction area provided by the depth camera to improve the calculation accuracy;

[0032] b1.2, performing convolution operation on the image obtained in b1.1 using Sobel operator to obtain a convolution image with pixel values in the interval [0, 255];

[0033] b1.3, converting the convolution image obtained in b1.2 into a binary image using Ostu threshold method, and removing the regions of connected domains with length or height less than a preset value using morphological operation to remove texture interference in the wall surface background of the construction area;

[0034] b1.4, using Hough transform to determine whether the number of abnormal points in the horizontal and vertical directions reaches a preset threshold value, if the threshold value is reached, it means that the current field of view range contains building hanging nets, and relevant instructions need to be sent to the control module and the current judgment is ended; if the preset threshold value is not reached, it means that the current field of view range does not contain building hanging nets, and no special operation is needed, entering b1.5;

[0035] b1.5, traversing each connected domain of the binary image obtained in b1.3 to obtain the ratio of the perimeter to the area of the connected domain, and judging whether the current connected domain is a wall crack according to the size of the ratio, if there is a wall crack, relevant instructions are sent to the control module; if not, no operation is performed.

[0036] Further, the distance calculation process between the construction module and the foreign matter in the construction area upper and lower limit and wall foreign matter identification module in the construction area and wall category confirmation module includes the following steps:

[0037] c1, if the called module is the construction area and wall category confirmation module, enter c1.1, if the called module is the wall foreign matter identification module, enter c1.2;

[0038] c1.1, match the wall plane equation with the depth value in the depth image, if the depth value in the depth image is less than the depth value at the corresponding position in the wall plane equation and is lower than the set threshold value, the point is determined as a depth value abnormal point; scan the depth image row by row, if the number of depth value abnormal points in a row is greater than or equal to 50, it means that the wall construction area upper and lower limit appears in the current depth camera field of view range, a line segment is determined in the pixel coordinate system according to the current row coordinate, representing the wall construction area lower limit.

[0039] c1.2, according to the foreign matter contour information, determine the foreign matter circumscribed rectangle, and determine four line segments therefrom;

[0040] c2, according to the relationship between the camera intrinsic parameter, the depth information value in the image, the set relationship between the origin of the world coordinate system and the origin of the camera coordinate system, the straight line or line segment in the pixel coordinate system is converted into the world coordinate system;

[0041] C3, if the called module is the construction area and wall category confirmation module, the distance between the straight line and the construction module is calculated under the world coordinate system, and the calculation result is sent to the control module; if the called module is the wall foreign matter identification module, the distance between the four line segments of the circumscribed rectangle and between each line segment and the construction module is calculated under the world coordinate system, and the calculation result is sent to the control module to control the movement of the construction module.

[0042] Further, in order to ensure that the depth image captured by the depth camera contains sufficient wall information, the wall construction intelligent robot is provided with two depth cameras, one of which is installed in the middle of the side of the construction module facing the top end, and the other is installed in the middle of the side of the construction module facing the bottom end.

[0043] After the above technical scheme is adopted, the present application has the following advantages:

[0044] 1. The present application determines the top end and bottom end positions of the wall through the construction area and wall category confirmation module, effectively solving the problems of poor edge area construction effect and easy damage to the construction module caused by the ambiguous boundaries of the bottom end and top end of the construction area in traditional wall intelligent construction robots. Because the bottom end and top end of the construction area are clear, the developed construction strategy is more perfect, and even manual operation by an operator is not required, thereby reducing labor costs and improving work efficiency.

[0045] 2. The present application determines the type of wall through the construction area and wall category confirmation module, so that the wall intelligent construction can adopt different construction strategies for different types of walls, thereby improving the work quality and intelligence of the wall intelligent construction robot.

[0046] 3. The foreign matter identification module of the present application can effectively identify foreign matters such as doors and windows, hollowed out, junction boxes, load-bearing columns, and building hanging nets, and by calculating the distance between the construction module and the foreign matter, the foreign matter is avoided, thereby solving the problem of the need for an operator to control the behavior of the wall construction intelligent robot during wall construction, and avoiding the collision and damage of the construction module. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is the wall construction intelligent robot model mentioned in the present embodiment;

[0048] Figure 2 is the overall architecture diagram of the wall feature analysis system based on the RGBD depth camera of the present embodiment;

[0049] Figure 3 is the algorithm flowchart of the construction area and wall category confirmation module of the present embodiment;

[0050] Figure 4is a camera coordinate system and world coordinate system relationship diagram of the depth camera below the construction module in the embodiment;

[0051] Figure 5 is a flow chart of the depth feature foreign matter identification algorithm in the foreign matter identification module of the embodiment

[0052] Figure 6 is a convolutional neural network model diagram used in the embodiment;

[0053] Figure 7 is a flow chart of the texture feature foreign matter identification algorithm in the foreign matter identification module of the embodiment. DETAILED DESCRIPTION

[0054] The technical solutions of the present application will be described in detail below in combination with the drawings and embodiments:

[0055] As Figure 2 shown, the wall feature analysis system based on an RGBD depth camera provided by the embodiment comprises a construction area wall and category confirmation module, a foreign matter identification module, and a wall construction intelligent robot.

[0056] The construction area and wall category confirmation module is used to confirm the construction area range and the category of the wall, and send the confirmation result to the wall construction intelligent robot.

[0057] The foreign matter identification module is used to identify the foreign matter features of the construction area, and send the identified foreign matter features to the wall construction intelligent robot.

[0058] The wall construction intelligent robot comprises a depth camera, a controller, and a construction module. The depth camera is used to acquire wall depth images and RGB images, and send them to the construction area wall and category confirmation module and the foreign matter identification module, respectively. The depth camera is installed at the middle position of the construction module, and there are two of them, one on each side of the middle part of the construction module facing the ceiling and the ground, respectively. When in use, the center of the camera field of view forms an angle of 60° and -60° with the horizontal plane, respectively.

[0059] The control module is used to receive the construction area and wall category confirmation result and the identified foreign matter features, and control the movement of the construction module according to the construction area and wall category confirmation result and the identified foreign matter features to complete the construction work of the construction area. The wall construction intelligent robots disclosed in the prior art are mostly applicable to the embodiment. Figure 1 For the wall construction intelligent robot model used in the embodiment, Figure 1 the guide rail shown is a liftable component of the wall construction intelligent robot, which is a vertical moving track controlling the up-and-down movement path of the construction module.

[0060] As Figure 3As shown, the construction area wall surface and category confirmation module confirms the construction area range according to the following steps:

[0061] a1, adjust the position relationship between the construction module, the depth camera and the wall surface, to avoid the depth camera from collecting the depth information of the ground or the ceiling;

[0062] a2, use the depth camera to obtain the depth image of the wall surface, and the wall surface image information includes the depth image information and the RGB image information;

[0063] a3, in order to improve the operation speed of the algorithm, the depth image obtained in a2 is grid sampled to obtain a series of sampling points on the depth image, and each sampling point includes the longitudinal and transverse coordinates under the image coordinates and the corresponding depth value information;

[0064] a4, using the intrinsic parameter of the depth camera, the sampling points on the depth image obtained in a3 are converted into points in the camera coordinate system, and then the RANSAC algorithm is used for wall surface fitting to obtain the wall surface plane equation;

[0065] a5, according to the preset fitting accuracy, it is judged whether the wall surface plane equation obtained in a4 meets the requirements, if it meets the requirements, it enters a6, if it does not meet the requirements, it moves the construction module and returns to a1, until the wall surface plane equation obtained meets the preset fitting accuracy or reaches the upper limit of the fitting number; for the case of reaching the upper limit of the fitting number, it is indicated that the overall situation of the wall surface is poor, and the upper and lower limits of the wall surface construction area need to be manually set;

[0066] a6, grab the bottom depth image, and use the wall surface plane equation obtained in a5 to detect the depth value points in the depth camera field of view; if the depth value points on the same horizontal line in the depth image do not meet the plane equation obtained in a5 and are greater than or equal to the preset number, it is considered that the bottom end of the wall surface is in the depth camera field of view, that is, the bottom end position can be determined.

[0067] a7, grab the top depth image, and determine the top end position of the construction area by using the same method as determining the bottom end position of the wall surface;

[0068] a8, according to the bottom end position of the construction area determined in a6 and the top end position of the construction area determined in a7, the construction area is determined.

[0069] In this embodiment, since the construction module is in a nearly parallel relationship with the wall surface, the depth information of the wall surface will not change much during the up and down movement of the construction module. When converted to the depth camera image, it means that the depth of the fixed pixel point will not change much during the up and down movement of the construction module. Only when the ceiling or the floor appears in the field of view of the depth camera, or there is a significant foreign matter at the bottom or top of the wall, the depth value of the pixel point will change significantly. Therefore, the number of points where the depth value of the pixel point changes significantly is recorded to determine the position of the bottom or top of the construction area.

[0070] In actual life, the types of walls that need to be constructed include concrete, masonry, aerated concrete block, etc. Different construction methods and standards are required for different types of construction walls, so it is necessary to identify different types of walls. There are many ways to identify the type of wall based on deep learning. This embodiment takes the VGG16 convolutional neural network model as an example (here, the deep learning method is emphasized, not the specific model structure of VGG16) to identify the type of wall. The model structure is shown in Figure 6 The first convolutional layer has a total of two convolutional kernels, and the convolutional kernel size is (3x3x64); the second layer has two convolutional kernels, and the convolutional kernel size is (3x3x128); the third layer has three convolutional kernels, and the convolutional kernel size is (3x3x256); the fourth layer has three convolutional kernels, and the convolutional kernel size is (3x3x512); the fourth layer has three convolutional kernels, and the convolutional kernel size is (3x3x512); the number of neurons of the fully connected layer is 4096, 2048, and 3, respectively. Max-pooling layer is used to connect between each two convolutional layers, and relu function is used as the activation function for each convolutional kernel. The input is the depth image of the wall to be constructed obtained by the depth camera, and the output is the type of the wall.

[0071] The wall foreign matter that can be identified by this embodiment includes door and window hollowing, junction box, exposed reinforcement, building hanging net, wall crack, etc. Different identification methods are used for different foreign matters. Since the building hanging net is generally composed of fine iron wires, forming a horizontal and vertical net structure, the diameter of the iron wire is generally 0.5 mm, which is smaller than the depth measurement resolution of the depth camera, and is often consistent with the background depth value on the depth image. In the wall construction process, lighting equipment can be installed on the construction module. The light reflection ability of the building hanging net and the wall is different, so before construction, the brightness information and contour information in the collected image should be used to judge whether the construction area has a building hanging net. The specific process is shown in Figure 7 .

[0072] b1.1, histogram equalization and median filtering are performed on the depth image of the construction area provided by the depth camera to improve the calculation accuracy;

[0073] b1.2, the Sobel operator is used to perform convolution operation on the image obtained in b1.11, and a convolution image with pixel values in the interval [0, 255] is obtained;

[0074] b1.3, after the convolution image is converted into a binary image by using the Ostu threshold method, morphological operation is used to remove the regions of connected domains with a length or height less than a preset value, so as to remove the texture interference in the wall surface background of the construction area;

[0075] b1.4, the Hough transform is used to detect whether the number of straight lines in the horizontal and vertical directions reaches a preset threshold value. If the threshold value is reached, it means that the current field of view range contains building hanging nets, and the hanging net information needs to be sent to the control module and the current judgment is ended. If the preset threshold value is not reached, it means that the current field of view range does not contain building hanging nets, and no special operation is needed, and b1.5 is entered;

[0076] b1.5, each connected domain of the binary image obtained in b1.3 is traversed, and the ratio of the perimeter to the area of the connected domain is calculated. According to the size of the ratio, it is judged whether the current connected domain is a wall crack. If there is a wall crack, it is sent to the control module and the current judgment is ended. If not, no operation is performed.

[0077] For foreign matters other than building hanging nets and cracks, the recognition method is as shown in the method: Figure 5

[0078] b2.1, morphological operation is used to remove the regions of connected domains with a length or height less than a preset value in the depth image of the construction area obtained by the depth camera, so as to remove the texture interference in the construction area;

[0079] b2.2, on the basis of b2.1, an edge extraction algorithm is used to extract the contour features of the foreign matter;

[0080] b2.3, it is judged whether there is a connected domain exceeding a preset threshold value in the contour features extracted in b2.2. If there is a connected domain exceeding the preset threshold value, it means that the contour features are foreign matter contours, and the foreign matter type is identified according to the foreign matter contour information and sent to the control module. If there is no connected domain exceeding the preset threshold value, it means that there is no foreign matter, and no operation is needed.

[0081] The distance calculation process between the construction module and the foreign matter in the construction area and wall surface category confirmation module and the wall surface foreign matter recognition module includes the following steps:

[0082] c1, if the called module is the construction area and wall surface category confirmation module, c1.1 is entered, and if the called module is the wall surface foreign matter recognition module, c1.2 is entered;

[0083] ​c1.1 Match the wall plane equation with the depth value in the depth image. If the depth value in the depth image is less than the depth value at the corresponding position in the wall plane equation and is below a set threshold, then that point is identified as a depth value anomaly. Scan the depth image line by line. If the number of depth value anomalies in a certain line is greater than or equal to 50, it means that the upper and lower boundaries of the wall construction area are within the current depth camera's field of view. Based on the current line coordinates, determine a straight line in the pixel coordinate system to represent the lower limit of the wall construction area.

[0084] c1.2. Based on the object's outline information, determine the bounding rectangle of the object, thereby determining the four line segments. Specifically, let the coordinate set P of all vertices of the object's outline be as shown in equation (1):

[0085] P = {(u1,v1),(u2,v2),…,(u...} n ,v n )} (1)

[0086] The four line segments of the circumscribed rectangle are shown in equation (2):

[0087]

[0088] In the formula, l l l r l u l d These represent the left, right, top, and bottom lines of the circumscribed rectangle, respectively.

[0089] c2. Since the obtained lines are in pixel coordinates (unit: pixels), they need to be converted to world coordinates (unit: pixels) for subsequent calculations of distances between them. The specific process is as follows:

[0090] c2.1 Convert the pixel coordinate system to the image coordinate system. The conversion process is shown in equation (3):

[0091]

[0092] In the formula, x and y are the coordinates in the pixel coordinate system; u0 and v0 are the pixel coordinates of the center point of the image in the pixel coordinate system; dx and dy are determined by the camera parameters and represent the size of the photosensitive element.

[0093] c2.2, Converting the image coordinate system to the camera coordinate system. The relationship between the camera coordinate system and the image coordinate system is as follows: Figure 4 As shown. Camera coordinate system center O c With respect to the center O of the image coordinate system i The line connecting the two coordinate systems is the Z-axis. The x-axis and y-axis of the two coordinate systems are parallel, and O... c With O iThe distance of the connection is the focal length f. The conversion relationship between the two coordinate systems is:

[0094]

[0095] where x c , y c are the coordinates in the camera coordinate system, and f is the focal length.

[0096] c2.3, image coordinate system to world coordinate system. When converting the image coordinate system to the world coordinate system, the relationship between the image coordinate system and the world coordinate system needs to be determined. In this embodiment, the middle of the construction module is selected as the origin of the world coordinate system, and the direction of the wall surface construction intelligent robot is selected as the Z w axis of the world coordinate system, and the construction surface of the wall surface construction intelligent robot is selected as the X w O w Z w plane, wherein the X w axis direction is parallel to the construction module. According to the position of the camera origin in the world coordinate system, the translation operation between the two coordinate systems is determined as shown in equation (5):

[0097]

[0098] where t x , t y , and t z are the translation amounts of the corresponding axes in the professional process, and the translation matrix T is determined as shown in equation (6):

[0099] T = [t x , t y , t z ] (6)

[0100] In the process of converting the camera coordinate system to the world coordinate system, the rotation operation between the coordinate axes also needs to be considered. The relationship between the selected world coordinate system and the two camera coordinate systems is shown in Figure 4 , and the conversion relationship of the corresponding coordinates of the two coordinate systems is shown in equation:

[0101]

[0102] where θ is the angle between the view center and the horizontal direction. Therefore, the rotation matrix R is determined as shown in equation (8):

[0103]

[0104] Therefore, the transformation formula of the camera coordinate system and the world coordinate system is shown in equation (9):

[0105]

[0106] The transformation formula of the final pixel coordinate system into the world coordinate system is shown as formula (10):

[0107]

[0108] c3, if the called module is the construction area and wall type confirmation module, the distance between the straight line and the construction module is calculated in the world coordinate system, and the calculation result is sent to the control module; if the called module is the wall foreign matter identification module, the distance between the four straight line segments of the circumscribed rectangle and the distance between each straight line segment and the construction module are calculated in the world coordinate system, and the calculation result is sent to the control module to control the construction module to avoid the foreign matter.

[0109] It can be seen from the above that the wall feature analysis system based on the RGBD depth camera provided by the application can accurately control the construction position by identifying the upper and lower limits of the wall construction area, and solve the problems of fuzzy edge limit of wall construction, large construction difficulty and poor effect; through the foreign matter identification module, the problem that the operator needs to control the behavior of the wall construction intelligent robot during the wall construction process is solved, and the problem that the construction module is damaged by collision with foreign matter is avoided; through the construction area and wall type confirmation module, the type of the wall to be constructed is identified, and the intelligence of the wall construction intelligent robot is further enhanced, so that the wall construction intelligent robot can adopt different types of construction strategies for different types of walls, thereby improving the construction efficiency and quality.

Claims

1. A wall feature analysis system based on an RGBD depth camera, comprising: The construction area and wall type confirmation module, the foreign object recognition module, and the intelligent wall construction robot are characterized by: The construction area and wall category confirmation module is used to confirm the scope and category of the wall construction area and send the confirmation result to the intelligent wall construction robot; the scope of the construction area in the construction area and wall category confirmation module is confirmed according to the following steps: a1. Adjust the positional relationship between the construction module, the depth camera, and the wall to prevent the depth camera from collecting depth information from the ground or ceiling; a2. Use a depth camera to acquire wall image information, which includes depth image information and RGB image information; a3. In order to improve the computation speed of the algorithm, the depth image obtained in a2 is subjected to grid sampling to obtain a series of sampling points on the depth image. Each sampling point contains the vertical and horizontal coordinates under the image coordinates and the corresponding depth value information. a4. Using the intrinsic parameters of the depth camera, after converting the sampling points on the depth image obtained in a3 into points in the camera coordinate system, the RANSAC algorithm is used to fit the wall surface to obtain the wall plane equation. a5. Based on the preset fitting accuracy, determine whether the wall plane equation obtained in a4 meets the requirements. If it does, proceed to a6. If it does not, move to the construction module and return to a1. Continue until the obtained wall plane equation meets the preset fitting accuracy or reaches the upper limit of the fitting times. If the upper limit of the fitting times is reached, it means that the overall condition of the wall is poor and it is necessary to manually set the upper and lower limits of the wall construction area. a6. Capture the bottom depth image and use the wall plane equation obtained in a5 to detect the depth points within the field of view of the depth camera. If the number of depth points in the same row of the depth image that do not satisfy the plane equation obtained in a5 is greater than or equal to the preset number, then the bottom of the wall is considered to be within the field of view of the depth camera, and the position of the bottom of the wall can be determined in this way. a7. Capture the top depth image and determine the top position of the construction area using the same method as determining the bottom position of the wall. a8. Determine the construction area based on the bottom position of the construction area determined by a6 and the top position of the construction area determined by a7. The foreign object identification module is used to identify foreign object features in the construction area and send the identified foreign object features to the intelligent wall construction robot. The intelligent robot for wall construction consists of a depth camera, a control module, and a construction module. The depth camera is used to acquire wall image information and sends the acquired wall image information to the construction area and wall type confirmation module and the foreign object recognition module, respectively. The control module is used to receive the construction area and wall type confirmation results and the identified foreign object characteristics, and control the construction module to move according to the construction area and wall type confirmation results and the identified foreign object characteristics to complete the construction work in the construction area.

2. The wall feature analysis system based on an RGBD depth camera according to claim 1, characterized in that: The wall category in the construction area and wall category identification module will be confirmed using a deep learning network model.

3. The wall feature analysis system based on an RGBD depth camera according to claim 1, characterized in that: The wall foreign object identification module identifies foreign object characteristics according to the following steps: b1. Based on the brightness information and foreign object outline information in the acquired RGB image, determine whether there is a building mesh in the construction area. If yes, send the message directly to the control module; otherwise, proceed to b2 to determine the type of foreign object. b2. The process for determining the type of foreign object is as follows: b2.

1. For the depth image of the construction area acquired by the depth camera, morphological operations are used to remove the regions of connected components whose length or height is less than the preset value, so as to remove texture interference in the construction area. b2.

2. Based on b2.1, an edge extraction algorithm is used to extract the contour features of the foreign object; b2.3 Determine whether there are connected components in the contour features extracted in b2.2 that exceed a preset threshold. If there are connected components that exceed the preset threshold, it means that the contour feature is a foreign object contour. Based on the foreign object contour information, identify the type of foreign object, calculate the vertical distance between the construction module and the foreign object, and send them to the control module together. If there are no connected components that exceed the preset threshold, it means that there is no foreign object, and no operation is required.

4. The wall feature analysis system based on an RGBD depth camera according to claim 3, characterized in that: The process by which the wall foreign object recognition module determines whether there is construction mesh in the construction area is as follows: b1.1 Perform histogram equalization and median filtering on the depth images of the construction area provided by the depth camera to improve the calculation accuracy; b1.

2. The Sobel operator is used to perform a convolution operation on the image obtained in b1.1 to obtain a convolution image with pixel values ​​in the range of [0,255]. b1.3 After converting the convolutional image obtained in b1.2 into a binary image using the Ostu thresholding method, morphological operations are then used to remove regions of connected components whose length or height is less than a preset value, in order to remove texture interference in the background of the wall surface in the construction area. b1.4 Use Hough transform to determine whether the number of straight lines detected in the horizontal and vertical directions has reached the preset threshold. If the threshold is reached, it means that there is a building net in the current field of view. Relevant instructions need to be sent to the control module and the judgment ends. If the preset threshold is not reached, it means that there is no building net in the current field of view. No special operation is required. Proceed to b1.

5. b1.

5. Traverse each connected component of the binary image obtained from b1.3, calculate the ratio of the perimeter to the area of ​​the connected component, and determine whether the current connected component is a wall crack based on the value of the ratio. If a wall crack exists, send relevant instructions to the control module; otherwise, do not perform any operation.

5. A wall feature analysis system based on an RGBD depth camera according to claim 1, characterized in that: The calculation process for the upper and lower boundaries of the construction area in the construction area and wall category confirmation module, and the calculation of the distance between the construction module and the foreign object in the wall foreign object identification module, includes the following steps: c1. If the module called is the construction area and wall type confirmation module, then proceed to c1.1; if the module called is the wall foreign object recognition module, then proceed to c1.

2. c1.1 Based on step a6 in weight 1, the depth value points that do not satisfy the plane equation obtained in step a5 are judged as follows: Match the wall plane equation obtained in a5 with the depth value in the depth image. For any pixel in the depth image, if the depth value is less than the calculated value at the corresponding position in the wall plane equation, and both are lower than the set threshold, then the point is determined as a depth value anomaly point. Scan the depth image line by line. If the number of depth value anomalies in a certain line is greater than or equal to 50, it means that the upper and lower limits of the wall construction area are within the current depth camera's field of view. Based on the current row coordinates, determine a straight line in the pixel coordinate system to represent the lower limit of the wall construction area. c1.

2. Based on the outline information of the foreign object, determine the bounding rectangle of the foreign object, and thus determine the four line segments; c2. Based on the camera intrinsic parameters, the depth information values ​​in the image, and the relationship between the preset world coordinate system origin and the camera coordinate system origin, convert the straight line or line segment in the pixel coordinate system to the world coordinate system. c3. If the module called is the construction area and wall type confirmation module, calculate the distance between the straight line and the construction module in the world coordinate system and send the calculation result to the control module; if the module called is the wall foreign object recognition module, calculate the distance between the four line segments of the circumscribed rectangle and the distance between each line segment and the construction module in the world coordinate system and send the calculation result to the control module to control the movement of the construction module.

6. The wall feature analysis system based on an RGBD depth camera according to claim 1, characterized in that: To ensure that the depth images captured by the depth cameras contain sufficient information about the wall surface, two depth cameras are installed on the intelligent wall construction robot. One depth camera is installed in the middle of the side of the construction module facing the top, and the other depth camera is installed in the middle of the side of the construction module facing the bottom.

Images