Intelligent brick laying method and device based on depth camera and brick laying robot

Abstract

The application discloses an intelligent brick paving method and device based on a depth camera, and a brick paving robot, and the method comprises the following steps: acquiring a brick paving area image collected by the depth camera, analyzing the brick paving area image to obtain an image analysis result, and determining whether the brick paving area image comprises a brick paving marker line according to the image analysis result to obtain a determination result; acquiring a brick paving information determination mode matched with the determination result and determining center point information corresponding to a to-be-paved brick based on the brick paving information determination mode; and controlling a mechanical hand to perform a brick paving operation according to the center point information corresponding to the to-be-paved brick. It can be seen that the application can realize intelligent brick paving based on the brick paving robot provided with the mechanical hand and the depth camera, which is favorable for improving the brick paving efficiency and the brick paving precision. In addition, the brick paving information determination mode can be matched according to the brick paving marker line recognition result, which is favorable for improving the determination efficiency and the accuracy of the brick paving information, and further favorable for further improving the brick paving efficiency and the brick paving precision.

Description

Technical Field

[0001] This invention relates to the field of intelligent brick-laying robot technology, and in particular to an intelligent brick-laying method and device based on a depth camera, as well as a brick-laying robot. Background Technology

[0002] Garden paving with solid bricks is an outdoor project that requires laying bricks in designated areas according to a pre-defined plan, and it demands a certain level of precision and efficiency. Currently, most garden paving with solid bricks in the industry is done manually. Manual paving suffers from low efficiency, and because the quality is greatly affected by the paving worker's experience and skills, it also suffers from low precision.

[0003] Therefore, it is particularly important to provide an intelligent tiling method to improve tiling efficiency and accuracy. Summary of the Invention

[0004] This invention provides an intelligent paving method and device based on a depth camera, as well as a paving robot. The paving robot can achieve intelligent and automated paving based on a depth camera and a robotic arm installed on it, which is beneficial to improve paving efficiency and accuracy.

[0005] The first aspect of this invention discloses an intelligent brick-laying method based on a depth camera. The method is applied to a brick-laying robot, which is equipped with a robotic arm and a depth camera. The depth camera is mounted on the end effector of the robotic arm and moves with the robotic arm. The method includes:

[0006] The image of the paved area captured by the depth camera is acquired, the image of the paved area is analyzed to obtain the image analysis result, and the image analysis result is used to determine whether the paved area image includes paving marking lines, and the determination result is obtained.

[0007] Obtain a paving information determination method that matches the judgment result, and determine the center point information corresponding to the brick to be laid based on the paving information determination method;

[0008] The robotic arm is controlled to perform a matching brick-laying operation based on the center point information corresponding to the bricks to be laid.

[0009] As an optional implementation, in the first aspect of the present invention, determining the center point information corresponding to the brick to be laid based on the brick laying information determination method includes:

[0010] Based on the brick laying information determination method, the origin and three-dimensional coordinate axes of the center point coordinate system to be established are determined, and the center point coordinate system corresponding to the brick to be laid is created based on the determined origin and three-dimensional coordinate axes, as the center point information corresponding to the brick to be laid.

[0011] And, controlling the robotic arm to perform a matching paving operation based on the center point information corresponding to the brick to be paved includes:

[0012] Based on the center point information of the brick to be laid, the robot arm is generated with corresponding brick-laying control parameters, and the robot arm is controlled to perform the brick-laying operation corresponding to the brick to be laid based on the brick-laying control parameters.

[0013] As an optional implementation, in the first aspect of the present invention, when the determination result indicates that the acquired image of the paved area includes the paved marking line, the paved information determination method is a line-based paving method based on the paved marking line.

[0014] When the judgment result indicates that the collected image of the paved area does not include the paved marking line, the paved information is determined by the paving method based on the paved bricks.

[0015] As an optional implementation, in the first aspect of the present invention, the step of determining the origin and three-dimensional coordinate axes of the center point coordinate system to be established based on the brick laying information determination method, and creating a center point coordinate system corresponding to the brick to be laid based on the determined origin and three-dimensional coordinate axes, as the center point information corresponding to the brick to be laid, includes:

[0016] When the brick laying information is determined by the brick-viewing and brick-laying method, all laid bricks within the field of view of the depth camera are identified based on the image of the brick laying area.

[0017] For each laid brick, determine the coordinate information of the four corner points of the target rectangle corresponding to the laid brick;

[0018] Based on the coordinate information corresponding to all the brick corner points, the four brick corner points of the target rectangle corresponding to all the laid bricks are fitted into a target plane, and all the brick corner points are vertically projected onto the target plane according to the normal vector to obtain all the projected brick corner points of all the brick corner points on the target plane.

[0019] For each of the laid bricks, a corresponding grid is fitted according to the four projected brick corner points in the horizontal and vertical directions, and the position number of the bricks to be laid is calculated according to the paving plan.

[0020] Based on the location number and all the fitted grids, determine two target horizontal lines and two target vertical lines that match the location number. Based on the two target horizontal lines and the two target vertical lines, determine the four target brick corner points of the brick to be laid. Based on the four target brick corner points, determine the center point of the brick to be laid. Create a center point coordinate system for the brick to be laid with the center point of the brick to be laid as the origin, the horizontal axis as the X-axis, and the vertical axis as the Y-axis, as the center point information of the brick to be laid.

[0021] As an optional implementation, in the first aspect of the present invention, determining the coordinate information corresponding to the four corner points of the target rectangle corresponding to each laid brick includes:

[0022] For each paved brick, all target point clouds falling into the target rectangle corresponding to the paved brick are identified from all point clouds acquired by the depth camera. All target point clouds falling into the target rectangle corresponding to the paved brick are fitted into a plane equation, and the four corner points of the target rectangle corresponding to the paved brick are mapped from the pixel coordinate system to the color camera coordinate system to obtain the mapping result. Based on the plane equation fitted by all target point clouds falling into the target rectangle corresponding to the paved brick and the mapping result, the coordinate information corresponding to the four corner points of the target rectangle corresponding to the paved brick is determined.

[0023] As an optional implementation, in the first aspect of the invention, identifying all target point clouds falling into the target rectangle corresponding to each laid brick from all point clouds acquired by the depth camera includes:

[0024] Based on the determined transformation matrix, all point clouds acquired by the depth camera are transformed into the color camera coordinate system to obtain the coordinate points corresponding to each point cloud in the color camera coordinate system;

[0025] Based on a predetermined transformation formula, the coordinates of all the point clouds in the color camera coordinate system are transformed to the pixel coordinate system to obtain the coordinates of all the point clouds in the pixel coordinate system.

[0026] For each laid brick, identify all target coordinate points located within the target rectangle corresponding to the laid brick from the coordinate points corresponding to all point clouds in the pixel coordinate system, and use these as all target point clouds falling into the target rectangle corresponding to the laid brick.

[0027] As an optional implementation, in the first aspect of the invention, for each of the laid bricks, the target rectangle corresponding to the laid brick is determined based on the brick corner points determined for the laid brick, and the brick corner points determined for the laid brick are determined in the following manner:

[0028] An initial rectangle is determined based on the location of the laid bricks. Sub-rectangles are generated on each side of the initial rectangle according to a predetermined distance value and a predetermined width value. The predetermined distance value is the distance between each sub-rectangle and the vertex of the initial rectangle, and the predetermined width value is the width of each sub-rectangle.

[0029] Within each sub-rectangle, identify several boundary points on the boundary of the laid bricks, and fit a straight line corresponding to each boundary based on the identified boundary points on each boundary.

[0030] Extend the straight lines corresponding to all the boundaries of the laid bricks to obtain the intersection points of the extended straight lines corresponding to all the boundaries, and determine the brick corner points of the target rectangle corresponding to the laid bricks.

[0031] As an optional implementation, in the first aspect of the present invention, the step of determining the origin and three-dimensional coordinate axes of the center point coordinate system to be established based on the brick laying information determination method, and creating a center point coordinate system corresponding to the brick to be laid based on the determined origin and three-dimensional coordinate axes, as the center point information corresponding to the brick to be laid, includes:

[0032] When the paving information is determined using the line-based paving method, the target marking area on the paving marking line is identified;

[0033] Determine the center point of the area corresponding to the cross intersection area of ​​the target marking area, and map the center point of the area to the color camera coordinate system to obtain the first mapped coordinate point of the center point of the area in the color camera coordinate system.

[0034] In the lateral direction of the cross-shaped area, a target reference point that does not coincide with the center point of the area is selected, and the target reference point is mapped to the color camera coordinate system to obtain the second mapped coordinate point of the target reference point in the color camera coordinate system.

[0035] An X-axis is established based on the first and second mapped coordinate points, a Z-axis is determined based on the plane normal vector, and a Y-axis is determined based on the determined X-axis and Z-axis. A center point coordinate system corresponding to the brick to be laid is created with the first mapped coordinate point as the origin and the determined X-axis, Y-axis, and Z-axis, which serves as the center point information corresponding to the brick to be laid.

[0036] A second aspect of this invention discloses an intelligent brick-laying device based on a depth camera. The device is applied to a brick-laying robot, which is equipped with a robotic arm and a depth camera. The depth camera is mounted on the end of the robotic arm and moves with the robotic arm. The device includes:

[0037] The image analysis module is used to acquire images of the paved area captured by the depth camera, analyze the images of the paved area, and obtain image analysis results.

[0038] The image judgment module is used to determine whether the paving area image includes paving marking lines based on the image analysis results, and to obtain the judgment result;

[0039] The determination module is used to obtain a paving information determination method that matches the judgment result, and determine the center point information corresponding to the brick to be laid based on the paving information determination method;

[0040] The brick-laying control module is used to control the robotic arm to perform matching brick-laying operations based on the center point information corresponding to the brick to be laid.

[0041] As an optional implementation, in a second aspect of the present invention, the specific method by which the determining module determines the center point information corresponding to the brick to be laid based on the brick laying information determining method includes:

[0042] Based on the brick laying information determination method, the origin and three-dimensional coordinate axes of the center point coordinate system to be established are determined, and the center point coordinate system corresponding to the brick to be laid is created based on the determined origin and three-dimensional coordinate axes, as the center point information corresponding to the brick to be laid.

[0043] Furthermore, the specific method by which the paving control module controls the robotic arm to perform matching paving operations based on the center point information corresponding to the bricks to be paved includes:

[0044] Based on the center point information of the brick to be laid, the robot arm is generated with corresponding brick-laying control parameters, and the robot arm is controlled to perform the brick-laying operation corresponding to the brick to be laid based on the brick-laying control parameters.

[0045] As an optional implementation, in a second aspect of the present invention, when the determination result indicates that the acquired image of the paved area includes the paved marking line, the paved information is determined by a line-based paving method based on the paved marking line.

[0046] When the judgment result indicates that the collected image of the paved area does not include the paved marking line, the paved information is determined by the paving method based on the paved bricks.

[0047] As an optional implementation, in a second aspect of the present invention, the determining module determines the origin and three-dimensional coordinate axes of the center point coordinate system to be established based on the brick laying information determination method, and creates a center point coordinate system corresponding to the brick to be laid based on the determined origin and three-dimensional coordinate axes. Specific methods for providing the center point information corresponding to the brick to be laid include:

[0048] When the brick laying information is determined by the brick-viewing and brick-laying method, all laid bricks within the field of view of the depth camera are identified based on the image of the brick laying area.

[0049] For each laid brick, determine the coordinate information of the four corner points of the target rectangle corresponding to the laid brick;

[0050] Based on the coordinate information corresponding to all the brick corner points, the four brick corner points of the target rectangle corresponding to all the laid bricks are fitted into a target plane, and all the brick corner points are vertically projected onto the target plane according to the normal vector to obtain all the projected brick corner points of all the brick corner points on the target plane.

[0051] For each of the laid bricks, a corresponding grid is fitted according to the four projected brick corner points in the horizontal and vertical directions, and the position number of the bricks to be laid is calculated according to the paving plan.

[0052] Based on the location number and all the fitted grids, determine two target horizontal lines and two target vertical lines that match the location number. Based on the two target horizontal lines and the two target vertical lines, determine the four target brick corner points of the brick to be laid. Based on the four target brick corner points, determine the center point of the brick to be laid. Create a center point coordinate system for the brick to be laid with the center point of the brick to be laid as the origin, the horizontal axis as the X-axis, and the vertical axis as the Y-axis, as the center point information of the brick to be laid.

[0053] As an optional implementation, in the second aspect of the present invention, for each of the laid bricks, the specific method by which the determining module determines the coordinate information corresponding to the four corner points of the target rectangle corresponding to the laid brick includes:

[0054] From all point clouds acquired by the depth camera, identify all target point clouds that fall into the target rectangle corresponding to the laid bricks. Fit all target point clouds that fall into the target rectangle corresponding to the laid bricks into a plane equation. Map the four corner points of the target rectangle corresponding to the laid bricks from the pixel coordinate system to the color camera coordinate system to obtain the mapping result. Determine the coordinate information corresponding to the four corner points of the target rectangle corresponding to the laid bricks based on the plane equation fitted by all target point clouds that fall into the target rectangle corresponding to the laid bricks and the mapping result.

[0055] As an optional implementation, in a second aspect of the invention, for each of the laid bricks, the specific method by which the determining module identifies all target point clouds falling into the target rectangle corresponding to the laid brick from all point clouds acquired by the depth camera includes:

[0056] Based on the determined transformation matrix, all point clouds acquired by the depth camera are transformed into the color camera coordinate system to obtain the coordinate points corresponding to each point cloud in the color camera coordinate system;

[0057] Based on a predetermined transformation formula, the coordinates of all the point clouds in the color camera coordinate system are transformed to the pixel coordinate system to obtain the coordinates of all the point clouds in the pixel coordinate system.

[0058] For each laid brick, identify all target coordinate points located within the target rectangle corresponding to the laid brick from the coordinate points corresponding to all point clouds in the pixel coordinate system, and use these as all target point clouds falling into the target rectangle corresponding to the laid brick.

[0059] As an optional implementation, in a second aspect of the invention, for each of the laid bricks, the target rectangle corresponding to the laid brick is determined based on the brick corner points determined for the laid brick, and the brick corner points determined for the laid brick are determined in the following manner:

[0060] An initial rectangle is determined based on the location of the laid bricks. Sub-rectangles are generated on each side of the initial rectangle according to a predetermined distance value and a predetermined width value. The predetermined distance value is the distance between each sub-rectangle and the vertex of the initial rectangle, and the predetermined width value is the width of each sub-rectangle.

[0061] Within each sub-rectangle, identify several boundary points on the boundary of the laid bricks, and fit a straight line corresponding to each boundary based on the identified boundary points on each boundary.

[0062] Extend the straight lines corresponding to all the boundaries of the laid bricks to obtain the intersection points of the extended straight lines corresponding to all the boundaries, and determine the brick corner points of the target rectangle corresponding to the laid bricks.

[0063] As an optional implementation, in a second aspect of the present invention, the determining module determines the origin and three-dimensional coordinate axes of the center point coordinate system to be established based on the brick laying information determination method, and creates a center point coordinate system corresponding to the brick to be laid based on the determined origin and three-dimensional coordinate axes. Specific methods for providing the center point information corresponding to the brick to be laid include:

[0064] When the paving information is determined using the line-based paving method, the target marking area on the paving marking line is identified;

[0065] Determine the center point of the area corresponding to the cross intersection area of ​​the target marking area, and map the center point of the area to the color camera coordinate system to obtain the first mapped coordinate point of the center point of the area in the color camera coordinate system.

[0066] In the lateral direction of the cross-shaped area, a target reference point that does not coincide with the center point of the area is selected, and the target reference point is mapped to the color camera coordinate system to obtain the second mapped coordinate point of the target reference point in the color camera coordinate system.

[0067] An X-axis is established based on the first and second mapped coordinate points, a Z-axis is determined based on the plane normal vector, and a Y-axis is determined based on the determined X-axis and Z-axis. A center point coordinate system corresponding to the brick to be laid is created with the first mapped coordinate point as the origin and the determined X-axis, Y-axis, and Z-axis, which serves as the center point information corresponding to the brick to be laid.

[0068] A third aspect of this invention discloses another intelligent tiling device based on a depth camera. The device is applied to a tiling robot, which is equipped with a robotic arm and a depth camera. The depth camera is mounted on the end of the robotic arm and moves with the robotic arm. The device includes:

[0069] Memory containing executable program code;

[0070] A processor coupled to the memory;

[0071] The processor calls the executable program code stored in the memory to execute some or all of the steps in any of the intelligent paving methods based on depth cameras disclosed in the first aspect of the present invention.

[0072] The fourth aspect of the present invention discloses a paving robot, wherein the paving robot is equipped with a robotic arm and a depth camera, the depth camera being mounted at the end of the robotic arm and moving with the robotic arm;

[0073] The paving robot is used to perform some or all of the steps in any of the intelligent paving methods based on depth cameras disclosed in the first aspect of the present invention.

[0074] The fifth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute some or all of the steps in any of the intelligent paving methods based on depth cameras disclosed in the first aspect of the present invention.

[0075] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0076] In this embodiment of the invention, an image of the paving area captured by a depth camera is acquired. The image is analyzed to obtain an image analysis result, and based on this result, it is determined whether the paving area image includes paving marking lines. A paving information determination method matching the determination result is acquired, and the center point information of the brick to be laid is determined based on this method. A robotic arm is then controlled to perform the paving operation based on the center point information of the brick to be laid. Therefore, this invention enables intelligent paving using a paving robot equipped with a robotic arm and a depth camera, which improves paving efficiency and accuracy. Furthermore, it can match the paving information determination method based on the paving marking line recognition result, further improving the efficiency and accuracy of paving information determination, and thus further enhancing paving efficiency and accuracy. Attached Figure Description

[0077] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0078] Figure 1 This is a flowchart illustrating an intelligent brick-laying method based on a depth camera, as disclosed in an embodiment of the present invention.

[0079] Figure 2This is a flowchart illustrating an intelligent brick-laying method based on brick-reading layout disclosed in an embodiment of the present invention.

[0080] Figure 3 This is a flowchart illustrating an intelligent brick-laying method based on line-based brick-laying disclosed in an embodiment of the present invention;

[0081] Figure 4 This is a schematic diagram of the intelligent brick-laying device based on a depth camera disclosed in an embodiment of the present invention;

[0082] Figure 5 This is a schematic diagram of another intelligent brick-laying device based on a depth camera disclosed in an embodiment of the present invention;

[0083] Figure 6 This is a schematic diagram showing the location numbering of the laid bricks as disclosed in an embodiment of the present invention;

[0084] Figure 7 This is a schematic diagram illustrating the principle of determining the corner points of the bricks corresponding to the target rectangular frame of the laid bricks, as disclosed in an embodiment of the present invention.

[0085] Figure 8 This is a schematic diagram illustrating the principle of determining the center point of the brick to be laid, as disclosed in an embodiment of the present invention.

[0086] Figure 9 This is a schematic diagram of the marked area identified under the line-reading brick-laying method disclosed in the embodiments of the present invention;

[0087] Figure 10 This is a schematic diagram illustrating the principle of identifying the center point of a cross-shaped area in a line-based paving method, as disclosed in an embodiment of the present invention. Detailed Implementation

[0088] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0089] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0090] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0091] This invention discloses an intelligent tiling method and device based on a depth camera, as well as a tiling robot. The tiling robot, equipped with a robotic arm and a depth camera, enables intelligent tiling, improving both tiling efficiency and accuracy. Furthermore, it can match tiling information determination methods based on tiling mark line recognition results, further enhancing the efficiency and accuracy of tiling information determination, thus further improving tiling efficiency and accuracy. Detailed descriptions follow.

[0092] Example 1

[0093] Please see Figure 1 , Figure 1 This is a flowchart illustrating an intelligent paving method based on a depth camera, as disclosed in an embodiment of the present invention. Wherein, Figure 1 The method shown can be applied to an intelligent paving control device, which controls a paving robot equipped with a robotic arm and a depth camera to perform paving operations. Specifically, the depth camera can be installed at the end of the robotic arm and move with it. Optionally, the intelligent paving control device can be integrated into the paving robot or exist independently of it, such as being located on a cloud control platform. This embodiment of the invention does not impose limitations. Figure 1 As shown, the intelligent paving method based on a depth camera can include the following operations:

[0094] 101. Acquire images of the paved area captured by the depth camera, analyze the images of the paved area, and obtain the image analysis results.

[0095] Optionally, the depth camera installed at the end of the robotic arm of the paving robot can be a Kinect-DK model with 12 megapixels. In actual operation, the depth camera is installed at the end of the robotic arm and moves with the robotic arm, taking pictures with its front facing down. This method of acquiring images of the paving area by using a depth camera installed at the end of the robotic arm and moving with the robotic arm improves the flexibility of image acquisition in the paving area.

[0096] 102. Based on the image analysis results, determine whether the paving area image includes paving marking lines, and obtain the judgment result.

[0097] The paving marking line can also be understood as the paving positioning line. It should be noted that in practical applications, if the depth camera is close to the paving area, its field of view is small. This may result in situations where "the paving area has paving marking lines, but the acquired image of the paving area does not include the paving marking lines" or "the paving area has paving marking lines, but multiple paving marking lines are acquired." To reduce or avoid this situation, during actual acquisition, the distance between the depth camera and the paving area can be adjusted to perform multiple acquisitions to obtain a suitable image of the paving area. This improves the accuracy and reliability of the acquired paving area image, thereby enhancing the accuracy and reliability of the aforementioned judgment results.

[0098] 103. Obtain the paving information determination method that matches the above judgment result, and determine the center point information corresponding to the brick to be laid based on the paving information determination method.

[0099] Optionally, when the above judgment result indicates that the acquired paved area image includes paving mark lines, the paving information is determined by laying paving tiles based on the paving mark lines; when the above judgment result indicates that the acquired paved area image does not include paving mark lines, the paving information is determined by laying paving tiles based on the already laid tiles. Most preferably, when the above judgment result indicates that the acquired paved area image includes paving mark lines and includes only one paving mark line, the paving information is determined by laying paving tiles based on the paving mark lines.

[0100] It should be noted that, in practical applications, the method described in the embodiments of the present invention adopts a combination of brick-laying by observation and brick-laying by line to realize the laying or tiling of bricks to be laid in the tiling area. That is, when laying bricks initially, brick-laying by observation is performed first, and then it is determined whether there are tiling marking lines in the collected tiling area, and then brick-laying by line is performed.

[0101] 104. Control the robotic arm to perform matching brick-laying operations based on the center point information of the bricks to be laid.

[0102] It is evident that the method described in the embodiments of the present invention can achieve intelligent tiling based on a tiling robot equipped with a robotic arm and a depth camera, which is beneficial to improving tiling efficiency and tiling accuracy. In addition, it can also match the tiling information determination method according to the tiling mark line recognition result, which is beneficial to improving the efficiency and accuracy of tiling information determination, and thus further improving tiling efficiency and tiling accuracy.

[0103] In an optional embodiment, the method of determining the center point information corresponding to the brick to be laid based on the brick laying information described above may include:

[0104] Based on the above method of determining paving information, the origin and three-dimensional coordinate axes of the center point coordinate system to be established are determined, and the center point coordinate system corresponding to the brick to be laid is created based on the determined origin and three-dimensional coordinate axes, which serves as the center point information corresponding to the brick to be laid.

[0105] It should be noted that the coordinate system of the center point of the brick to be laid can also be understood as the position of the brick to be laid.

[0106] As can be seen, this optional embodiment can determine the origin and coordinate axes for creating the coordinate system based on the paving information determination method that matches the paving mark line recognition result, which improves the accuracy of the determined origin and coordinate axes, and helps to improve the accuracy of the center point information corresponding to the paving bricks to be laid, thereby improving the control accuracy and control efficiency of the paving robot to perform paving control operations.

[0107] In yet another optional embodiment, the above-mentioned control of the robotic arm to perform a matching paving operation based on the center point information corresponding to the brick to be paved may include:

[0108] Based on the center point information of the bricks to be laid, the robot arm is generated with corresponding brick-laying control parameters, and the robot arm is controlled to perform the brick-laying operation corresponding to the bricks to be laid based on the brick-laying control parameters.

[0109] As can be seen, this optional embodiment can generate the corresponding brick-laying control parameters for the robotic arm based on the center point information of the brick to be laid, and then control the robotic arm to perform the corresponding brick-laying operation based on the brick-laying control parameters, which is beneficial to improving the brick-laying efficiency and the accuracy of brick-laying control.

[0110] Example 2

[0111] Please see Figure 2 , Figure 2 This is a flowchart illustrating an intelligent brick-laying method based on a brick-reading pattern, as disclosed in an embodiment of the present invention. Figure 2The method shown can be applied to an intelligent brick-laying control device, which controls a brick-laying robot equipped with a robotic arm and a depth camera to perform brick-laying operations. Specifically, the depth camera can be installed at the end of the robotic arm and move with it. Optionally, the intelligent brick-laying control device can be integrated into the brick-laying robot or exist independently of it, such as being located on a cloud control platform. This embodiment of the invention does not impose limitations. It should be noted that... Figure 2 The method shown is applicable to scenarios where there are no paving marker lines in the paved area image acquired by a depth camera. For example... Figure 2 As shown, the method may include the following operations:

[0112] 201. Acquire images of the paved area captured by the depth camera, analyze the images of the paved area, and obtain the image analysis results.

[0113] 202. Based on the image analysis results, determine whether the paving area image includes paving marking lines, and obtain the judgment result.

[0114] For a detailed description of steps 201-202, please refer to the detailed description of steps 101-102 in Embodiment 1. This embodiment of the invention will not repeat the description.

[0115] 203. When the above judgment result indicates that the paving area image does not contain paving mark lines, identify all paved bricks within the field of view of the depth camera based on the paving area image.

[0116] Where the above judgment result indicates that the paving area image does not contain paving mark lines, the corresponding paving information determination method is the "look-and-lay" method, that is: paving the bricks to be laid is completed based on the already laid bricks. In this embodiment of the invention, after identifying all the laid bricks within the field of view of the depth camera, a unique brick identifier can be set for each laid brick according to a pre-set naming or encoding rule. Optionally, the unique brick identifier corresponding to the laid brick can specifically be the position identifier (such as position number) of the laid brick in the image coordinate system corresponding to the paving area image. This position identifier can be composed of the row identifier of the row where the laid brick is located and the column identifier of the column where the laid brick is located. For example, the position numbers of all the laid bricks within the field of view of the depth camera identified based on the paving area image can specifically be as follows: Figure 6 As shown, Figure 6 This is a schematic diagram showing the location numbering of the laid bricks as disclosed in an embodiment of the present invention.

[0117] 204. For each laid brick, determine the coordinate information of the four corner points of the target rectangle corresponding to the laid brick.

[0118] In an optional embodiment, the process of determining the coordinate information corresponding to the four corner points of the target rectangle for each laid brick can include:

[0119] For each laid brick, all target point clouds falling into the target rectangle corresponding to the laid brick are identified from all point clouds acquired by the depth camera. All target point clouds falling into the target rectangle corresponding to the laid brick are fitted into a plane equation, and the four corner points of the target rectangle corresponding to the laid brick are mapped from the pixel coordinate system to the color camera coordinate system to obtain the mapping result. Based on the plane equation fitted by all target point clouds falling into the target rectangle corresponding to the laid brick and the mapping result, the coordinate information corresponding to the four corner points of the target rectangle corresponding to the laid brick is determined.

[0120] In this optional embodiment, further optionally, for each laid brick, identifying all target point clouds falling into the target rectangle corresponding to that laid brick from all point clouds acquired by the depth camera may include:

[0121] Based on the determined transformation matrix, all point clouds acquired by the depth camera are transformed into the color camera coordinate system to obtain the coordinate points of each point cloud in the color camera coordinate system;

[0122] Based on the predetermined transformation formula, the coordinates of all point clouds in the color camera coordinate system are transformed to the pixel coordinate system, thus obtaining the coordinates of all point clouds in the pixel coordinate system.

[0123] For each laid brick, identify all target coordinate points located within the target rectangle corresponding to the laid brick from the coordinate points of all point clouds in the pixel coordinate system, and use these as all target point clouds that fall into the target rectangle corresponding to the laid brick.

[0124] In this optional embodiment, the transformation formula (1) used to transform point cloud coordinates to the color camera coordinate system can be:

[0125] P color = c H d P depth ……(1)

[0126] in, c H d P represents the position and orientation of the depth camera in the color camera coordinate system. depth For depth point cloud information (i.e., the coordinate information of the point cloud under the depth camera), P color This refers to the coordinate information of the point cloud in the color camera coordinate system.

[0127] Furthermore, in this optional embodiment, the transformation formulas used to transform the coordinates of all point clouds corresponding to the coordinates in the color camera coordinate system to the pixel coordinate system include formulas (2), (3), and (4), which are as follows:

[0128]

[0129]

[0130]

[0131] The relevant parameters involved in formulas (2), (3), and (4) above are explained as follows: Point cloud coordinate information P in the color camera coordinate system color Let (x, y, z) correspond to the point cloud's coordinates in the pixel coordinate system as (u, v), where k1, k2, k3, k4, k5, and k6 are radial distortion coefficients, p1 and p2 are tangential distortion coefficients, and r 2 =x' 2 +y' 2 Among them, the radial distortion coefficient and the tangential distortion coefficient are internal parameters of the depth camera.

[0132] Alternatively, for each laid brick, the target rectangle corresponding to that laid brick is determined based on the corner points of the bricks identified for that laid brick, and the corner points of the bricks identified for that laid brick are determined in the following way:

[0133] An initial rectangle is determined based on the location of the laid bricks. Sub-rectangles are generated on each side of the initial rectangle according to the predetermined distance and width values. The predetermined distance value is the distance between each sub-rectangle and the vertex of the initial rectangle, and the predetermined width value is the width of each sub-rectangle.

[0134] Within each sub-rectangle, identify several boundary points on the boundary of the laid bricks, and fit a straight line corresponding to each boundary based on the identified boundary points on each boundary.

[0135] Extend the straight lines corresponding to all boundaries of the laid bricks to obtain the intersection points of the extended straight lines of all boundaries. Determine the brick corner points of the target rectangle corresponding to the laid bricks as the brick corner points.

[0136] The principle for determining the corner point of the target rectangle corresponding to any laid brick can be as follows: Figure 7 As shown, Figure 7This is a schematic diagram illustrating the principle of determining the corner points of the target rectangle corresponding to the laid bricks, as disclosed in an embodiment of the present invention. Specifically, the initial rectangle A'B'C'D' can be identified using yolov4-tiny. After identifying the initial rectangle A'B'C'D', a sub-rectangle is generated on each side of the initial rectangle A'B'C'D'. The preset distance value of each sub-rectangle from each point is L1, and the preset width value of each sub-rectangle is L2. The four sub-rectangles are then rotated as follows: Figure 7 The clockwise direction is shown. Then, the caliper algorithm is used to find several points on the boundary of the laid bricks. Then, the several points on each boundary are fitted into a straight line. The intersection of the four straight lines are the four corner points of the laid bricks in the pixel coordinate system.

[0137] As can be seen, the embodiments of the present invention can generate a target rectangle based on the intersection of the extensions of straight lines fitted by several points on the boundary after the initial rectangle is identified by the deep learning algorithm, which is beneficial to improving the accuracy of the generated rectangle.

[0138] Alternatively, for any paved brick, the plane equation for fitting all target point clouds falling into the target rectangle corresponding to the paved brick can be: ax + by + cz = 0. The formulas used to map the four corner points of the target rectangle corresponding to the paved brick from the pixel coordinate system to the color camera coordinate system include formulas (5), (6), and (7), which are as follows:

[0139]

[0140]

[0141]

[0142] The relevant parameters involved in the above formulas (5), (6) and (7) are explained as follows: the coordinate information of the corner point of the target rectangle corresponding to the laid bricks in the pixel coordinate system is (u, v), the coordinate information of the corner point of the target rectangle corresponding to the laid bricks mapped from the pixel coordinate system to the color camera coordinate system is (x, y, z), k1, k2, k3, k4, k5, k6 are radial distortion coefficients, p1 and p2 are tangential distortion coefficients, and r 2 =x' 2 +y' 2 Among them, the radial distortion coefficient and the tangential distortion coefficient are internal parameters of the depth camera.

[0143] 205. Based on the coordinate information of all brick corner points, fit the four brick corner points of the target rectangle corresponding to all laid bricks to the target plane, and project all brick corner points vertically onto the target plane according to the normal vector to obtain all projected brick corner points on the target plane.

[0144] 206. For each laid brick, fit the corresponding grid according to the four corner points of the projected brick in the horizontal and vertical directions, and calculate the position number of the brick to be laid according to the paving plan.

[0145] 207. Based on the above location numbers and all fitted grids, determine two target horizontal lines and two target vertical lines that match the above location numbers.

[0146] 208. Based on the two target horizontal lines and the two target vertical lines, determine the four target brick corner points of the brick to be laid, and determine the center point of the brick to be laid based on the four target brick corner points.

[0147] In this embodiment of the invention, the center point of the brick to be laid can specifically be the intersection of the line connecting one pair of opposite corner points of the four target bricks and the line connecting the other pair of opposite corner points. Specifically, the principle for determining the center point of the brick to be laid can be as follows: Figure 8 As shown, Figure 8 This is a schematic diagram illustrating the principle of determining the center point corresponding to the brick to be laid, as disclosed in an embodiment of the present invention. Specifically, the center point is first determined based on the position number of the area corresponding to the brick to be laid and all the grids fitted from the already laid bricks. Figure 8 The two target horizontal lines and two target vertical lines shown in the diagram, and the intersection points A1, B1, C1 and D1 between the two target horizontal lines and the two target vertical lines are the four target brick corner points to be laid. The intersection point E of the line connecting A1 and D1 and the line connecting B1 and C1 is the center point of the brick to be laid.

[0148] 209. Create a coordinate system for the center point of the brick to be laid, with the center point of the brick to be laid as the origin, the horizontal axis as the X-axis, and the vertical axis as the Y-axis. This system serves as the center point information for the brick to be laid.

[0149] Among them, with Figure 7 Taking the target brick corner point and center point as an example, a coordinate system is created with center point E as the origin, the direction from A1 to B1 as the X-axis, and the direction from A1 to C1 as the Y-axis. This system serves as the center point information for the brick to be laid. The center point information can also be understood as the brick's orientation. camera H brick .

[0150] 210. Based on the center point information of the bricks to be laid, generate the brick-laying control parameters corresponding to the robot arm, and control the robot arm to perform the brick-laying operation corresponding to the bricks to be laid based on the brick-laying control parameters.

[0151] Among them, based on the above-mentioned paving posture camera H brick The paving control parameters can be generated by referring to formula (8), and formula (8) is as follows:

[0152] base H target = base H tool · tool H camera · camera H brick · brick H target ......(8);

[0153] in, camera H brick The posture is ready for paving. camera H brick The transformation matrix from the bricks to be laid to the depth camera. tool H camera This is the transformation matrix from depth camera to robotic arm. base H tool This is the transformation matrix from the robotic arm to the end effector.

[0154] In this embodiment of the invention, the robot arm is controlled to move based on the paving control parameters corresponding to the robot arm, so that the depth camera stays above the bricks to be paved to take pictures. Then, the robot arm is controlled to perform the paving operation based on the picture results and the paving control parameters.

[0155] As can be seen, the method described in the embodiments of the present invention can determine the coordinate system of the center point of the brick to be laid based on the relevant information of the already laid bricks, and realize the intelligent and accurate determination of the posture of the brick to be laid based on the brick-reading method, which is conducive to achieving efficient and highly accurate automated brick laying based on the brick-laying robot.

[0156] Example 3

[0157] Please see Figure 3 , Figure 3 This is a flowchart illustrating an intelligent tile laying method based on line-based tile laying, as disclosed in an embodiment of the present invention. Figure 3The method shown can be applied to an intelligent brick-laying control device, which controls a brick-laying robot equipped with a robotic arm and a depth camera to perform brick-laying operations. Specifically, the depth camera can be installed at the end of the robotic arm and move with it. Optionally, the intelligent brick-laying control device can be integrated into the brick-laying robot or exist independently of it, such as being located on a cloud control platform. This embodiment of the invention does not impose limitations. It should be noted that... Figure 3 The method shown is applicable to scenarios where paving marking lines exist in paved area images acquired using a depth camera. For example... Figure 3 As shown, the method may include the following operations:

[0158] 301. Acquire images of the paved area captured by the depth camera, analyze the images of the paved area, and obtain the image analysis results.

[0159] 302. Based on the image analysis results, determine whether the paving area image includes paving marking lines, and obtain the judgment result.

[0160] For a detailed description of steps 301-302, please refer to the detailed description of steps 101-102 in Embodiment 1. This embodiment of the invention will not repeat the description.

[0161] 303. When the above judgment result indicates that the paving area image contains paving mark lines, identify the target mark area on the paving mark lines.

[0162] In this embodiment of the invention, the target marking area on the paving marking line can be identified through color enhancement or deep learning algorithms (such as SSD). For example, the identified target marking area can be as follows: Figure 9 As shown, Figure 9 This is a schematic diagram of the marked area identified under the line-reading paving method disclosed in the embodiments of the present invention.

[0163] 304. Determine the center point of the area corresponding to the cross-shaped area of ​​the target marking area, and map the center point of the area to the color camera coordinate system to obtain the first mapped coordinate point of the center point of the area in the color camera coordinate system.

[0164] In this embodiment of the invention, after determining the center point (u1, v1) of the region, a mask rectangular region can be set. Referring to the method in Embodiment 2, the average value of the Z-axis coordinates of all point cloud coordinates falling within the mask rectangular region is determined as the Z-axis coordinate of the marker point. Then, the center point (u1, v1) of the region is mapped from the pixel coordinate system to the color camera coordinate system to obtain the first mapped coordinate point (x1, y1, z1) of the center point (u1, v1) in the color camera coordinate system. Then, based on the first mapped coordinate point (x1, y1, z1) and the plane equation of the ground to be paved, ax + by + cz + d = 0, the plane equation passing through the first mapped coordinate point (x1, y1, z1) is obtained as a(x-x1) + b(y-y1) + c(z-z1) = 0, where a and b are both 0. That is, the plane equation a(x-x1) + b(y-y1) + c(z-z1) = 0 is the plane normal vector (0, 0, 1).

[0165] In this embodiment of the invention, the formula used to map the center point of the region from the pixel coordinate system to the color camera coordinate system can refer to formulas (5), (6), and (7) used in Embodiment 2 to map the corner point of the target rectangle from the pixel coordinate system to the color camera coordinate system. This embodiment of the invention will not repeat these formulas. It should be noted that when using formulas (5), (6), and (7) to map the center point of the region from the pixel coordinate system to the color camera coordinate system, (u, v) specifically refers to the coordinate information (u1, v1) of the center point of the region in the pixel coordinate system, and (x, y, z) refers to the coordinate information (x1, y1, z1) of the center point of the region in the color camera coordinate system after mapping from the pixel coordinate system to the color camera coordinate system.

[0166] For example, the principle or process for determining the center point of a region can be as follows: Figure 10 As shown, Figure 10 This is a schematic diagram illustrating the principle of identifying the center point of a cross-shaped area in a line-based paving method, as disclosed in an embodiment of the present invention.

[0167] 305. Select a target reference point that does not coincide with the center point of the area in the horizontal direction of the cross intersection area, and map the target reference point to the color camera coordinate system to obtain the second mapped coordinate point of the target reference point in the color camera coordinate system.

[0168] Among them, selecting a target reference point that does not coincide with the center point of the cross-shaped area in the horizontal direction can be done by selecting a target reference point that does not coincide with the center point of the area in the horizontal direction and to the right of the center point of the area. For example, selecting a target reference point that does not coincide with the center point of the area in the horizontal direction and to the right of the center point of the area, and whose distance from the center point of the area is 100 pixels. The coordinate information (u2, v2) of the target reference point in the pixel coordinate system can be represented by the following formula (9):

[0169]

[0170] In this embodiment of the invention, the formula used to map the target reference point from the pixel coordinate system to the color camera coordinate system can refer to formulas (5), (6), and (7) used in Embodiment 2 to map the corner points of the target rectangle from the pixel coordinate system to the color camera coordinate system. This embodiment of the invention will not repeat these formulas. It should be noted that when using the above formulas (5), (6), and (7) to map the target reference point from the pixel coordinate system to the color camera coordinate system, (u, v) specifically refers to the coordinate information (u2, v2) of the target reference point in the pixel coordinate system, and (x, y, z) refers to the coordinate information (x2, y2, z2) of the target reference point in the color camera coordinate system after mapping from the pixel coordinate system to the color camera coordinate system.

[0171] 306. Establish the X-axis based on the first and second mapped coordinate points, determine the Z-axis based on the plane normal vector, and determine the Y-axis based on the determined X-axis and Z-axis. Use the first mapped coordinate point as the origin and the determined X-axis, Y-axis, and Z-axis to create a coordinate system for the center point of the brick to be laid, which serves as the center point information for the brick to be laid.

[0172] 307. Based on the center point information of the bricks to be laid, generate the corresponding brick-laying control parameters for the robotic arm, and control the robotic arm to perform the brick-laying operation corresponding to the bricks to be laid based on the brick-laying control parameters.

[0173] Wherein, the coordinates of the bricks to be laid in the color camera coordinate system are (x0, y0, z0), which can be obtained by formula (10), specifically:

[0174]

[0175] In formula (10), s is the average side length of the brick, and w is the standard line width.

[0176] Furthermore, in the line-based paving method, a series of coordinate transformations can be performed using formula (11) to obtain the actual working position (i.e., paving control parameters), and formula (11) is as follows:

[0177] base H target = base H tool · tool H camera · camera H marker · marker H target ......(11)

[0178] in, marker H target The transformation matrix from the target point to the marker point. camera H mar Ker is the transformation matrix from the marker point to the depth camera. tool H camera This is the transformation matrix from depth camera to robotic arm. base H tool This is the transformation matrix from the robotic arm to the end effector.

[0179] As can be seen, the method described in the embodiments of the present invention can determine the coordinate system of the center point of the brick to be laid based on the line-reading brick-laying method. The line-reading brick-laying method realizes the intelligent and accurate determination of the posture of the brick to be laid, which is conducive to achieving efficient and highly accurate automated brick laying based on the brick-laying robot.

[0180] Example 4

[0181] Please see Figure 4 , Figure 4 This is a schematic diagram of the intelligent paving device based on a depth camera disclosed in an embodiment of the present invention. Figure 4 The described device is used to control a tiling robot equipped with a robotic arm and a depth camera to perform tiling operations. Specifically, the depth camera can be mounted on the end effector of the robotic arm and move with it. Optionally, this device can be integrated into the tiling robot or exist independently of it, such as being located on a cloud control platform. This embodiment of the invention does not impose limitations. Figure 4 As shown, the intelligent paving device based on a depth camera may include:

[0182] The image analysis module 401 is used to acquire images of the paved area captured by the depth camera, analyze the images of the paved area, and obtain image analysis results.

[0183] Image judgment module 402 is used to determine whether the paving area image includes paving mark lines based on the image analysis results, and obtain the judgment result;

[0184] The determination module 403 is used to obtain the paving information determination method that matches the judgment result, and determine the center point information corresponding to the brick to be laid based on the paving information determination method;

[0185] The brick-laying control module 404 is used to control the robotic arm to perform matching brick-laying operations based on the center point information of the bricks to be laid.

[0186] As can be seen, the device described in the embodiments of the present invention can realize intelligent tiling based on a tiling robot equipped with a robotic arm and a depth camera, which is beneficial to improving tiling efficiency and tiling accuracy. In addition, it can also match the tiling information determination method according to the tiling mark line recognition result, which is beneficial to improving the efficiency and accuracy of tiling information determination, and thus further improving tiling efficiency and tiling accuracy.

[0187] In an optional embodiment, the specific method by which the determining module 403 determines the center point information corresponding to the brick to be laid based on the brick laying information determination method includes:

[0188] The origin and three-dimensional coordinate axes of the center point coordinate system to be established are determined based on the brick laying information determination method. The center point coordinate system corresponding to the brick to be laid is created based on the determined origin and three-dimensional coordinate axes, which serves as the center point information corresponding to the brick to be laid.

[0189] As can be seen, this optional embodiment can also determine the origin and coordinate axes for creating the coordinate system based on the paving information determination method that matches the paving mark line recognition result, which improves the accuracy of the determined origin and coordinate axes, and helps to improve the accuracy of the center point information corresponding to the paving bricks to be laid, thereby improving the control accuracy and control efficiency of the paving robot to perform paving control operations.

[0190] In another optional embodiment, the brick-laying control module 404 controls the robotic arm to perform a matching brick-laying operation based on the center point information corresponding to the brick to be laid, specifically in the following ways:

[0191] Based on the center point information of the bricks to be laid, the robot arm is generated with corresponding brick-laying control parameters, and the robot arm is controlled to perform the brick-laying operation corresponding to the bricks to be laid based on the brick-laying control parameters.

[0192] Optionally, when the judgment result indicates that the acquired paving area image includes paving mark lines, the paving information is determined by the line-based paving method based on the paving mark lines; when the judgment result indicates that the acquired paving area image does not include paving mark lines, the paving information is determined by the line-based paving method based on the already laid bricks.

[0193] As can be seen, this optional embodiment can generate the corresponding brick-laying control parameters for the robotic arm based on the center point information of the brick to be laid, and then control the robotic arm to perform the corresponding brick-laying operation based on the brick-laying control parameters, which is beneficial to improving the brick-laying efficiency and the accuracy of brick-laying control.

[0194] In another optional embodiment, the determining module 403 determines the origin and three-dimensional coordinate axes of the center point coordinate system to be established based on the brick laying information determination method, and creates the center point coordinate system corresponding to the brick to be laid based on the determined origin and three-dimensional coordinate axes. The specific methods for providing the center point information corresponding to the brick to be laid include:

[0195] When the bricklaying information is determined by the bricklaying method, all laid bricks within the field of view of the depth camera are identified based on the image of the bricklaying area.

[0196] For each laid brick, determine the coordinate information of the four corner points of the target rectangle corresponding to the laid brick;

[0197] Based on the coordinate information corresponding to all brick corner points, fit the four brick corner points of the target rectangle corresponding to all laid bricks to the target plane, and project all brick corner points vertically onto the target plane according to the normal vector to obtain all projected brick corner points on the target plane.

[0198] For each laid brick, the corresponding grid is fitted according to the four projected brick corner points in the horizontal and vertical directions, and the position number of the brick to be laid is calculated according to the paving plan.

[0199] Based on the location number and all fitted meshes, determine two target horizontal lines and two target vertical lines that match the location number. Based on the two target horizontal lines and two target vertical lines, determine the four target brick corner points of the brick to be laid. Based on the four target brick corner points, determine the center point of the brick to be laid. Create a coordinate system for the center point of the brick to be laid, with the center point of the brick to be laid as the origin, the horizontal axis as the X-axis, and the vertical axis as the Y-axis, as the center point information of the brick to be laid.

[0200] In this optional embodiment, further optionally, for each laid brick, the determining module 403 determines the coordinate information corresponding to the four corner points of the target rectangle corresponding to the laid brick in the following specific ways:

[0201] From all point clouds acquired by the depth camera, identify all target point clouds that fall within the target rectangle corresponding to the laid bricks. Fit a plane equation to all target point clouds falling within the target rectangle corresponding to the laid bricks, and map the four corner points of the target rectangle corresponding to the laid bricks from the pixel coordinate system to the color camera coordinate system to obtain the mapping result. Based on the plane equation fitted by all target point clouds falling within the target rectangle corresponding to the laid bricks and the mapping result, determine the coordinate information corresponding to the four corner points of the target rectangle corresponding to the laid bricks.

[0202] Further optionally, for each laid brick, the specific method by which the determining module 403 identifies all target point clouds falling into the target rectangle corresponding to that laid brick from all point clouds acquired by the depth camera includes:

[0203] Based on the determined transformation matrix, all point clouds acquired by the depth camera are transformed into the color camera coordinate system to obtain the coordinate points of each point cloud in the color camera coordinate system;

[0204] Based on the predetermined transformation formula, the coordinates of all point clouds in the color camera coordinate system are transformed to the pixel coordinate system, thus obtaining the coordinates of all point clouds in the pixel coordinate system.

[0205] For each laid brick, identify all target coordinate points located within the target rectangle corresponding to the laid brick from the coordinate points of all point clouds in the pixel coordinate system, and use these as all target point clouds that fall into the target rectangle corresponding to the laid brick.

[0206] Alternatively, for each laid brick, the target rectangle corresponding to that laid brick is determined based on the corner points of the bricks identified for that laid brick, and the corner points of the bricks identified for that laid brick are determined in the following way:

[0207] An initial rectangle is determined based on the location of the laid bricks. Sub-rectangles are generated on each side of the initial rectangle according to the predetermined distance and width values. The predetermined distance value is the distance between each sub-rectangle and the vertex of the initial rectangle, and the predetermined width value is the width of each sub-rectangle.

[0208] Within each sub-rectangle, identify several boundary points on the boundary of the laid bricks, and fit a straight line corresponding to each boundary based on the identified boundary points on each boundary.

[0209] Extend the straight lines corresponding to all boundaries of the laid bricks to obtain the intersection points of the extended straight lines of all boundaries. Determine the brick corner points of the target rectangle corresponding to the laid bricks as the brick corner points.

[0210] As can be seen, this optional embodiment can also determine the coordinate system of the center point of the brick to be laid based on the relevant information of the already laid bricks. Based on the brick-laying method, it realizes the intelligent and precise determination of the posture of the brick to be laid, which is conducive to achieving efficient and highly accurate automated brick laying based on the brick-laying robot.

[0211] In another optional embodiment, the determining module 403 determines the origin and three-dimensional coordinate axes of the center point coordinate system to be established based on the brick laying information determination method, and creates the center point coordinate system corresponding to the brick to be laid based on the determined origin and three-dimensional coordinate axes. The specific methods for providing the center point information corresponding to the brick to be laid include:

[0212] When the paving information is determined by line-based paving, identify the target marked area on the paving mark line;

[0213] Determine the center point of the area corresponding to the cross intersection of the target marking area, and map the center point of the area to the color camera coordinate system to obtain the first mapped coordinate point of the center point of the area in the color camera coordinate system.

[0214] In the horizontal direction of the cross-shaped area, select a target reference point that does not coincide with the center point of the area, and map the target reference point to the color camera coordinate system to obtain the second mapped coordinate point of the target reference point in the color camera coordinate system.

[0215] The X-axis is established based on the first and second mapped coordinate points, the Z-axis is determined based on the plane normal vector, and the Y-axis is determined based on the determined X-axis and Z-axis. The first mapped coordinate point is used as the origin, and the center point coordinate system corresponding to the brick to be laid is created based on the determined X-axis, Y-axis and Z-axis, which serves as the center point information of the brick to be laid.

[0216] As can be seen, this optional embodiment can also determine the coordinate system of the center point of the brick to be laid based on the line-based brick laying method. The line-based brick laying method realizes the intelligent and accurate determination of the posture of the brick to be laid, which is conducive to achieving efficient and highly accurate automated brick laying based on the brick laying robot.

[0217] Example 5

[0218] Please see Figure 5 , Figure 5 This is a schematic diagram of another intelligent brick-laying device based on a depth camera disclosed in an embodiment of the present invention. Figure 5 The described device is used to control a tiling robot equipped with a robotic arm and a depth camera to perform tiling operations. Specifically, the depth camera can be mounted on the end effector of the robotic arm and move with it. Optionally, this device can be integrated into the tiling robot or exist independently of it, such as being located on a cloud control platform. This embodiment of the invention does not impose limitations. Figure 5 As shown, the intelligent paving device based on a depth camera may include:

[0219] Memory 501 storing executable program code;

[0220] Processor 5402 coupled to memory 501;

[0221] The processor 502 calls the executable program code stored in the memory 501 to execute some or all of the steps in the intelligent paving method based on a depth camera disclosed in any of Embodiments 1-3 of the present invention.

[0222] Example 6

[0223] This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute some or all of the steps in any of the intelligent paving methods based on depth cameras disclosed in Embodiments 1-3 of this invention.

[0224] Example 7

[0225] This invention discloses a tiling robot equipped with a robotic arm and a depth camera. The depth camera is mounted on the end of the robotic arm and moves with it. The tiling robot performs some or all of the steps in any of the depth camera-based intelligent tiling methods disclosed in Embodiments 1-3 of this invention. Alternatively, the tiling robot may include any of the depth camera-based intelligent tiling devices described in Embodiment 4.

[0226] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0227] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0228] Finally, it should be noted that the intelligent paving method and device based on depth camera and the paving robot disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent brick-laying based on a depth camera, characterized in that, The method is applied to a tiling robot, which is equipped with a robotic arm and a depth camera. The depth camera is mounted on the end of the robotic arm and moves with the robotic arm. The method includes: The image of the paved area captured by the depth camera is acquired, the image of the paved area is analyzed to obtain the image analysis result, and the image analysis result is used to determine whether the paved area image includes paving marking lines, and the determination result is obtained. Obtain a paving information determination method that matches the judgment result. When the judgment result indicates that the collected paving area image includes the paving mark line, the paving information determination method is a line-based paving method based on the paving mark line. When the judgment result indicates that the collected paving area image does not include the paving mark line, the paving information determination method is a line-based paving method based on the already laid bricks. Based on the brick laying information determination method, the origin and three-dimensional coordinate axes of the center point coordinate system to be established are determined, and the center point coordinate system corresponding to the brick to be laid is created based on the determined origin and three-dimensional coordinate axes, as the center point information corresponding to the brick to be laid. The robotic arm is controlled to perform a matching brick-laying operation based on the center point information corresponding to the brick to be laid; The process of determining the origin and three-dimensional coordinate axes of the center point coordinate system to be established based on the brick laying information determination method, and creating a center point coordinate system corresponding to the brick to be laid based on the determined origin and three-dimensional coordinate axes, as the center point information corresponding to the brick to be laid, includes: When the brick laying information is determined by the brick-viewing and brick-laying method, all laid bricks within the field of view of the depth camera are identified based on the image of the brick laying area. For each laid brick, determine the coordinate information of the four corner points of the target rectangle corresponding to the laid brick; Based on the coordinate information corresponding to all the brick corner points, the four brick corner points of the target rectangle corresponding to all the laid bricks are fitted into a target plane, and all the brick corner points are vertically projected onto the target plane according to the normal vector to obtain all the projected brick corner points of all the brick corner points on the target plane. For each of the laid bricks, a corresponding grid is fitted according to the four projected brick corner points in the horizontal and vertical directions, and the position number of the bricks to be laid is calculated according to the paving plan. Based on the location number and all the fitted grids, determine two target horizontal lines and two target vertical lines that match the location number. Based on the two target horizontal lines and the two target vertical lines, determine the four target brick corner points of the brick to be laid. Based on the four target brick corner points, determine the center point of the brick to be laid. Create a center point coordinate system for the brick to be laid with the center point of the brick to be laid as the origin, the horizontal axis as the X-axis, and the vertical axis as the Y-axis, as the center point information of the brick to be laid.

2. The intelligent paving method based on a depth camera according to claim 1, characterized in that, The step of controlling the robotic arm to perform a matching brick-laying operation based on the center point information corresponding to the brick to be laid includes: Based on the center point information of the brick to be laid, the robot arm is generated with corresponding brick-laying control parameters, and the robot arm is controlled to perform the brick-laying operation corresponding to the brick to be laid based on the brick-laying control parameters.

3. The intelligent paving method based on a depth camera according to claim 1, characterized in that, For each laid brick, determining the coordinate information of the four corner points of the target rectangle corresponding to that laid brick includes: For each paved brick, all target point clouds falling into the target rectangle corresponding to the paved brick are identified from all point clouds acquired by the depth camera. All target point clouds falling into the target rectangle corresponding to the paved brick are fitted into a plane equation, and the four corner points of the target rectangle corresponding to the paved brick are mapped from the pixel coordinate system to the color camera coordinate system to obtain the mapping result. Based on the plane equation fitted by all target point clouds falling into the target rectangle corresponding to the paved brick and the mapping result, the coordinate information corresponding to the four corner points of the target rectangle corresponding to the paved brick is determined.

4. The intelligent paving method based on a depth camera according to claim 3, characterized in that, For each of the laid bricks, identifying all target point clouds falling within the target rectangle corresponding to that laid brick from all point clouds acquired by the depth camera includes: Based on the determined transformation matrix, all point clouds acquired by the depth camera are transformed into the color camera coordinate system to obtain the coordinate points corresponding to each point cloud in the color camera coordinate system; Based on a predetermined transformation formula, the coordinates of all the point clouds in the color camera coordinate system are transformed to the pixel coordinate system to obtain the coordinates of all the point clouds in the pixel coordinate system. For each laid brick, identify all target coordinate points located within the target rectangle corresponding to the laid brick from the coordinate points corresponding to all point clouds in the pixel coordinate system, and use these as all target point clouds falling into the target rectangle corresponding to the laid brick.

5. The intelligent paving method based on a depth camera according to claim 3 or 4, characterized in that, For each laid brick, the target rectangle corresponding to the laid brick is determined based on the brick corner points identified for that laid brick, and the brick corner points identified for that laid brick are determined in the following manner: An initial rectangle is determined based on the location of the laid bricks. Sub-rectangles are generated on each side of the initial rectangle according to a predetermined distance value and a predetermined width value. The predetermined distance value is the distance of each sub-rectangle from the vertex of the initial rectangle, and the predetermined width value is the width of each sub-rectangle. Within each sub-rectangle, identify several boundary points on the boundary of the laid bricks, and fit a straight line corresponding to each boundary based on the identified boundary points on each boundary. Extend the straight lines corresponding to all the boundaries of the laid bricks to obtain the intersection points of the extended straight lines corresponding to all the boundaries, and determine the brick corner points of the target rectangle corresponding to the laid bricks.

6. The intelligent paving method based on a depth camera according to claim 1, characterized in that, The method of determining the origin and three-dimensional coordinate axes of the center point coordinate system to be established based on the paving information determination method, and creating the center point coordinate system corresponding to the paved bricks based on the determined origin and three-dimensional coordinate axes, as the center point information corresponding to the paved bricks, further includes: When the paving information is determined using the line-based paving method, the target marking area on the paving marking line is identified; Determine the center point of the area corresponding to the cross intersection area of ​​the target marking area, and map the center point of the area to the color camera coordinate system to obtain the first mapped coordinate point of the center point of the area in the color camera coordinate system. In the lateral direction of the cross-shaped area, a target reference point that does not coincide with the center point of the area is selected, and the target reference point is mapped to the color camera coordinate system to obtain the second mapped coordinate point of the target reference point in the color camera coordinate system. An X-axis is established based on the first and second mapped coordinate points, a Z-axis is determined based on the plane normal vector, and a Y-axis is determined based on the determined X-axis and Z-axis. A center point coordinate system corresponding to the brick to be laid is created with the first mapped coordinate point as the origin and the determined X-axis, Y-axis, and Z-axis, which serves as the center point information corresponding to the brick to be laid.

7. An intelligent brick-laying device based on a depth camera, characterized in that, The device is applied to a paving robot, which is equipped with a robotic arm and a depth camera. The depth camera is mounted on the end of the robotic arm and moves with the robotic arm. The device includes: The image analysis module is used to acquire images of the paved area captured by the depth camera, analyze the images of the paved area, and obtain image analysis results. The image judgment module is used to determine whether the paving area image includes paving marking lines based on the image analysis results, and to obtain the judgment result; The determination module is used to obtain a paving information determination method that matches the judgment result. When the judgment result indicates that the acquired paving area image includes the paving mark line, the paving information determination method is a line-based paving method based on the paving mark line. When the judgment result indicates that the acquired paving area image does not include the paving mark line, the paving information determination method is a line-based paving method based on already laid bricks. Based on the paving information determination method, the origin and three-dimensional coordinate axes of the center point coordinate system to be established are determined, and a center point coordinate system corresponding to the brick to be laid is created based on the determined origin and three-dimensional coordinate axes, serving as the center point information corresponding to the brick to be laid. The brick-laying control module is used to control the robotic arm to perform matching brick-laying operations based on the center point information corresponding to the brick to be laid. The determining module determines the origin and three-dimensional coordinate axes of the center point coordinate system to be established based on the paving information determination method, and creates the center point coordinate system corresponding to the paving bricks based on the determined origin and three-dimensional coordinate axes. Specific methods for providing the center point information corresponding to the paving bricks include: When the brick laying information is determined by the brick-viewing and brick-laying method, all laid bricks within the field of view of the depth camera are identified based on the image of the brick laying area. For each laid brick, determine the coordinate information of the four corner points of the target rectangle corresponding to the laid brick; Based on the coordinate information corresponding to all the brick corner points, the four brick corner points of the target rectangle corresponding to all the laid bricks are fitted into a target plane, and all the brick corner points are vertically projected onto the target plane according to the normal vector to obtain all the projected brick corner points of all the brick corner points on the target plane. For each of the laid bricks, a corresponding grid is fitted according to the four projected brick corner points in the horizontal and vertical directions, and the position number of the bricks to be laid is calculated according to the paving plan. Based on the location number and all the fitted grids, determine two target horizontal lines and two target vertical lines that match the location number. Based on the two target horizontal lines and the two target vertical lines, determine the four target brick corner points of the brick to be laid. Based on the four target brick corner points, determine the center point of the brick to be laid. Create a center point coordinate system for the brick to be laid with the center point of the brick to be laid as the origin, the horizontal axis as the X-axis, and the vertical axis as the Y-axis, as the center point information of the brick to be laid.

8. An intelligent brick-laying device based on a depth camera, characterized in that, The device is applied to a paving robot, which is equipped with a robotic arm and a depth camera. The depth camera is mounted on the end of the robotic arm and moves with the robotic arm. The device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the intelligent paving method based on a depth camera as described in any one of claims 1-6.

9. A brick-laying robot, characterized in that, The paving robot is equipped with a robotic arm and a depth camera. The depth camera is mounted on the end of the robotic arm and moves with the robotic arm. The paving robot is used to perform the intelligent paving method based on a depth camera as described in any one of claims 1-6.

Images