Vision system, vision device and vision guidance method

By using a combination of tilted projection film and laser module in intelligent robots, the problem of difficulty in laser line recognition during low-level stacking in traditional technology is solved, and the accurate identification of laser line and stacking accuracy is achieved.

CN115446825BActive Publication Date: 2024-11-08JIUZHANG LINGZHI (GUANGZHOU) DIGITAL TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110638320.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-08
Publication Date
2024-11-08
Estimated Expiration
2041-06-08

AI Technical Summary

Technical Problem

When traditional intelligent robots stack at low levels, the plane projection film cannot recognize laser lines, resulting in the problem of incorrect stacking.

Method used

Using a combination of an inclined projection film and a laser module, the laser module emits horizontal laser lines, and the inclined projection film is set inclined in the horizontal direction to ensure that the laser lines are between the upper and lower edges of the inclined projection film. The acquisition component is used to capture real-time projection images, and the control module is based on the ratio. The result output action commands are used to guide the action of the movable actuator to achieve correct stacking.

Benefits of technology

The accurate identification and comparison of laser lines is achieved in the low-level area, ensuring the accuracy and correctness of the stacking, and solving the problem of identification difficulties in traditional technology.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115446825B_ABST
    Figure CN115446825B_ABST
Patent Text Reader

Abstract

The present application relates to a visual system, a visual device and a visual guidance method. The visual system includes: a laser module, fixed in the construction space and located outside the construction range of the movable actuator, the laser module is used to emit a laser beam, and the laser beam includes a horizontal laser line; a visual module, movably connected to the movable actuator, the visual module includes: an inclined projection film, facing the ground of the construction space and arranged along an inclined direction relative to the horizontal direction, for imaging the projection of the horizontal laser line; a collection component, located on the side of the inclined projection film away from the ground of the construction space, for shooting the projection of the horizontal laser line; when the movable actuator is located at the bottom working position, at least part of the horizontal height of the inclined projection film is higher than the horizontal laser line. The use of this device can achieve correct stacking at a low position.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of robotics, and in particular to a vision system, a vision device, and a vision guidance method. Background Art

[0002] With the development of computer technology, intelligent robots have emerged. Intelligent robots are intelligent machines that can work semi-autonomously or fully autonomously. They can be applied to multiple technical fields to perform various tasks. For example, intelligent robots can be used to achieve automatic stacking. For example, in the construction field, intelligent robots can be used to lay bricks.

[0003] In traditional technology, when using intelligent robots to achieve automatic stacking, the stacking method used is to use a visual box + plane projection film to identify the laser line, use the laser line as a reference line, and use a unified method of aligning with the reference line to achieve stacking.

[0004] However, when stacking at a low position close to the ground, the flat projection film used in the traditional stacking method has the problem of being unable to recognize the laser line, thereby failing to achieve correct stacking. Summary of the invention

[0005] Based on this, it is necessary to provide a visual system that can be used to achieve correct stacking at a low position, a visual device that can achieve correct stacking at a low position, and a visual guidance method to address the above technical problems.

[0006] A visual system is used for the posture calibration of a movable actuator of a construction robot in a construction space. The construction range of the movable actuator includes a preset bottom working position, and the bottom working position corresponds to a preset bottom waiting area in the construction space. The visual system includes:

[0007] A laser module is fixed in the construction space and is located outside the construction range of the movable actuator, and the laser module is used to emit a laser beam, and the laser beam includes a horizontal laser line;

[0008] The visual module is connected to the movable actuator, and the visual module includes:

[0009] An inclined projection film, facing the floor of the construction space and arranged along an inclination relative to the horizontal direction, is used for imaging the projection of the horizontal laser line;

[0010] A collection component is located on the side of the inclined projection film facing away from the ground of the construction space, and is used to capture the projection of the horizontal laser line;

[0011] When the movable actuator is located at the bottom working position, at least a portion of the horizontal height of the inclined projection film is higher than the horizontal laser line.

[0012] In one embodiment, there are two inclined projection films, the two inclined projection films are symmetrically arranged with respect to a plane along the vertical direction, and the distance between the top ends of the inclined projection films is greater than the distance between the bottom ends.

[0013] In one embodiment, when the movable actuator is located at the bottom working position, the upper edge of the inclined projection film is higher than the horizontal laser line, and the lower edge of the inclined projection film is lower than the horizontal laser line.

[0014] A visual device, comprising a construction robot and the above-mentioned visual system;

[0015] Construction robots include:

[0016] A movable actuator for performing operations within the construction space;

[0017] The control module is connected to the visual module and the movable actuator through communication, and outputs action instructions to guide the action of the movable actuator according to the real-time projection image taken by the acquisition component.

[0018] In one embodiment, the construction robot is a masonry robot, the movable actuator is a masonry robot arm, the end of the masonry robot arm has a clamp, the clamp is used to take and place bricks, and the vision module is installed on the clamp.

[0019] In one embodiment, the construction robot further includes an inclination sensor, which is connected to the movable actuator and is used to collect an inclination angle of the movable actuator relative to a horizontal plane.

[0020] A visual guidance method, applied to the above-mentioned visual device, comprising:

[0021] The laser module emits a laser beam including a horizontal laser line;

[0022] When the movable actuator is located at the bottom working position, the control module outputs a laser line comparison instruction to the vision module;

[0023] The visual module, according to the laser line comparison instruction, uses the acquisition component to capture the real-time projection image of the horizontal laser line imaged on the inclined projection film, and feeds back the laser line comparison result to the control module according to the real-time projection image;

[0024] The control module outputs action instructions to guide the action of the movable actuator according to the laser line comparison result.

[0025] In one embodiment, the control module outputs an action instruction according to the laser line comparison result to guide the movable actuator to act, including:

[0026] When the laser line comparison result meets the preset operation condition, the operation instruction is output to control the movable actuator to perform the operation. The preset operation condition is that the angle and offset between the horizontal laser line image in the real-time projection image and the preset teaching laser line image are both less than the corresponding preset operation threshold;

[0027] When the laser line comparison result does not meet the preset operating conditions, the output posture adjustment instruction controls the movable actuator to adjust the posture.

[0028] In one embodiment, when the laser line comparison result does not meet the preset working condition, after outputting the posture adjustment instruction to control the movable actuator to adjust the posture, it also includes:

[0029] Obtain secondary laser line comparison results;

[0030] When the secondary laser line comparison result meets the preset operation conditions, the operation instruction is output to control the movable actuator to perform the operation;

[0031] When the secondary laser line comparison result does not meet the preset working conditions, the step of outputting the posture adjustment instruction to control the movable actuator to adjust the posture is returned until the latest secondary laser line comparison result meets the preset working conditions, and the output working instruction is used to control the movable actuator to perform the operation.

[0032] In one embodiment, the method further comprises:

[0033] Also includes:

[0034] The control module obtains the inclination angle of the movable actuator relative to the horizontal plane, and controls the movable actuator to adjust the posture according to the inclination angle until the inclination angle is less than a preset inclination angle threshold.

[0035] The above-mentioned visual system utilizes the horizontal laser line emitted by the tilted projection film imaging laser module, and utilizes the acquisition component to capture the projection of the horizontal laser line. When the movable actuator is located at the bottom working position, at least part of the horizontal height of the tilted projection film is higher than the horizontal laser line, so that the horizontal laser line that cannot be recognized by the flat projection film is located between the upper and lower edges of the tilted projection film, thereby realizing accurate recognition of the horizontal laser line when stacking in the bottom working area, so that the horizontal laser line can be used for visual guidance to achieve correct stacking.

[0036] The above-mentioned visual device utilizes the control module to communicate and connect the visual module and the movable actuator, and outputs action instructions according to the real-time projection image taken by the acquisition component until the movable actuator is in motion, so that the movable actuator can operate in the construction space, and can enable the movable actuator to achieve correct stacking according to the guidance of the real-time projection image of the visual system.

[0037] The above-mentioned visual guidance method, when the movable actuator is located at the bottom working position, first uses the control module to output a laser line comparison instruction to the visual module, so that the visual module captures the real-time projection image of the horizontal laser line imaged on the inclined projection film through the acquisition component according to the laser line comparison instruction, and feeds back the laser line comparison result to the control module according to the real-time projection image, and then uses the control module to output an action instruction according to the laser line comparison result to guide the action of the movable actuator, so that the movable actuator can operate in the construction space, and can achieve correct stacking according to the guidance of the real-time projection image of the visual system. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a structural block diagram of a visual system in one embodiment;

[0039] Figure 2 is a schematic diagram of a laser emitting surface of a laser module in one embodiment;

[0040] Figure 3 is a schematic diagram of laser alignment in one embodiment;

[0041] Figure 4 is a schematic diagram of a visual module in one embodiment;

[0042] Figure 5 is a structural block diagram of a visual device in one embodiment;

[0043] Figure 6 is a flow chart of a visual guidance method in one embodiment;

[0044] Figure 7 is a schematic diagram of teaching under visual guidance in one embodiment;

[0045] Figure 8 A schematic diagram of a construction robot during masonry construction in one embodiment;

[0046] Fig. 9 A schematic diagram of a construction robot during masonry construction in another embodiment;

[0047] Fig.10 A schematic diagram of a construction robot during masonry construction in another embodiment;

[0048] Fig.11 A schematic diagram of a construction robot during masonry construction in yet another embodiment;

[0049] Fig.12 It is a schematic diagram of a construction robot laying bricks in another embodiment. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] In one embodiment, Figure 1 As shown, a visual system is provided, which is applied to the posture calibration of the movable actuator of the construction robot in the construction space. The construction range of the movable actuator includes a preset bottom working position, and the bottom working position corresponds to a preset bottom waiting area in the construction space. The visual system includes:

[0052] A laser module 102 is fixed in the construction space and is located outside the construction range of the movable actuator. The laser module is used to emit a laser beam, and the laser beam includes a horizontal laser line 110;

[0053] The visual module 104 is movably connected to the movable actuator, and the visual module 104 includes:

[0054] The inclined projection film 106 faces the floor of the construction space and is arranged along an inclination relative to the horizontal direction, and is used for imaging the projection of the horizontal laser line;

[0055] The collection component 108 is located on the side of the inclined projection film 106 away from the ground of the construction space, and is used to capture the projection of the horizontal laser line;

[0056] When the movable actuator is located at the bottom working position, at least a portion of the horizontal height of the tilted projection film 106 is higher than the horizontal laser line.

[0057] The preset bottom waiting area refers to a preset waiting area close to the ground, and the bottom operation position corresponds to the bottom waiting area. For example, the bottom waiting area may specifically refer to an operation area whose distance from the ground is less than a preset bottom distance threshold, and the bottom distance threshold may be set as needed. For example, the distance threshold may specifically be 0.5 meters.

[0058] The laser module refers to a device that can emit a laser beam. For example, the laser module can specifically refer to a laser that emits a laser beam. The laser beam can form a laser surface, including a horizontal laser line in the laser beam. For example, when the laser module is placed in a fixed position in the lower left corner, the laser surface formed by the emitted laser beam can be as follows: Figure 2As shown, the horizontal laser line is 202. The tilted projection film refers to a device that can project the laser line in the laser beam, and is used to image the projection of the laser line. The acquisition component refers to a component that captures the laser line projected on the tilted projection film. For example, the acquisition component can specifically refer to a camera, and the acquisition component is located on the side of the tilted projection film away from the ground of the construction space. Furthermore, the acquisition component can be set directly opposite the tilted projection film to achieve accurate acquisition.

[0059] Specifically, when the movable actuator is located at the bottom working position, it means that work needs to be performed on the bottom working area. The inclined projection film will have at least part of its horizontal height higher than the horizontal laser line in the laser beam emitted by the laser module, so that the horizontal laser line is located between the upper edge and the lower edge of the inclined projection film, that is, the projection of the horizontal laser line will be imaged on the inclined projection film. Therefore, the acquisition component located on the side of the inclined projection film away from the ground of the construction space can capture the projection of the horizontal laser line. The captured projection of the horizontal laser line can be used to calibrate the position of the movable actuator of the construction robot in the construction space.

[0060] Furthermore, the acquisition component will also extract a real-time horizontal laser line image based on the projection of the captured horizontal laser line, that is, the real-time projection image, and output the laser line comparison result of the horizontal laser line image and the preset teaching laser line image to the construction robot, so that the construction robot can calibrate the movable actuator according to the laser line comparison result. Among them, the teaching laser line image refers to the image stored according to the laser line used for teaching, which can be used to compare with the real-time laser line image to obtain the laser line comparison result. For example, in the stacking process, the teaching laser line image can specifically refer to the image obtained according to the teaching line parallel to the wall to be stacked. For example, when laying bricks, the teaching laser line image can specifically refer to the image obtained according to the teaching line parallel to the wall to be built. The laser line comparison result refers to the offset angle and offset of the real-time laser line image relative to the teaching laser line image obtained by comparing the real-time laser line image and the teaching laser line image. In this embodiment, the real-time laser line image refers to the horizontal laser line image. The offset angle refers to the offset angle of the real-time laser line image compared to the teaching laser line image, and the offset amount refers to the position offset of the real-time laser line image compared to the teaching laser line image. For example, the offset angle and offset amount between the real-time laser line image and the teaching laser line image can be as follows: Figure 3 shown.

[0061] Specifically, in this embodiment, during posture calibration, it is necessary to obtain the teaching laser line image in advance, and the method of obtaining the teaching laser line image is: the laser module emits a laser beam, the laser beam includes a teaching laser line, the projection of the teaching laser line is imaged on a tilted projection film, and the acquisition component captures the projection of the teaching laser line to obtain the teaching laser line image.

[0062] It should be noted that, in this embodiment, the acquisition component in the visual module is set in a light-proof environment, so the real-time projection image captured will only include the real-time laser line image without other interference. Therefore, the method of extracting the horizontal laser line image from the real-time projection image can be to extract it by detecting the pixel value of each pixel point in the real-time projection image. In the real-time projection image, only the pixel value of the pixel point corresponding to the horizontal laser line image is a non-zero value.

[0063] The above-mentioned visual system utilizes the horizontal laser line emitted by the tilted projection film imaging laser module, and utilizes the acquisition component to capture the projection of the horizontal laser line. When the movable actuator is located at the bottom working position, at least part of the horizontal height of the tilted projection film is higher than the horizontal laser line, so that the horizontal laser line that cannot be recognized by the flat projection film is located between the upper and lower edges of the tilted projection film, thereby realizing accurate recognition of the horizontal laser line when stacking in the bottom working area, so that the horizontal laser line can be used for visual guidance to achieve correct stacking.

[0064] In one embodiment, there are two inclined projection films, the two inclined projection films are symmetrically arranged with respect to a plane along the vertical direction, and the distance between the top ends of the inclined projection films is greater than the distance between the bottom ends.

[0065] Specifically, the number of the inclined projection films can be two and they are symmetrically arranged with respect to the plane along the numerical direction, the distance between the top ends of the inclined projection films is greater than the distance between the top ends, and the distance between the top ends and the ground is greater than the distance between the bottom ends and the ground. Figure 4 In the V-shaped projection film shown, the collection component is arranged opposite to the inclined projection film, and the control component is arranged at any position inside the visual module without blocking the collection component.

[0066] In one embodiment, when the movable actuator is located at the bottom working position, the upper edge of the inclined projection film is higher than the horizontal laser line, and the lower edge of the inclined projection film is lower than the horizontal laser line.

[0067] Specifically, when the movable actuator is located at the bottom operating position, the upper edge of the inclined projection film is higher than the horizontal laser line, and the lower edge is lower than the horizontal laser line, and the horizontal laser line can be accurately projected onto the inclined projection film.

[0068] In one embodiment, Figure 5 As shown, a visual device is provided, including a construction robot 502 and the above-mentioned visual system 504;

[0069] The construction robot 502 includes:

[0070] A movable actuator 506 is used to perform operations in the construction space;

[0071] The control module 508 is communicatively connected to the visual module and the movable actuator, and outputs action instructions to guide the action of the movable actuator according to the real-time projection image captured by the acquisition component.

[0072] Specifically, when the movable actuator is located at the bottom working position, the laser module emits a laser beam, and the horizontal laser line included in the laser beam will be projected onto the inclined projection film. At least part of the horizontal height of the inclined projection film is higher than the horizontal laser line. The control module will output a laser line comparison instruction to the visual module, so that the acquisition component in the visual module can capture the real-time projection image and feedback the laser line comparison result based on the real-time projection image. The control module will output an action instruction based on the laser line comparison result to guide the action of the movable actuator.

[0073] Specifically, the action instructions can be divided into performing operations and performing posture adjustments. When the laser line comparison results meet the preset operating conditions, the control module will output operation instructions to control the movable actuator to perform operations. The preset operating conditions are that the angle and offset between the horizontal laser line image in the real-time projection image and the preset teaching laser line image are both less than the corresponding preset operation threshold. When the laser line comparison results do not meet the preset operating conditions, the control module will output posture adjustment instructions to control the movable actuator to perform posture adjustments.

[0074] Furthermore, after controlling the movable actuator to adjust the posture, the control module will again output a laser line comparison instruction to the vision module when the posture adjustment is completed, and receive the secondary laser line comparison result fed back by the vision module. When the secondary laser line comparison result meets the preset operating conditions, the control module outputs an operating instruction to control the movable actuator to perform the operation. When the secondary laser line comparison result does not meet the preset operating conditions, the control module outputs a posture adjustment instruction to control the movable actuator to adjust the posture, and continues to output the laser line comparison instruction after the posture adjustment until the received laser line comparison result meets the preset operating conditions, and then outputs an operating instruction to control the movable actuator to perform the operation.

[0075] The above-mentioned visual device utilizes the control module to communicate and connect the visual module and the movable actuator, and outputs action instructions according to the real-time projection image taken by the acquisition component until the movable actuator is in motion, so that the movable actuator can operate in the construction space, and can enable the movable actuator to achieve correct stacking according to the guidance of the real-time projection image of the visual system.

[0076] In one embodiment, the construction robot is a masonry robot, the movable actuator is a masonry robot arm, the end of the masonry robot arm has a clamp, the clamp is used to take and place bricks, and the vision module is installed on the clamp.

[0077] Specifically, when the masonry robot arm is located at the bottom working position and brick laying is required, the vision module is installed on the fixture and will move with the movement of the masonry robot arm. The inclined projection film in the vision module will image the projection of the horizontal laser line in the laser beam emitted by the laser module. The acquisition component will capture the corresponding real-time projection image and feedback the laser line comparison result based on the real-time projection image, so that the control module can output action instructions according to the laser line comparison result to guide the movement of the masonry robot arm.

[0078] Specifically, when the laser line comparison result meets the preset operating conditions, the control module will output an operating instruction to control the masonry robot arm to perform masonry, and the angle at the end of the masonry robot arm will take bricks for masonry. When the laser line comparison result does not meet the preset operating conditions, the control module will output a posture adjustment instruction to control the masonry robot arm to adjust the posture, wherein the preset operating conditions are that the angle and offset between the horizontal laser line and the preset teaching laser line are both less than the corresponding preset operating threshold.

[0079] In one embodiment, the construction robot further includes an inclination sensor, which is connected to the movable actuator and is used to collect an inclination angle of the movable actuator relative to a horizontal plane.

[0080] Specifically, the construction robot also includes a tilt sensor, which is connected to the movable actuator and is used to collect the tilt angle of the movable actuator relative to the horizontal plane. When starting the operation, the control module will output a start operation instruction to the tilt sensor, so that the tilt sensor collects the tilt angle of the movable actuator relative to the horizontal plane and feeds back. The control module will control the movable actuator to adjust its posture according to the tilt angle until the tilt angle is less than the preset tilt angle threshold. Among them, the start operation instruction includes the start teaching instruction and the start actual operation instruction, that is, before teaching and before operation, the tilt sensor needs to be used to detect whether the movable actuator has been leveled. Only when the movable actuator has been leveled can teaching or operation be performed.

[0081] In this embodiment, the tilt sensor can be used to adjust the position of the movable actuator.

[0082] In one embodiment, the control module is also used to move the movable actuator to a preset displacement feature point, obtain the device movement coordinates corresponding to the preset displacement feature point, output the device movement coordinates and the calibration instructions to the visual module, so that the visual module controls the acquisition component to acquire the calibration image corresponding to the preset displacement feature point according to the calibration instructions, and obtains the calibration matrix according to the pixel coordinates of the preset displacement feature point in the calibration image and the device movement coordinates, so that the visual module obtains the laser line comparison result according to the calibration matrix, the real-time projection image and the preset teaching laser line image.

[0083] The preset displacement feature points refer to the feature points pre-set for calibrating the visual module. The device movement coordinates refer to the movement coordinates of the movable actuator. The calibration matrix refers to the matrix used to characterize the conversion relationship between the pixel coordinates of the image collected by the visual module and the robot world coordinates.

[0084] Specifically, before starting the operation, the visual module needs to be calibrated. The control module will move the movable actuator to the preset displacement feature point, obtain the device movement coordinates corresponding to the preset displacement feature point, and output the device movement coordinates and calibration instructions to the visual module, so that the visual module collects the calibration image corresponding to the preset displacement feature point, and obtains the calibration matrix according to the pixel coordinates and device movement coordinates of the preset displacement feature point in the calibration image. The calibration matrix can be used to combine the real-time projection image and the preset teaching laser line image to obtain the laser line comparison result. In this embodiment, the specific calibration method is not limited. For example, the calibration method in this embodiment can be specifically a nine-point calibration, that is, using the visual module to collect and specify unique nine feature points, and record the device movement coordinates of the movable actuator, and establish a conversion matrix between pixel coordinates and robot world coordinates through the nine-point calibration method, that is, the calibration matrix.

[0085] Among them, combining the calibration matrix, the real-time projection image and the preset teaching laser line image to obtain the laser line comparison result means that after extracting the real-time laser line image from the real-time projection image, the real-time laser line image can be converted into the robot world coordinate system according to the pixel coordinates of the real-time laser line image in the real-time projection image and the calibration matrix, so as to obtain the coordinate data of the real-time laser line image in the robot world coordinate system, and thus obtain the laser line comparison result according to the coordinate data and the preset teaching laser line image.

[0086] In this embodiment, the image acquisition component can be calibrated by combining preset displacement feature points, device movement coordinates, and captured calibration images.

[0087] In one embodiment, Figure 6 As shown, a visual guidance method is provided, which is applied to the above-mentioned visual device, comprising:

[0088] Step 602, the laser module emits a laser beam including a horizontal laser line;

[0089] Step 604, when the movable actuator is located at the bottom working position, the control module outputs a laser line comparison instruction to the vision module;

[0090] Step 606, the visual module captures the real-time projection image of the horizontal laser line imaged on the inclined projection film through the acquisition component according to the laser line comparison instruction, and feeds back the laser line comparison result to the control module according to the real-time projection image;

[0091] Step 608: The control module outputs an action instruction to guide the movable actuator to move according to the laser line comparison result.

[0092] Specifically, the laser module emits a laser beam including a horizontal laser line. When the movable actuator is located at the bottom working position, the control module outputs a laser line comparison instruction to the vision module. The vision module captures a real-time projection image of the horizontal laser line imaged on the inclined projection film through the acquisition component according to the laser line comparison instruction, and feeds back the laser line comparison result to the control module according to the real-time projection image. The control module outputs an action instruction according to the laser line comparison result to guide the action of the movable actuator.

[0093] Specifically, the action instructions can be divided into performing operations and performing posture adjustments. When the laser line comparison results meet the preset operating conditions, the control module will output operation instructions to control the movable actuator to perform operations. The preset operating conditions are that the angle and offset between the horizontal laser line image in the real-time projection image and the preset teaching laser line image are both less than the corresponding preset operation threshold. When the laser line comparison results do not meet the preset operating conditions, the control module will output posture adjustment instructions to control the movable actuator to perform posture adjustments.

[0094] Furthermore, after controlling the movable actuator to adjust the posture, the control module will again output a laser line comparison instruction to the vision module when the posture adjustment is completed, and receive the secondary laser line comparison result fed back by the vision module. When the secondary laser line comparison result meets the preset operating conditions, the control module outputs an operating instruction to control the movable actuator to perform the operation. When the secondary laser line comparison result does not meet the preset operating conditions, the control module outputs a posture adjustment instruction to control the movable actuator to adjust the posture, and continues to output the laser line comparison instruction after the posture adjustment until the received laser line comparison result meets the preset operating conditions, and then outputs an operating instruction to control the movable actuator to perform the operation.

[0095] The above-mentioned visual guidance method, when the movable actuator is located at the bottom working position, first uses the control module to output a laser line comparison instruction to the visual module, so that the visual module captures the real-time projection image of the horizontal laser line imaged on the inclined projection film through the acquisition component according to the laser line comparison instruction, and feeds back the laser line comparison result to the control module according to the real-time projection image, and then uses the control module to output an action instruction according to the laser line comparison result to guide the action of the movable actuator, so that the movable actuator can operate in the construction space, and can achieve correct stacking according to the guidance of the real-time projection image of the visual system.

[0096] In one embodiment, the control module outputs an action instruction according to the laser line comparison result to guide the movable actuator to act, including:

[0097] When the laser line comparison result meets the preset operation condition, the operation instruction is output to control the movable actuator to perform the operation. The preset operation condition is that the angle and offset between the horizontal laser line image in the real-time projection image and the preset teaching laser line image are both less than the corresponding preset operation threshold;

[0098] When the laser line comparison result does not meet the preset operating conditions, the output posture adjustment instruction controls the movable actuator to adjust the posture.

[0099] Specifically, when the laser line comparison result meets the preset operating conditions, the control module will output an operating instruction to control the movable actuator to perform the operation. When the laser line comparison result does not meet the preset operating conditions, the control module will output a posture adjustment instruction to control the movable actuator to adjust the posture.

[0100] In one embodiment, when the laser line comparison result does not meet the preset working condition, after outputting the posture adjustment instruction to control the movable actuator to adjust the posture, it also includes:

[0101] Obtain secondary laser line comparison results;

[0102] When the secondary laser line comparison result meets the preset operation conditions, the operation instruction is output to control the movable actuator to perform the operation;

[0103] When the secondary laser line comparison result does not meet the preset working conditions, the step of outputting the posture adjustment instruction to control the movable actuator to adjust the posture is returned until the latest secondary laser line comparison result meets the preset working conditions, and the output working instruction is used to control the movable actuator to perform the operation.

[0104] Specifically, after the posture adjustment, the control module will again output the laser line comparison instruction to the vision module, and the vision module will feedback the secondary laser line comparison result. When the secondary laser line comparison result meets the preset operating conditions, the control module outputs the operating instruction to control the movable actuator to perform the operation. When the secondary laser line comparison result does not meet the preset operating conditions, the control module will return to the step of outputting the posture adjustment instruction to control the movable actuator to perform the posture adjustment, until the latest secondary laser line comparison result meets the preset operating conditions, and outputs the operating instruction to control the movable actuator to perform the operation.

[0105] In one embodiment, the method further comprises:

[0106] Also includes:

[0107] The control module obtains the inclination angle of the movable actuator relative to the horizontal plane, and controls the movable actuator to adjust the posture according to the inclination angle until the inclination angle is less than a preset inclination angle threshold.

[0108] Specifically, the control module also outputs a tilt angle acquisition instruction to the tilt angle sensor, uses the tilt angle sensor to obtain the tilt angle of the movable actuator relative to the horizontal plane, and controls the movable actuator to adjust the posture according to the tilt angle until the tilt angle is less than a preset tilt angle threshold. In one embodiment, the control module is also used to perform teaching before starting the operation.

[0109] Specifically, the teaching process diagram can be as follows Figure 7 During teaching, the laser module emits a teaching laser line to project onto the tilted projection film, the control module sends a teaching instruction TO to the visual module in the visual system, the visual module collects the real-time teaching laser line projection image of the teaching laser line projected onto the tilted projection film, extracts the teaching laser line image based on the real-time teaching laser line projection image, obtains the teaching laser line image and stores it, and sends a completion prompt to the control module to complete the teaching.

[0110] In one embodiment, Figure 8 As shown, the present application also provides an application scenario, and the application scenario applies the above-mentioned construction robot. Specifically, the construction robot in the present application can be used to build walls to eliminate the accumulated errors in the current building process and achieve a good building effect. The application of the construction robot in this application scenario is as follows:

[0111] like Fig. 9 As shown, the laser module in the vision system is fixedly set at the lower left corner of the wall to be built. During teaching, the laser module can emit a teaching laser line parallel to the guide wall surface. The control module outputs a clamping instruction to the robot arm, so that the robot arm clamps the bricks to be built and moves the bricks to be built to the position to be laid. The end of the robot arm is leveled by the inclination sensor to adjust the position of the bricks to be built. At this time, the teaching laser line will be projected on the inclined projection film, and the brick surface of the brick to be built coincides with the guide wall surface. The control module outputs a teaching instruction to the vision module, and the vision module collects the real-time teaching laser line projection image of the teaching laser line projected on the inclined projection film, extracts the teaching laser line image from the real-time teaching laser line projection image, and obtains the teaching laser line image to complete the teaching.

[0112] It should be noted that before teaching and operation, the visual module in the visual system needs to be calibrated. The calibration of the visual module here mainly refers to the calibration of the acquisition component in the visual module. In this embodiment, the acquisition component can be a camera, such as Fig.10As shown, the visual module may specifically include a camera calibration module, a feature positioning module, a data storage module, an angle and offset calculation module, and a TCP (Transmission Control Protocol) / DDS (Direct Digital Synthesis) communication module. When calibrating, the control module moves the movable actuator to a preset displacement feature point, obtains the device movement coordinates corresponding to the preset displacement feature point, outputs the device movement coordinates and the calibration instructions to the visual module, and the visual module controls the image acquisition module (i.e., the acquisition component) to acquire the calibration image corresponding to the preset displacement feature point according to the calibration instructions. The camera calibration module and the feature positioning module obtain the calibration matrix according to the pixel coordinates of the preset displacement feature point in the calibration image and the device movement coordinates. The data storage module stores the calibration matrix. During operation, the angle and offset calculation module in the visual module obtains the offset angle and offset (i.e., the laser line comparison result) according to the calibration matrix, the real-time projection image, and the preset teaching laser line image, and feeds it back to the control module.

[0113] Specifically, the visual system can be a visual box integrated with an inclined projection film, a collection component, etc. The visual box can be installed at the clamping position of the end of the mechanical arm of the construction robot through a bracket to form a hand-eye device, or installed on the robot chassis. Furthermore, when the visual box is installed at the clamping position of the end of the mechanical arm of the construction robot, it is arranged perpendicular to the clamping position. Taking into account the large distance difference between the highest position of masonry and the lowest position of masonry in the actual masonry process, for example, when the highest position of masonry may be 3150 mm, the lowest position of masonry may be only 200 mm. In order to ensure accurate and clear imaging, the inclined projection film in the visual box in this embodiment, the inclined structure angle needs to be controlled within a certain range and cannot be too large. Preferably, the size of the inclined structure can be 170 (150) * 115 * 70, in mm (millimeter). The following is an explanation of the visual guidance in the masonry process using the visual box and the laser module:

[0114] After completing the teaching, the operation can be carried out. When operating, the construction robot will first stack the bottom layer of bricks in the preset bottom waiting area. As the number of stacked bricks increases, it will gradually turn to stacking higher layer of bricks in the preset high-position waiting area. Among them, the preset high-position waiting area is relative to the preset high working position. The preset high working position can be set as needed. For example, the high working position can specifically refer to a position whose distance from the ground is greater than a preset high distance threshold.

[0115] like Fig.11As shown, when the construction robot is stacking bricks on the bottom layer, the visual box is in a relatively low position (i.e., the visual box is in a preset bottom working position), and its lower end is lower than the horizontal laser line in the laser surface. In this embodiment, by setting the inclined projection film to an inclined slope from top to bottom, the horizontal laser line can be located between the upper and lower edges of the inclined projection film, and then the visual box can be used to collect the real-time projection image, compare the horizontal laser line image in the real-time projection image with the preset teaching laser line image, determine the offset angle and offset, and adjust the position of the end gripper of the robot arm according to the offset angle and offset. When the offset angle and offset are less than the preset threshold, masonry is carried out. Further, the number of inclined inclined projection films can be two, which are arranged symmetrically on the left and right, so that the laser line can be projected onto the visual box from the first direction or from the second direction, thereby increasing the flexibility of the installation position of the laser module. Among them, the threshold value of the offset can be specifically 0.5mm for the straight-line distance, and the threshold value of the offset angle can be specifically 0.1°, so as to ensure that the accuracy of bricklaying meets the requirements of 5 horizontal and 5 vertical.

[0116] like Fig.12 As shown in the figure, when the construction robot is stacking high-rise bricks, the vision box is in a relatively high position (that is, the vision box is in a high working position), and the relative position of the vision box and the laser module is shown in the figure. The real-time laser line imaging in the vision box, that is, the real-time laser line image is Fig.12 By comparing the line segment with the preset teaching laser line image, the offset angle and offset amount can be determined. The position and posture of the gripper at the end of the robot arm can be adjusted according to the offset angle and offset amount. When the offset angle and offset amount are less than the preset threshold value, masonry is carried out. It should be noted here that due to the distance from the laser module, the real-time projection image of the inclined projection film on the farther side is blurred relative to the real-time projection image of the inclined projection film on the nearer side. Therefore, the offset amount and offset angle between the real-time laser line image and the teaching laser line image in the real-time projection image of the inclined projection film on the nearer side are taken as the effective position and posture data.

[0117] It should be understood that, although the steps in each flow chart involved in the above-described embodiment are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a portion of the steps in each flow chart involved in the above-described embodiment may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0118] For the specific definition of the visual guidance method, please refer to the definition of the visual system and the visual device above, which will not be repeated here. The above control module can be implemented in whole or in part by software, hardware and a combination thereof. The above control module can be embedded in or independent of the processor in the computer device in the form of hardware, or it can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the above corresponding operations.

[0119] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0120] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A visual system is used for calibrating the position of a movable actuator of a construction robot in a construction space, wherein the construction range of the movable actuator includes a preset bottom working position, and the bottom working position corresponds to a preset bottom waiting area in the construction space. It is characterized in that The visual system comprises: A laser module, fixed in the construction space and outside the construction range of the movable actuator, the laser module is used to emit a laser beam, and the laser beam includes a horizontal laser line; A visual module is movably connected to the movable actuator, and the visual module includes: An inclined projection film, facing the ground of the construction space and arranged along an inclination relative to a horizontal direction, for imaging a projection of the horizontal laser line; A collection component, located on a side of the inclined projection film away from the ground of the construction space, for photographing the projection of the horizontal laser line; When the movable actuator is located at the bottom working position, at least part of the horizontal height of the inclined projection film is higher than the horizontal laser line, the upper edge of the inclined projection film is higher than the horizontal laser line, and the lower edge of the inclined projection film is lower than the horizontal laser line, so that the horizontal laser line is projected onto the inclined projection film.

2. The system according to claim 1, It is characterized in that The number of the inclined projection films is two, and the two inclined projection films are symmetrically arranged with respect to a plane along the vertical direction as a reference, and the distance between the top ends of the inclined projection films is greater than the distance between the bottom ends.

3. A visual device, It is characterized in that Comprising a construction robot and the visual system according to any one of claims 1 to 2; The construction robot comprises: A movable actuator for performing operations in the construction space; The control module is communicatively connected to the visual module and the movable actuator, and outputs action instructions to guide the action of the movable actuator according to the real-time projection image captured by the acquisition component.

4. The device according to claim 3, It is characterized in that The construction robot is a masonry robot, the movable actuator is a masonry robot arm, the end of the masonry robot arm is provided with a clamp, the clamp is used to take and place bricks, and the visual module is installed on the clamp.

5. The device according to claim 3, It is characterized in that The construction robot further comprises an inclination sensor, which is connected to the movable actuator and is used to collect an inclination angle of the movable actuator relative to a horizontal plane.

6. A visual guidance method, applied to the visual device according to any one of claims 3 to 5, It is characterized in that The method comprises: The laser module emits a laser beam including a horizontal laser line; When the movable actuator is located at the bottom working position, the control module outputs a laser line comparison instruction to the vision module; The visual module captures the real-time projection image of the horizontal laser line imaged on the inclined projection film through the acquisition component according to the laser line comparison instruction, and feeds back the laser line comparison result to the control module according to the real-time projection image; The control module outputs an action instruction according to the laser line comparison result to guide the action of the movable actuator.

7. The method according to claim 6, It is characterized in that The control module outputs an action instruction according to the laser line comparison result to guide the action of the movable actuator, including: When the laser line comparison result meets the preset operation condition, an operation instruction is output to control the movable actuator to perform the operation, and the preset operation condition is that the angle and offset between the horizontal laser line image in the real-time projection image and the preset teaching laser line image are both less than the corresponding preset operation threshold; When the laser line comparison result does not meet the preset operating conditions, a posture adjustment instruction is output to control the movable actuator to adjust the posture.

8. The method according to claim 7, It is characterized in that When the laser line comparison result does not meet the preset working condition, after outputting the posture adjustment instruction to control the movable actuator to adjust the posture, the method further includes: Obtain secondary laser line comparison results; When the secondary laser line comparison result meets the preset operation condition, outputting an operation instruction to control the movable actuator to perform the operation; When the secondary laser line comparison result does not meet the preset operating conditions, the step of outputting the posture adjustment instruction to control the movable actuator to perform posture adjustment is returned until the latest secondary laser line comparison result meets the preset operating conditions, and the output operating instruction controls the movable actuator to perform the operation.

9. The method according to claim 6, It is characterized in that Also includes: The control module obtains the tilt angle of the movable actuator relative to the horizontal plane, and controls the movable actuator to adjust its position according to the tilt angle until the tilt angle is less than a preset tilt angle threshold.

10. The method according to claim 6, It is characterized in that Also includes: The laser module emits a teaching laser line to project onto the inclined projection film, the control module sends a teaching instruction to the visual module, the visual module collects a real-time teaching laser line projection image of the teaching laser line projected onto the inclined projection film, extracts the teaching laser line image based on the real-time teaching laser line projection image, obtains the teaching laser line image and stores it.