Curtain wall installation method, device and equipment and storage medium

Abstract

The invention relates to the technical field of curtain wall installation, and discloses a curtain wall installation method, device and equipment and a storage medium, the method comprises the steps that a first point cloud and a second point cloud are obtained, the first point cloud is obtained by shooting a curtain wall plate through a 3D color depth camera installed on one side of an end effector, and the second point cloud is obtained by shooting a curtain wall plate through a 3D color depth camera installed on one side of the end effector; the second point cloud is obtained by shooting the curtain wall plate by a 3D color depth camera mounted on the other side of the end effector; performing point cloud fusion based on the first point cloud and the second point cloud to obtain a curtain wall plate point cloud; based on the curtain wall plate point cloud, controlling the end effector to pre-align the curtain wall plates; and controlling the end effector to mount the pre-aligned curtain wall plates through teaching playback or visual guidance. Shielding points and visual blind points in the curtain wall plate installation process can be reduced, and the installation efficiency, operability and quality are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of curtain wall installation, and in particular to a curtain wall installation method, device, equipment and storage medium. BACKGROUND

[0002] In the application scenarios of high altitude, high load and complex construction environment, the installation buckle area presented in the camera field of view of the curtain wall installation robot during the operation process is narrow, which makes the detection of the installation object and the installation target very difficult, resulting in problems such as shielding points and visual blind spots. SUMMARY

[0003] Therefore, it is necessary to propose a curtain wall installation method, device, equipment and storage medium for the technical problem of shielding points and visual blind spots in the camera field of view of the curtain wall installation robot during the operation process.

[0004] In a first aspect, a curtain wall installation method is provided, and the method comprises:

[0005] obtaining a first point cloud and a second point cloud, wherein the first point cloud is obtained by a 3D color depth camera installed on one side of an end effector shooting a curtain wall panel, and the second point cloud is obtained by a 3D color depth camera installed on the other side of the end effector shooting the curtain wall panel;

[0006] performing point cloud fusion based on the first point cloud and the second point cloud to obtain a curtain wall panel point cloud;

[0007] controlling the end effector to pre-align the curtain wall panel based on the curtain wall panel point cloud;

[0008] controlling the end effector to install the pre-aligned curtain wall panel through demonstration playback or visual guidance.

[0009] In a second aspect, a curtain wall installation device is provided, and the device comprises:

[0010] In a third aspect, a computer device is provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above curtain wall installation method when executing the computer program.

[0011] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program implements the steps of the above curtain wall installation method when executed by a processor.

[0012] The curtain wall installation method provided by the application obtains a first point cloud and a second point cloud, wherein the first point cloud is obtained by a 3D color depth camera installed on one side of an end effector and shooting a curtain wall plate, the second point cloud is obtained by a 3D color depth camera installed on the other side of the end effector and shooting the curtain wall plate, then point cloud fusion is performed based on the first point cloud and the second point cloud to obtain a curtain wall plate point cloud, then the end effector is controlled to pre-align the curtain wall plate based on the curtain wall plate point cloud, finally, the end effector is controlled to install the pre-aligned curtain wall plate through demonstration playback or visual guidance. The method can reduce the shielding points and visual blind spots in the curtain wall plate installation process, and improve the installation efficiency, operability and quality. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0014] Among them:

[0015] Figure 1 It is an application environment diagram of the curtain wall installation method in an embodiment;

[0016] Figure 2 It is a flowchart of the curtain wall installation method in an embodiment;

[0017] Figure 3 It is a structure block diagram of the curtain wall installation device in an embodiment;

[0018] Figure 4 It is a structure block diagram of the computer device in an embodiment;

[0019] Figure 5 It is a structure block diagram of the computer device in another embodiment. DETAILED DESCRIPTION

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the specification herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application; the description and the drawings of the present application together with the attached claims serve to explain the features of the present application; the terms "comprise", "have" and any variations thereof in the specification and the claims of the present application and the above description of drawings are intended to cover non-exclusive inclusion; the terms "first", "second" and the like in the specification and the claims of the present application and the above description of drawings are used to distinguish different objects, not to describe a particular order.

[0021] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] The curtain wall installation method provided in this embodiment of the invention can be applied to, for example... Figure 1 In this application environment, client 110 communicates with server 120 via a network. Server 120 can receive a first point cloud and a second point cloud from client 110. The first point cloud is obtained by a 3D color depth camera mounted on one side of the end effector, capturing images of the curtain wall panel. The second point cloud is obtained by a 3D color depth camera mounted on the other side of the end effector, capturing images of the curtain wall panel. Point cloud fusion is then performed based on the first and second point clouds to obtain a curtain wall panel point cloud. Next, based on the curtain wall panel point cloud, the end effector is controlled to pre-align the curtain wall panel. Finally, through teaching playback or visual guidance, the end effector is controlled to install the pre-aligned curtain wall panel. This reduces obstructed points and blind spots during curtain wall panel installation, improving installation efficiency, operability, and quality. Client 110 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. Server 120 can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0024] Please see Figure 2 As shown, Figure 2 A schematic flowchart of a curtain wall installation method according to an embodiment of the present invention includes the following steps:

[0025] Step S101: Obtain a first point cloud and a second point cloud, wherein the first point cloud is obtained by a 3D color depth camera installed on one side of the end effector and capturing images of the curtain wall panel, and the second point cloud is obtained by a 3D color depth camera installed on the other side of the end effector and capturing images of the curtain wall panel.

[0026] In this embodiment, due to issues such as obstructions and blind spots in the target itself and its installation location, a single-angle shot cannot acquire comprehensive 3D data. Therefore, it is necessary to use a multi-degree-of-freedom binocular vision calibration based on a 3D color depth camera. Two identical 3D color depth cameras are fixed on the end effector. A mature camera calibration method is directly adopted, and the internal and external parameters of the two 3D color depth cameras are calibrated by calculating point coordinates. The two color depth cameras take pictures of the curtain wall panel from different angles, acquire point cloud data, and record the current position. The cameras are fixed to the end of the actuator, and moving the cameras can acquire the end position of the end effector.

[0027] Before the steps of acquiring the first curtain wall panel image and the second curtain wall panel image, the following are included:

[0028] Step A: Determine the intrinsic and extrinsic parameters of the 3D color depth camera. The intrinsic parameters include focal length, principal point coordinates, and distortion coefficients. The extrinsic parameters include the camera's position and orientation relative to the world coordinate system.

[0029] Step B: Calculate the uniform transformation matrix between the 3D color depth camera and the end effector through hand-eye calibration, wherein the uniform transformation matrix describes the relative position and orientation between the camera coordinate system of the 3D color depth camera and the robot coordinate system of the end effector.

[0030] In this embodiment, the pre-operation calibration of the end effector first includes vision sensor calibration and hand-eye calibration. Vision sensor calibration addresses the intrinsic and extrinsic calibration parameters of the 3D color depth camera and establishes the relationship between the image coordinate system and the world coordinate system. Hand-eye calibration calculates the uniform transformation matrix between the 3D color depth camera and the actuator end effector. The purpose of pre-operation calibration is to calculate the actual installation position based on the target information and calibration results when the vision sensor acquires target information.

[0031] Step S102: Perform point cloud fusion based on the first point cloud and the second point cloud to obtain the point cloud of the curtain wall panel;

[0032] In this embodiment, based on the coordinate transformation of the point cloud captured by the 3D color depth camera for each shot, the captured point cloud is transformed to the camera coordinate system. Under this unified camera coordinate system, the point cloud is fused using a point cloud fusion algorithm. This, along with model stitching, denoising, and optimization, allows the curtain wall panel to be scaled proportionally in virtual space, ultimately generating a fully unobstructed point cloud of the curtain wall panel. After point cloud fusion, a clear and complete 3D real-world color point cloud model of the curtain wall panel can be obtained. The location of the target work point can be easily obtained by specifying or matching the data.

[0033] In one embodiment, the step of fusing the first point cloud and the second point cloud to obtain the point cloud of the curtain wall panel includes:

[0034] Step S1021: Perform preprocessing operations on the first point cloud and the second point cloud, the preprocessing including denoising, enhancement and filtering;

[0035] Step S1022: Extract feature points from the preprocessed first point cloud and second point cloud respectively. The feature points include corner points and edges.

[0036] Step S1023: Using a stereo matching algorithm, calculate the positional deviation between corresponding feature points in the first point cloud and the second point cloud to obtain a disparity map;

[0037] Step S1023: Based on the disparity map, and combined with the intrinsic and extrinsic parameters of the 3D color depth camera, a 3D reconstruction algorithm is used to perform 3D reconstruction of the scene to obtain the point cloud of the curtain wall panel.

[0038] Step S103: Based on the point cloud of the curtain wall panel, control the end effector to pre-align the curtain wall panel;

[0039] As an example, by using the point cloud of the curtain wall panel, the end effector can identify the approximate installation position and control the end effector to move to the position where the curtain wall panel is to be installed, thus achieving pre-alignment.

[0040] Step S104: Control the end effector to install the pre-aligned curtain wall panels through teaching playback or visual guidance.

[0041] In this embodiment, the grasping / installation positioning mainly includes teaching playback and visual guidance. Teaching playback requires human-machine collaboration to control the robotic arm to move the end effector to the vicinity of the installation position. After determining the installation frame, teaching playback with repetitive predetermined actions is performed. Visual guidance can perceive and respond to obstacles in real time. Recognition, calculation, and guidance control are the basic stages of visual positioning. This method has high positioning accuracy and achieves autonomous positioning.

[0042] As an example, the actual spatial relationships between target locations, end effectors, and surrounding objects can be obtained from a 3D solid space model, enabling real-time perception of the current operational status using visual guidance. Specifically, the point cloud of the curtain wall panel is processed using an octree method to obtain a tree-like data structure describing the 3D space, quickly determining the position of objects in the 3D scene or identifying surrounding obstacles. Each point cloud node of the curtain wall panel, after octree processing, is represented as a volume element of a cube. Each node has 8 child nodes, and the sum of the volume elements represented by the 8 child nodes equals the volume of the parent node. Finally, obstacles are represented as green cubes in the 3D solid space of the end effector.

[0043] Finally, trajectory tracking is performed based on the three-dimensional solid space. For trajectory tracking, the positional deviation information between the current point and the reference point must be continuously calculated. The deviation signal is obtained through the controller, and several control strategies can be used, such as PID control, fuzzy control, and iterative learning control, to correct the end effector position to the correct installation position.

[0044] The curtain wall installation method proposed in this embodiment acquires a first point cloud and a second point cloud. The first point cloud is obtained by photographing the curtain wall panel with a 3D color depth camera mounted on one side of the end effector, and the second point cloud is obtained by photographing the curtain wall panel with a 3D color depth camera mounted on the other side of the end effector. Then, point cloud fusion is performed based on the first and second point clouds to obtain a curtain wall panel point cloud. Next, based on the curtain wall panel point cloud, the end effector is controlled to pre-align the curtain wall panel. Finally, through teaching playback or visual guidance, the end effector is controlled to install the pre-aligned curtain wall panel. This method can reduce obstructed points and blind spots during the installation process of curtain wall panels, improving installation efficiency, operability, and quality.

[0045] Please see Figure 3 As shown, in one embodiment, a curtain wall mounting device is provided, the device comprising:

[0046] The acquisition module 10 is used to acquire a first point cloud and a second point cloud. The first point cloud is obtained by a 3D color depth camera installed on one side of the end effector and capturing images of the curtain wall panel. The second point cloud is obtained by a 3D color depth camera installed on the other side of the end effector and capturing images of the curtain wall panel.

[0047] Point cloud module 20 is used to perform point cloud fusion based on the first point cloud and the second point cloud to obtain the point cloud of the curtain wall panel;

[0048] The pre-alignment module 30 is used to control the end effector to pre-align the curtain wall panel based on the point cloud of the curtain wall panel.

[0049] Installation module 40 is used to control the end effector to install the pre-aligned curtain wall panels through teaching playback or visual guidance.

[0050] The curtain wall installation device is also used to: determine the intrinsic and extrinsic parameters of a 3D color depth camera, wherein the intrinsic parameters include focal length, principal point coordinates, and distortion coefficients, and the extrinsic parameters include the position and orientation of the camera relative to the world coordinate system;

[0051] Through hand-eye calibration, a uniform transformation matrix between the 3D color depth camera and the end effector is calculated, wherein the uniform transformation matrix describes the relative position and orientation between the camera coordinate system of the 3D color depth camera and the robot coordinate system of the end effector.

[0052] Point cloud module 20 is also used to perform preprocessing operations on the first point cloud and the second point cloud, the preprocessing including denoising, enhancement and filtering;

[0053] Feature points are extracted from the preprocessed first point cloud and second point cloud, respectively. The feature points include corner points and edges.

[0054] Using a stereo matching algorithm, the positional deviation between corresponding feature points in the first point cloud and the second point cloud is calculated to obtain a disparity map;

[0055] Based on the parallax map, and combined with the intrinsic and extrinsic parameters of the 3D color depth camera, a 3D reconstruction algorithm is used to reconstruct the scene in 3D, resulting in a point cloud of the curtain wall panel.

[0056] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a curtain wall installation method on the server side.

[0057] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the client side of a curtain wall installation method.

[0058] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, performs the following steps:

[0059] A first point cloud and a second point cloud are obtained. The first point cloud is obtained by taking a picture of the curtain wall panel with a 3D color depth camera installed on one side of the end effector, and the second point cloud is obtained by taking a picture of the curtain wall panel with a 3D color depth camera installed on the other side of the end effector.

[0060] Point cloud fusion is performed based on the first point cloud and the second point cloud to obtain the point cloud of the curtain wall panel.

[0061] Based on the point cloud of the curtain wall panels, the end effector is controlled to pre-align the curtain wall panels;

[0062] The end effector is controlled to install the pre-aligned curtain wall panels through teaching playback or visual guidance.

[0063] In one embodiment, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, performs the following steps:

[0064] A first point cloud and a second point cloud are obtained. The first point cloud is obtained by taking a picture of the curtain wall panel with a 3D color depth camera installed on one side of the end effector, and the second point cloud is obtained by taking a picture of the curtain wall panel with a 3D color depth camera installed on the other side of the end effector.

[0065] Point cloud fusion is performed based on the first point cloud and the second point cloud to obtain the point cloud of the curtain wall panel.

[0066] Based on the point cloud of the curtain wall panels, the end effector is controlled to pre-align the curtain wall panels;

[0067] The end effector is controlled to install the pre-aligned curtain wall panels through teaching playback or visual guidance.

[0068] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0069] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAM bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0070] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0071] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method of installing a curtain wall, characterized by, The curtain wall installation method comprises: obtaining a first point cloud and a second point cloud, wherein the first point cloud is obtained by a 3D color depth camera installed on one side of an end effector shooting a curtain wall panel, and the second point cloud is obtained by a 3D color depth camera installed on the other side of the end effector shooting the curtain wall panel; performing point cloud fusion based on the first point cloud and the second point cloud to obtain a curtain wall panel point cloud; controlling the end effector to pre-align the curtain wall panel based on the curtain wall panel point cloud; controlling the end effector to install the pre-aligned curtain wall panel through teaching playback or visual guidance.

2. The curtain wall installation method according to claim 1, wherein, Before the step of obtaining the first curtain wall panel image and the second curtain wall panel image, the method comprises: determining internal parameters and external parameters of the 3D color depth camera, wherein the internal parameters comprise focal length, principal point coordinates, distortion coefficients, and the external parameters comprise the position and attitude of the camera relative to the world coordinate system; calculating a uniform transformation matrix between the 3D color depth camera and the end effector through hand-eye calibration, wherein the uniform transformation matrix describes the relative position and attitude between the camera coordinate system of the 3D color depth camera and the robot coordinate system of the end effector.

3. The curtain wall installation method according to claim 1, wherein, The step of performing point cloud fusion based on the first point cloud and the second point cloud to obtain a curtain wall panel point cloud comprises: performing preprocessing operations on the first point cloud and the second point cloud, wherein the preprocessing comprises denoising, enhancement, and filtering; extracting feature points from the preprocessed first point cloud and the preprocessed second point cloud, wherein the feature points comprise corner points and edges; calculating the position deviation between corresponding feature points in the first point cloud and the second point cloud using a stereo matching algorithm to obtain a disparity map; performing three-dimensional reconstruction of the scene using a three-dimensional reconstruction algorithm based on the disparity map, the internal parameters of the 3D color depth camera, and the external parameters to obtain the curtain wall panel point cloud.

4. A curtain wall mounting device, characterized by, The curtain wall installation device comprises: an obtaining module configured to obtain a first point cloud and a second point cloud, wherein the first point cloud is obtained by a 3D color depth camera installed on one side of an end effector shooting a curtain wall panel, and the second point cloud is obtained by a 3D color depth camera installed on the other side of the end effector shooting the curtain wall panel; a point cloud module configured to perform point cloud fusion based on the first point cloud and the second point cloud to obtain a curtain wall panel point cloud; a pre-alignment module configured to control the end effector to pre-align the curtain wall panel based on the curtain wall panel point cloud; an installation module configured to control the end effector to install the pre-aligned curtain wall panel through teaching playback or visual guidance.

5. A curtain wall installation robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the curtain wall installation method according to any one of claims 1 to 3.

6. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 5. The computer program is executed by the processor to implement the steps of the curtain wall installation method according to any one of claims 1 to 3.

Images