Humanoid robot intelligent decoration construction system and method based on multi-modal perception

Abstract

The invention discloses a humanoid robot intelligent decoration construction system and method based on multi-modal sensing, and the system obtains the geometric data of a construction surface in real time through multi-modal sensing, obtains the posture data of a robot through an IMU inertial measurement unit, facilitates the data calibration, guarantees the modeling precision of a building model, and improves the construction efficiency. Real-time feedback of construction progress and construction defects is carried out; the intelligent decision-making module is combined to dynamically adjust the movement and construction operation track of the robot; by means of an end effector of the flexible mechanical arm, high-precision multi-mode building construction operation is achieved; and the collaborative operation module can realize multi-robot construction operation task scheduling and progress management, so that the building comprehensive construction efficiency is improved. Compared with a traditional decoration construction mode, the construction efficiency, the construction precision and the robot operation and moving stability are remarkably improved, autonomous operation of a complex curved surface is supported, and the construction consistency and the continuity of multi-mode construction operation are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of construction robots, and in particular to a humanoid robot intelligent decoration construction system and method based on multimodal perception. Background Art

[0002] Building decoration construction includes construction modes and processes such as wall grinding and leveling, tile laying, and paint painting. Traditional building decoration relies on manual operation, which has problems such as low efficiency, poor precision, and high risk of high-altitude operations. Existing construction robots for wall construction are mostly limited to operations on flat and conventional structured buildings, and cannot adapt to complex curved surfaces. The deviation of the robot's movement trajectory and the deviation of the construction position caused by its own posture make the construction accuracy insufficient, and lack the ability to coordinate multiple tasks and construction modes. In addition, the construction quality relies on manual inspection, which is prone to missed inspections and misjudgments, and it is impossible to make real-time judgments, which makes the construction deviations gradually accumulate and affects the construction quality. Summary of the invention

[0003] The purpose of the present invention is to provide a humanoid robot intelligent decoration construction system and method based on multimodal perception in order to solve the problems that the existing construction methods are difficult to adapt to complex curved surfaces, the construction accuracy is insufficient, the multi-tasking and construction mode coordination capabilities are lacking, the construction quality cannot be accurately and real-time determined, the construction deviations gradually accumulate, and the construction quality and construction efficiency are affected.

[0004] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solution: a humanoid robot intelligent decoration construction system based on multimodal perception, comprising:

[0005] A humanoid robot body, which is equipped with a bionic bipedal walking mechanism, a flexible mechanical arm, and an end effector docked at the end of the flexible mechanical arm;

[0006] A multimodal perception module, which is equipped with a laser radar, a visual camera, an IMU inertial measurement unit and an environmental modeling module. The laser radar and the visual camera respectively scan and generate real-time point cloud data and image data corresponding to the construction building entity. The IMU inertial measurement unit is used to solve the posture of the humanoid robot body, form posture data and cooperate with the environmental modeling module to model the building model;

[0007] An intelligent decision-making module generates an operation path according to the construction drawings and the building model, and dynamically corrects the motion trajectory of the humanoid robot body;

[0008] The collaborative operation module is synchronized with the cloud BIM model and the multi-robot scheduling system through the communication module. The cloud BIM model is associated with the intelligent decision-making module to support multi-robot collaborative obstacle avoidance and operation task allocation.

[0009] As a further description of the above technical solution:

[0010] The bionic biped walking mechanism is provided with a magnetically adsorbed foot and a dynamic balance module, and the end effector comprises a vacuum suction cup, a high-pressure nozzle, a grinding tool and other decoration construction devices respectively used for laying tiles on the building surface, spraying paint, and grinding the wall.

[0011] As a further description of the above technical solution:

[0012] A construction quality detection unit is also arranged on the humanoid robot body, and the construction quality detection unit includes an infrared thermal imager and an acoustic emission sensor.

[0013] As a further description of the above technical solution:

[0014] The laser radar is an integrated solid-state laser radar, the visual camera is an RGB-D camera, and the multimodal perception module fuses the point cloud data and image data through Kalman filtering to construct a construction surface model of the building model with sub-millimeter accuracy in real time, and mark the operational defect areas of hollows and cracks.

[0015] As a further description of the above technical solution:

[0016] A force / torque sensor is also provided on the multimodal perception module, and a corresponding motion control module is also provided on the intelligent decision-making module, so as to dynamically correct the motion trajectory of the humanoid robot body based on a force control algorithm in cooperation with the intelligent decision-making module.

[0017] As a further description of the above technical solution:

[0018] The intelligent decision-making module generates a collision-free operation path based on the dynamic priority scheduling algorithm of the task planning engine, the RRT* fast expansion random tree algorithm and obstacle avoidance strategy of the path planning module, combined with the curvature of the construction surface of the building model, and the path smoothness error is less than 0.5mm.

[0019] As a further description of the above technical solution:

[0020] The communication module includes a 5G-MEC edge computing module and a human-computer interaction module, and the human-computer interaction module includes an AR remote control module and a voice interaction module.

[0021] As a further description of the above technical solution:

[0022] The collaborative operation module supports multi-robot collaborative obstacle avoidance and operation task allocation based on the dynamic priority algorithm and conflict resolution mechanism of the multi-robot scheduling system.

[0023] A humanoid robot intelligent decoration construction method based on multimodal perception comprises the following steps:

[0024] S1 data input: acquiring point cloud data, image data, and posture data in real time through the multimodal perception module;

[0025] S2 data preprocessing: filtering the point cloud data, extracting features from the image data, and calibrating the data using the posture data;

[0026] S3 data fusion: Based on the calibrated real-time point cloud data and image data, the building model is modeled through the environmental modeling module to generate a high-precision three-dimensional construction surface model;

[0027] S4 construction operation path generation: generating a collision-free operation path of the humanoid robot body through the intelligent decision-making module;

[0028] S5 Collaborative work: Assign tasks to multiple robots and plan the work sequence.

[0029] In summary, due to the adoption of the above technical solution, the present invention has the following beneficial effects compared with the prior art:

[0030] 1. The humanoid robot intelligent decoration construction system and method of the present invention are suitable for scenes such as wall decoration, tile laying and paint spraying at construction sites. The system includes a humanoid robot body, a multimodal perception module, an intelligent decision-making module and a collaborative operation module. Through the three-dimensional environment modeling technology that integrates laser radar and visual camera, the geometric data of the construction surface is acquired in real time, and the robot posture data is acquired through the IMU inertial measurement unit to facilitate data calibration and ensure the accuracy of building model modeling; combined with the adaptive path planning algorithm of the intelligent decision-making module, the robot movement and construction operation trajectory are dynamically adjusted; the end effector of the flexible mechanical arm is used to achieve high-precision multi-mode construction operations; the collaborative operation module can realize the construction operation task scheduling and progress management of multiple robots, and improve the comprehensive construction efficiency of the building. Compared with the traditional decoration construction method, the construction efficiency, construction accuracy, robot operation, and movement stability of the present invention are significantly improved, and it supports autonomous operation of complex curved surfaces, reduces labor costs, and improves construction consistency and the continuity of multi-mode construction operations.

[0031] 2. Through multimodal construction surface perception, real-time acquisition of building models, and IMU inertial measurement units, high-precision feedback of construction progress and corresponding construction defects can be achieved to facilitate dynamic correction of subsequent construction paths; the trajectory of the robotic arm can be dynamically adjusted based on the curvature of the construction surface to support operations in unstructured environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 This is a system architecture diagram of a humanoid robot intelligent decoration construction system based on multimodal perception.

[0034] Figure 2 This is a partial flow chart of a humanoid robot intelligent decoration construction method based on multimodal perception. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0036] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] It should be noted that similar reference numerals and letters refer to similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in the subsequent drawings.

[0038] In the description of the embodiments of the present invention, it should be noted that the terms "upper", "inner", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, or are directions or positional relationships in which the inventive product is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.

[0039] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection, an indirect connection through an intermediate medium, or the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0040] See also Figure 1-2 The present invention provides a technical solution: a humanoid robot intelligent decoration construction system based on multimodal perception, comprising:

[0041] A humanoid robot body, which is equipped with a bionic bipedal walking mechanism, a flexible mechanical arm, and an end effector docked at the end of the flexible mechanical arm;

[0042] A multimodal perception module, which is equipped with a laser radar, a visual camera, an IMU inertial measurement unit and an environmental modeling module. The laser radar and the visual camera respectively scan and generate real-time point cloud data and image data corresponding to the construction building entity. The IMU inertial measurement unit is used to solve the posture of the humanoid robot body, form posture data and cooperate with the environmental modeling module to model the building model;

[0043] Among them, the IMU inertial measurement unit is a device for measuring the three-axis attitude angle (or angular rate) and acceleration of an object. It contains three single-axis accelerometers and three single-axis gyroscopes. The accelerometer detects the acceleration signal of the object in the independent three axes of the carrier coordinate system, and the gyroscope detects the angular velocity signal of the carrier relative to the navigation coordinate system, measures the angular velocity and acceleration of the object in three-dimensional space, and solves the attitude of the object. The construction surface model formed by the point cloud data and image data is calibrated with the coordinate system based on this attitude data to ensure the accuracy of the subsequent bionic bipedal walking mechanism and the position adjustment of the end effector by the flexible robotic arm, thereby improving the construction quality;

[0044] An intelligent decision-making module generates an operation path according to the construction drawings and the building model, and dynamically corrects the motion trajectory of the humanoid robot body; based on the construction drawings, the construction path is simulated on the building model, and the deep learning module and the reinforcement learning module are used to analyze the path with the highest construction efficiency;

[0045] The collaborative operation module is synchronized with the cloud BIM model and the multi-robot scheduling system through the communication module. The cloud BIM model is associated with the intelligent decision-making module to support multi-robot collaborative obstacle avoidance and task allocation. The building model formed by modeling is transmitted to the cloud BIM model platform in real time to realize the visualization of the construction operation process, and it is convenient to realize the intelligent and precise control of a single robot and the efficient collaborative operation of multiple robots through the intelligent decision-making module and the multi-robot scheduling system.

[0046] The humanoid robot intelligent decoration construction system and method of the present invention are suitable for scenes such as wall decoration, tile laying and paint spraying at construction sites. The system includes a humanoid robot body, a multimodal perception module, an intelligent decision-making module and a collaborative operation module. Through the three-dimensional environment modeling technology that integrates laser radar and visual camera, the geometric data of the construction surface is acquired in real time, and the robot posture data is acquired through the IMU inertial measurement unit to facilitate data calibration and ensure the accuracy of building model modeling; combined with the adaptive path planning algorithm of the intelligent decision-making module, the robot movement and construction operation trajectory are dynamically adjusted; the end effector of the flexible mechanical arm is used to realize high-precision multi-mode construction operations; the collaborative operation module can realize the construction operation task scheduling and progress management of multiple robots, and improve the comprehensive construction efficiency of the building. Compared with the traditional decoration construction method, the construction efficiency, construction accuracy, robot operation, and movement stability are significantly improved, and it supports autonomous operation of complex curved surfaces, reduces labor costs, and improves construction consistency and the continuity of multi-mode construction operations.

[0047] In order to improve the stability of the humanoid robot when moving and the robotic arm when constructing, the following design is made based on this demand: a magnetically adsorbed foot and a dynamic balance module are arranged on the bionic bipedal walking mechanism. When the two feet walk alternately, the foot in contact with the ground can be adsorbed on the ground, and the center of gravity of the robot can be adjusted through the dynamic balance module to improve its stability during movement and the swing and construction of the robotic arm.

[0048] The end effector includes a vacuum suction cup, a high-pressure nozzle, and a grinding tool, which are used for laying tiles on the building surface, spraying paint, and grinding the wall. In addition to the above-mentioned conventional building wall operation processes, it can also be equipped with operation tools corresponding to other operation processes.

[0049] The humanoid robot body is also provided with a construction quality detection unit, which includes an infrared thermal imager and an acoustic emission sensor. When the construction work phase is completed, the infrared thermal imager and the acoustic emission sensor are used to perform infrared thermal imaging and appearance detection of the construction surface, and the AI ​​deep learning algorithm is used to automatically evaluate the construction quality and generate phase and overall construction reports.

[0050] The laser radar is an integrated solid-state laser radar, the visual camera is an RGB-D camera, and the multimodal perception module fuses the point cloud data and image data through Kalman filtering to construct a construction surface model of the building model with sub-millimeter accuracy in real time, and mark the defective areas of hollows and cracks. Among them, the Kalman filter uses the dynamic information of the target to try to remove the influence of background noise and obtain a more accurate estimate of the target position. The acquired initial point cloud data and image data are processed to eliminate noise influence and data fusion to ensure the accuracy of data and subsequent model establishment; the operation defects existing in the previous operation process are marked in real time, and the robots of the corresponding process are controlled by the collaborative operation module to perform post-processing of the construction surface to improve the continuity of the operation and the quality of construction.

[0051] The multimodal perception module is also provided with a force / torque sensor, and the intelligent decision module is also provided with a corresponding motion control module, so as to dynamically correct the motion trajectory of the humanoid robot body based on the force control algorithm in cooperation with the intelligent decision module. Since the end effector will generate a reaction force on the robot during construction, the consideration of this influencing factor is added to the robot's motion trajectory planning, and this influencing factor is eliminated to achieve dynamic correction of the motion trajectory.

[0052] The intelligent decision-making module generates a collision-free operation path based on the dynamic priority scheduling algorithm of the task planning engine, the RRT* rapidly expanding random tree algorithm and the obstacle avoidance strategy of the path planning module, combined with the curvature of the construction surface of the building model, with a path smoothness error of less than 0.5mm. Among them, the dynamic priority scheduling algorithm and the RRT* rapidly expanding random tree algorithm are used to dynamically adjust the priorities of the construction area and multiple robots in collaborative operations based on real-time conditions such as construction progress and construction surface conditions, and the obstacle avoidance strategy avoids obstacles such as the original building and other construction robots on the moving path when the robot moves;

[0053] The communication module includes a 5G-MEC edge computing module and a human-computer interaction module, and the human-computer interaction module includes an AR remote control module and a voice interaction module. Through 5G network wireless communication, real-time transmission of construction progress and remote interactive control are carried out.

[0054] The collaborative operation module supports multi-robot collaborative obstacle avoidance and operation task allocation based on the dynamic priority algorithm and conflict resolution mechanism of the multi-robot scheduling system.

[0055] A humanoid robot intelligent decoration construction method based on multimodal perception comprises the following steps:

[0056] S1 data input: point cloud data, image data, and posture data are acquired in real time through the multimodal perception module; the robot scans the construction surface through the laser radar to generate point cloud data, and detects the flatness of the wall and the gap between tiles through the visual camera;

[0057] S2 data preprocessing: filtering the point cloud data, extracting features from the image data, and calibrating the data using the posture data;

[0058] S3 data fusion: Based on the calibrated real-time point cloud data and image data, the building model is modeled through the environmental modeling module to generate a high-precision three-dimensional construction surface model;

[0059] S4 construction operation path generation: calling BIM drawing data through the intelligent decision-making module to generate a collision-free operation path for the humanoid robot body;

[0060] During construction, the IMU inertial measurement unit is used to provide real-time feedback on the robot's posture deviation, so as to facilitate dynamic correction of the subsequent motion trajectory; after the stage operation process is completed, quality acceptance is carried out;

[0061] S5 Collaborative work: Assign tasks to multiple robots and plan the work sequence.

[0062] In summary, due to the adoption of the above technical solution, the humanoid robot intelligent decoration construction system and method based on multimodal perception in this embodiment has the following beneficial effects compared with the prior art:

[0063] 1. The humanoid robot intelligent decoration construction system and method of the present invention are suitable for scenes such as wall decoration, tile laying and paint spraying at construction sites. The system includes a humanoid robot body, a multimodal perception module, an intelligent decision-making module and a collaborative operation module. Through the three-dimensional environment modeling technology that integrates laser radar and visual camera, the geometric data of the construction surface is acquired in real time, and the robot posture data is acquired through the IMU inertial measurement unit to facilitate data calibration and ensure the accuracy of building model modeling; combined with the adaptive path planning algorithm of the intelligent decision-making module, the robot movement and construction operation trajectory are dynamically adjusted; the end effector of the flexible mechanical arm is used to achieve high-precision multi-mode construction operations; the collaborative operation module can realize the construction operation task scheduling and progress management of multiple robots, and improve the comprehensive construction efficiency of the building. Compared with the traditional decoration construction method, the construction efficiency, construction accuracy, robot operation, and movement stability of the present invention are significantly improved, and it supports autonomous operation of complex curved surfaces, reduces labor costs, and improves construction consistency and the continuity of multi-mode construction operations.

[0064] 2. Through multimodal construction surface perception, real-time acquisition of building models, and IMU inertial measurement units, high-precision feedback of construction progress and corresponding construction defects can be achieved to facilitate dynamic correction of subsequent construction paths; the trajectory of the robotic arm can be dynamically adjusted based on the curvature of the construction surface to support operations in unstructured environments.

[0065] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A humanoid robot intelligent decoration construction system based on multimodal perception, characterized in that: include: A humanoid robot body, which is equipped with a bionic bipedal walking mechanism, a flexible mechanical arm, and an end effector docked at the end of the flexible mechanical arm; A multimodal perception module, which is equipped with a laser radar, a visual camera, an IMU inertial measurement unit and an environmental modeling module. The laser radar and the visual camera respectively scan and generate real-time point cloud data and image data corresponding to the construction building entity. The IMU inertial measurement unit is used to solve the posture of the humanoid robot body, form posture data and cooperate with the environmental modeling module to model the building model; An intelligent decision-making module generates an operation path according to the construction drawings and the building model, and dynamically corrects the motion trajectory of the humanoid robot body; The collaborative operation module is synchronized with the cloud BIM model and the multi-robot scheduling system through the communication module. The cloud BIM model is associated with the intelligent decision-making module to support multi-robot collaborative obstacle avoidance and operation task allocation.

2. According to claim 1, a humanoid robot intelligent decoration construction system based on multimodal perception is characterized in that: The bionic biped walking mechanism is provided with a magnetically attracted foot and a dynamic balance module, and the end effector comprises a vacuum suction cup, a high-pressure nozzle and a grinding tool which are respectively used for laying tiles on the building surface, spraying paint and grinding the wall.

3. According to claim 1, a humanoid robot intelligent decoration construction system based on multimodal perception is characterized in that: A construction quality detection unit is also arranged on the humanoid robot body, and the construction quality detection unit includes an infrared thermal imager and an acoustic emission sensor.

4. According to claim 1, a humanoid robot intelligent decoration construction system based on multimodal perception is characterized in that: The laser radar is an integrated solid-state laser radar, the visual camera is an RGB-D camera, and the multimodal perception module fuses the point cloud data and image data through Kalman filtering to construct a construction surface model of the building model with sub-millimeter accuracy in real time, and mark the operational defect areas of hollows and cracks.

5. The humanoid robot intelligent decoration construction system based on multimodal perception according to claim 1 is characterized in that: A force / torque sensor is also provided on the multimodal perception module, and a corresponding motion control module is also provided on the intelligent decision-making module, so as to dynamically correct the motion trajectory of the humanoid robot body based on a force control algorithm in cooperation with the intelligent decision-making module.

6. The humanoid robot intelligent decoration construction system based on multimodal perception according to claim 1 is characterized in that: The intelligent decision-making module generates a collision-free operation path based on the dynamic priority scheduling algorithm of the task planning engine, the RRT* fast expansion random tree algorithm and obstacle avoidance strategy of the path planning module, combined with the curvature of the construction surface of the building model, and the path smoothness error is less than 0.5mm.

7. The humanoid robot intelligent decoration construction system based on multimodal perception according to claim 1 is characterized in that: The communication module includes a 5G-MEC edge computing module and a human-computer interaction module, and the human-computer interaction module includes an AR remote control module and a voice interaction module.

8. The humanoid robot intelligent decoration construction system based on multimodal perception according to claim 1 is characterized in that: The collaborative operation module supports multi-robot collaborative obstacle avoidance and operation task allocation based on the dynamic priority algorithm and conflict resolution mechanism of the multi-robot scheduling system.

9. The construction method corresponding to the humanoid robot intelligent decoration construction system based on multimodal perception according to claim 1 is characterized in that: The following steps are involved: S1 data input: acquiring point cloud data, image data, and posture data in real time through the multimodal perception module; S2 data preprocessing: filtering the point cloud data, extracting features from the image data, and calibrating the data using the posture data; S3 data fusion: Based on the calibrated real-time point cloud data and image data, the building model is modeled through the environmental modeling module to generate a high-precision three-dimensional construction surface model; S4 construction operation path generation: generating a collision-free operation path of the humanoid robot body through the intelligent decision-making module; S5 Collaborative work: Assign tasks to multiple robots and plan the work sequence.

Images