An intelligent system, method and medium for automatically installing steel bars

Through the intelligent system and action command prediction framework, the operation robot arm is used to automatically perform actions, and the problem of low installation efficiency of steel bars in the existing technology is solved, fully automatic installation of steel bars is realized, and construction quality and efficiency are improved.

CN118617426BActive Publication Date: 2024-12-03HUNAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411106789.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-12-03
Estimated Expiration
2044-08-13

AI Technical Summary

Technical Problem

Due to frequent changes in demand during steel bar construction, existing automatic installation equipment needs to continuously recreate three-dimensional models, resulting in low installation efficiency and unable to achieve fully automated installation.

Method used

Using an intelligent system, the environment image is obtained at preset times through the work robot arm, and input it into the action command prediction framework to generate action prediction results, automatically perform actions, realizing automatic installation of steel bars without the need for a three-dimensional model.

Benefits of technology

It improves the efficiency and quality of steel bars installation, saves time in creating three-dimensional models, ensures construction quality, and realizes automatic installation of steel bars in any scenario.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118617426B_ABST
    Figure CN118617426B_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide an intelligent system, method and medium for automatically installing steel bars, which are used to realize automatic installation of steel bars. The intelligent system includes: a control terminal and an operating robot arm; the operating robot arm obtains an environmental image of the current location at preset time intervals; when the operating robot arm obtains the environmental image, the control terminal inputs the environmental image into an action instruction prediction framework to generate an action prediction result; when the environmental image includes steel bars that have not been installed, the action prediction result includes an execution action; the execution action is used to indicate the action that the operating robot arm needs to perform; when the environmental image includes steel bars that have been installed, or the environmental image does not include steel bars, the action prediction result is empty; if the action prediction result is not empty, the operating robot arm performs the execution action contained in the action prediction result; if the action prediction result is empty, the installation is terminated.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of automation technology, and in particular to an intelligent system, method and medium for automatically installing steel bars. Background Art

[0002] The time required for steel bar construction accounts for 33% of the total construction process time, which is one of the most time-consuming links. In order to reduce the time required for steel bar installation in the prior art, a three-dimensional model is usually created based on current demand, so that the steel bar installation equipment can automatically install the steel bar based on the three-dimensional model. However, in actual applications, due to the constant changes in current demand, the steel bar installation equipment needs to constantly re-create the three-dimensional model based on current demand, resulting in low installation efficiency of the steel bar installation equipment. Therefore, how to improve the installation efficiency of steel bars has become a technical problem that needs to be solved at present. Summary of the invention

[0003] In view of the above problems, the purpose of the present invention is to provide an intelligent system, method and medium for automatically installing steel bars, so as to realize fully automated installation of steel bars and improve installation efficiency. The specific scheme is as follows:

[0004] In a first aspect, an embodiment of the present application provides an intelligent system for automatically installing steel bars, the intelligent system comprising: a control terminal and an operating robot arm;

[0005] The operating robot arm obtains an environmental image of the current location at a preset time interval;

[0006] When the operating robot arm acquires the environment image, the control terminal inputs the environment image into the action instruction prediction framework in the control terminal to generate an action prediction result; when the environment image includes steel bars that have not been installed, the action prediction result is used to indicate the execution action that the operating robot arm needs to perform; when the environment image includes steel bars that have been installed, or the environment image does not include steel bars, the action prediction result is empty;

[0007] If the action prediction result is not empty, the working robot arm executes the execution action included in the action prediction result;

[0008] If the action prediction result is empty, the installation is terminated.

[0009] Optionally, the intelligent system further comprises a control robot arm;

[0010] When the control robot arm is training the motion instruction prediction framework, the operation robot arm controls the operation robot arm to install the training steel bars until the installation of the training steel bars is completed; the training steel bars are the steel bars used when training the motion instruction prediction framework;

[0011] The control terminal stores the execution actions of the operating robot arm and the images collected by the operating robot arm during the training of the motion instruction prediction framework according to the timestamp;

[0012] The control terminal constructs the motion instruction prediction framework based on the execution actions of the operating robot arm, the images collected by the operating robot arm, and the timestamp during the training of the motion instruction prediction framework.

[0013] Optionally, the action prediction result includes k execution actions with a time sequence relationship, where k is an integer greater than or equal to 2, and when the preset time is k×n, n is the time required to execute one of the k execution actions. When the operating robot arm acquires the environment image, the control terminal inputs the environment image into the action instruction prediction framework to generate an action prediction result, including:

[0014] The control terminal inputs the environment image into the action instruction prediction framework to obtain k execution actions and the execution times corresponding to the k execution actions;

[0015] The control terminal determines the k execution actions and the execution times corresponding to the k execution actions as the action prediction results.

[0016] Optionally, the action prediction result includes k execution actions with a time sequence relationship, where k is an integer greater than or equal to 2; when the preset time is the time required to execute an execution action, when the operating robot arm acquires the environment image, the control terminal inputs the environment image into the action instruction prediction framework to generate an action prediction result, including:

[0017] The control terminal inputs the environment image into the action instruction prediction framework to obtain k execution actions and k execution times corresponding to the k execution actions;

[0018] If there are historical action prediction results, the control terminal selects b historical action prediction results from each historical action prediction result as b target historical action prediction results; b is less than or equal to k-1; the historical action prediction result is the action prediction result generated by the action instruction prediction framework within the preset time; the preset time is the time range formed by the historical time obtained by subtracting k-1 preset times from the current time to the current time;

[0019] The control terminal performs weighted calculation on the k execution actions and the execution actions included in the b target historical action prediction results to determine the execution action with the highest weight value corresponding to each of the k execution moments;

[0020] The control terminal determines the execution action with the highest weight value corresponding to each of the k execution moments as the action prediction result;

[0021] If the historical action prediction result does not exist, the control terminal uses the k execution actions and the execution times corresponding to the k execution actions as the action prediction result.

[0022] Optionally, the operating robot arm includes: a building machine truss, a sliding carriage, a telescopic platform, a robot arm body and an end effector; the sliding carriage is used to slide on the building machine truss; the telescopic platform is located on the sliding carriage and is used to adjust the height of the robot arm body; the robot arm body is located on the telescopic platform; the end effector is connected to the robot arm body and is used to install steel bars; when the environmental image includes steel bars that have not been installed, the action prediction result is used to indicate the execution action that the operating robot arm needs to perform, including:

[0023] When the environmental image includes steel bars that have not been installed, the execution action in the action prediction result is used to indicate the sliding distance of the sliding carriage, the telescopic height of the telescopic platform, the joint adjustment angle on the robot arm body and the adjustment angle of the end effector.

[0024] Optionally, if the action prediction result is not empty, the operation robot arm executes the execution action included in the action prediction result, including:

[0025] If the action prediction result is not empty, the operating robot arm adjusts the sliding distance of the sliding carriage, the telescopic height of the telescopic platform, the angles of each joint on the robot arm body and the angle of the end effector according to the action prediction result.

[0026] Optionally, the operating mechanical arm further includes: a first image acquisition device and a second image acquisition device; the first image acquisition device is located on the telescopic platform, and the second image acquisition device is located on the mechanical arm; the operating mechanical arm acquires an environmental image of the current location at a preset time interval, including:

[0027] The operating robot arm controls the first image acquisition device and the second image acquisition device to acquire an environmental image of a current location at preset time intervals.

[0028] In a second aspect, an embodiment of the present application provides a method for automatically installing steel bars, the method being applied to an intelligent system for automatically installing steel bars, the method comprising:

[0029] Obtaining an environmental image of the current location at a preset time interval;

[0030] When the environment image is acquired, the environment image is input into the action instruction prediction framework to generate an action prediction result; when the environment image includes steel bars that have not been installed, the action prediction result is used to indicate the execution action that needs to be performed; when the environment image includes steel bars that have been installed, or when the environment image does not include steel bars, the action prediction result is empty;

[0031] If the action prediction result is not empty, executing the execution action included in the action prediction result;

[0032] If the action prediction result is empty, the installation is terminated.

[0033] Optionally, before acquiring the environment image of the current location at every preset time, the method further includes:

[0034] When training the motion instruction prediction framework, installing the training steel bars until the installation of the training steel bars is completed; the training steel bars are the steel bars used when training the motion instruction prediction framework;

[0035] Storing the executed actions and the collected images during the training of the action instruction prediction framework according to the timestamps;

[0036] Based on the execution actions in the process of training the action instruction prediction framework, the collected images, and the timestamp, the action instruction prediction framework is constructed. In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, any of the above-mentioned methods for automatically installing steel bars is implemented.

[0037] Compared with the prior art, this application has the following beneficial effects:

[0038] The operating robot arm obtains the environmental image of the current location at preset intervals, and the control terminal inputs the environmental image obtained by the operating robot arm into the action instruction prediction framework to generate an action prediction result. When the environmental image includes steel bars that have not been installed, the action prediction result includes an execution action. When the environmental image includes steel bars that have been installed, or the environmental image does not include steel bars, the action prediction result is empty. If the action prediction result is not empty, the operating robot arm executes the execution action contained in the action prediction result. If the action prediction result is empty, the installation is terminated. In this way, the control terminal can automatically generate an execution action corresponding to the actual scene based on the environmental image collected by the operating robot arm based on the actual scene. Without creating a three-dimensional model, the operating robot arm can be controlled to automatically execute the execution action that conforms to the current scene, so that the steel bars can be automatically installed in the entire process in any scene, saving the time required to create a three-dimensional model, improving the installation quality and efficiency, and ensuring the construction quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A schematic diagram of the connection relationship of devices in an intelligent system for automatically installing steel bars provided in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of the structure of a working robot arm provided in an embodiment of the present application;

[0042] Figure 3 A schematic structural diagram of a stirrup clamp provided in an embodiment of the present application;

[0043] Figure 4 A schematic structural diagram of a straight rib clamp provided in an embodiment of the present application;

[0044] Figure 5 A schematic diagram of the structure of a steel bar binding device provided in an embodiment of the present application;

[0045] Figure 6 A schematic diagram of an action prediction result provided in an embodiment of the present application;

[0046] Figure 7 A schematic diagram of another action prediction result provided in an embodiment of the present application;

[0047] Figure 8A schematic diagram of the structure of an expert demonstration system provided in an embodiment of the present application;

[0048] Fig. 9 A schematic diagram of a data transmission process of a control robot arm, a control terminal and an operating robot arm provided in an embodiment of the present application;

[0049] Fig.10 A schematic diagram of a training process of an action instruction prediction model provided in an embodiment of the present application;

[0050] Fig.11 A schematic diagram of a test flow of an action instruction prediction model provided in an embodiment of the present application;

[0051] Fig.12 A schematic flow chart of a method for automatically installing steel bars provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0054] like Figure 1 As shown, the embodiment of the present application provides an intelligent system for automatically installing steel bars, and the intelligent system includes: a control terminal 102 and an operating robot arm 101;

[0055] The operating robot arm 101 obtains an environmental image of the current location at a preset time interval;

[0056] When the operating robot arm 101 acquires the environment image, the control terminal 102 inputs the environment image into the action instruction prediction framework in the control terminal 102 to generate an action prediction result; when the environment image includes steel bars that have not been installed, the action prediction result is used to indicate the execution action that the operating robot arm 101 needs to perform; when the environment image includes steel bars that have been installed, or when the environment image does not include steel bars, the action prediction result is empty;

[0057] If the action prediction result is not empty, the working robot arm 101 performs the execution action included in the action prediction result;

[0058] If the action prediction result is empty, the installation ends.

[0059] The operating robot arm 101 collects the environment image of the current location to obtain the environment image. When the difference between the current time and the time when the environment image was collected last time reaches the preset time, the operating robot arm 101 re-collects the environment image of the current location to realize the automatic update of the environment image. After each acquisition of the environment image, the operating robot arm 101 sends the collected environment image to the control terminal 102 in the form of wireless data transmission, so that the control terminal 102 can obtain the environment image collected by the operating robot arm 101.

[0060] Each time the control terminal 102 acquires an environmental image, it inputs the environmental image acquired this time into the action instruction prediction framework. If the action instruction prediction framework determines that the steel bar has not been installed based on the environmental image input this time, it generates an action prediction result for indicating the execution action that the working robot arm 101 needs to perform, and sends the action prediction result to the working robot arm 101, so that the working robot arm 101 performs the execution action included in the action prediction result. If the action instruction prediction framework determines that the steel bar has been installed based on the environmental image input this time, or the environmental image does not contain steel bars, it generates an action prediction result that does not contain any information, and feeds back a notification message of the end of installation to the control terminal 102.

[0061] For easier understanding, the following example is given:

[0062] The structure of the operating robot arm 101 is as follows: Figure 2As shown, it includes a building machine truss 201, a sliding carriage 207, a telescopic platform 202, a mechanical arm 205, an end effector 206, a second image acquisition device 204 installed at the end of the mechanical arm 205, and a first image acquisition device 203 installed at the telescopic platform 202. The first image acquisition device 203 and the second image acquisition device 204 are specifically color cameras (RGB cameras). The sliding carriage 207 is used to move horizontally on the building machine truss 201, and the building machine truss 201 can move longitudinally on the horizontal plane. The telescopic platform 202 is mounted on the sliding carriage 207 and is telescoped up and down using a gear guide rail. The mechanical arm 205 has six joints, and the end effector 206 is connected to the end of the mechanical arm 205 using a flange. The end effector 206 is used to install steel bars and can be replaced according to work requirements, such as stirrup clamps, straight bar clamps, and steel bar tiers.

[0063] The structure of the stirrup clamp is as follows Figure 3 As shown, it is composed of two two-finger clamps 301, wherein the rotation axes of the two-finger clamps are vertical, and a protruding fulcrum 302 is provided at the vertical intersection to prevent the stirrups from turning over after being clamped. Figure 4 As shown, it is composed of two semicircular finger-like clamping devices 401. After grabbing the straight steel bar, the clamp closes and moves in the opposite direction to lock the straight steel bar in the clamp. Figure 5 As shown, there is a welding head 501 for welding.

[0064] When the environment image of the current location is acquired by the working robot arm 101, the first image acquisition device 203 and the second image acquisition device 204 are used to respectively acquire images to obtain two images containing the steel bars to be installed. The two images containing the steel bars to be installed are used as the environment image, so that the global information of the working state of the working robot arm 101 for the steel bars to be installed is acquired by the first image acquisition device 203 installed on the telescopic platform 202, and the local information of the working state of the working robot arm 101 for the steel bars to be installed is acquired by the second image acquisition device 204 installed at the end of the robot arm 205, so that the content contained in the acquired environment image is richer and more accurate, and the accuracy of subsequent action prediction is improved.

[0065] The control terminal 102 inputs the environment image into the action instruction prediction framework. If the action instruction prediction framework determines that the steel bar is not installed based on the environment image input this time, an action prediction result including an execution action is generated. The execution action is used to indicate the sliding distance of the sliding carriage 207, the telescopic height of the telescopic platform 202, the adjustment angle of each joint on the robot arm 205, and the adjustment angle of the end effector 206. The operating robot arm 101 adjusts the sliding distance of the sliding carriage 207, the telescopic height of the telescopic platform 202, the angle of each joint on the robot arm 205, and the angle of the end effector 206 based on the execution action included in the action prediction result.

[0066] It should be noted that the execution of an action includes multiple degrees of freedom, and one degree of freedom is used to instruct a component or joint in the working robot arm 101 to perform a corresponding action. Figure 2 Taking the working robot arm 101 shown as an example, the execution action may include ten degrees of freedom, including x, y, z, θ1, θ2, θ3, θ4, θ5, θ6, and θ7. x represents the lateral displacement of the sliding carriage 207 on the building machine truss 201, y represents the longitudinal displacement of the building machine truss 201 in the horizontal direction, z represents the height displacement of the telescopic platform 202, θ1, θ2, θ3, θ4, θ5, and θ6 represent the adjustment angles of the six joints of the robot arm 205, and θ7 represents the adjustment angle of the end effector 206. Then the execution action corresponding to the time T=i+1 can be expressed as:

[0067] i+1 (i) =[ xyz θ1θ2θ3θ4θ5θ6θ7] T i→i+1

[0068] i+1 (i) The execution action corresponding to the time T=i+1 is generated by the action instruction prediction framework at T=i.

[0069] Thus, by treating the actions that need to be performed by various components or joints in the working robot arm 101 at the same time as a whole, an execution action is obtained to reduce the effective field of view of the long trajectory action, so that the execution action can be stored and executed more efficiently.

[0070] The operating robot arm obtains the environmental image of the current location at preset intervals, and the control terminal inputs the environmental image obtained by the operating robot arm into the action instruction prediction framework to generate an action prediction result. When the environmental image includes steel bars that have not been installed, the action prediction result includes an execution action. When the environmental image includes steel bars that have been installed, or the environmental image does not include steel bars, the action prediction result is empty. If the action prediction result is not empty, the operating robot arm executes the execution action contained in the action prediction result. If the action prediction result is empty, the installation is terminated. In this way, the control terminal can automatically generate an execution action corresponding to the actual scene based on the environmental image collected by the operating robot arm based on the actual scene. Without creating a three-dimensional model, the operating robot arm can be controlled to automatically execute the execution action that conforms to the current scene, so that the steel bars can be automatically installed in the entire process in any scene, saving the time required to create a three-dimensional model, improving the installation quality and efficiency, and ensuring the construction quality.

[0071] In order to reduce the cumulative error in the actual application process, in an optional embodiment, the action prediction result includes k execution actions with a time sequence relationship, k is an integer greater than or equal to 2, and n is the time required to execute one of the k execution actions. When the preset time is k×n, when the operating robot arm 101 obtains the environment image, the control terminal 102 inputs the environment image into the action instruction prediction framework for action prediction, and obtains the action prediction result, including:

[0072] The control terminal 102 inputs the environment image into the action instruction prediction framework to obtain k execution actions and the execution time corresponding to each of the k execution actions;

[0073] The control terminal 102 uses the k execution actions and the execution times corresponding to the k execution actions as action prediction results.

[0074] The execution time required for each of the k execution actions is the same execution time n, so that the environment image is updated every k×n time, and the environment image can be automatically updated in time after each execution of the k execution actions in the action prediction result is completed, and a new action prediction result is generated based on the action instruction prediction framework. In this way, the normal operation of the intelligent system for automatic installation of steel bars is ensured while reducing the amount of environment images required to be acquired.

[0075] like Figure 6 As shown in the figure, taking k=5 and n=1 second as an example, the environment image at time T=i is input into the action instruction prediction framework, and the action prediction results of the working robot arm 101 from T=i+1 second to T=i+5 second are predicted. At this time, the action prediction results include the execution action i+1 corresponding to T=i+1 second. (i)、Execute action i+2 corresponding to T=i+2 seconds (i) 、Execute action i+3 corresponding to T=i+3 seconds (i) 、Execute action i+4 corresponding to T=i+4 seconds (i) 、Execute action i+5 corresponding to T=i+5 seconds (i) , and execute the execution actions from T=i+1 seconds to T=i+5 seconds in sequence. (i) When the time interval between the environment image and the last update is 5 seconds, the environment image is updated and the updated environment image is input into the action instruction prediction framework. If the steel bars in the environment image have not been installed yet, the action prediction results from T=i+6 seconds to T=i+10 seconds are obtained. At this time, the action prediction results include the execution action i+6 corresponding to T=i+6 seconds. (i) 、Execute action i+7 corresponding to T=i+7 seconds (i) 、Execution action i+8 corresponding to T=i+8 seconds (i) 、Execution action i+9 corresponding to T=i+9 seconds (i) 、Execute action i+10 corresponding to T=i+10 seconds (i) If the steel bars in the environment image have been installed at this time, the prediction result is empty. Therefore, by making the action prediction result output by the action instruction prediction framework include k actions each time, the effective field of view is reduced by k times, reducing the cumulative error caused by the large effective field of view.

[0076] To further improve the smoothness of the movement of the operating robot arm 101, in another optional embodiment, the action prediction result includes k execution actions with a time sequence relationship, where k is an integer greater than or equal to 2; when the preset time is the time required to execute an execution action, when the operating robot arm 101 obtains the environment image, the control terminal 102 inputs the environment image into the action instruction prediction framework for action prediction, and obtains the action prediction result, including:

[0077] The control terminal 102 inputs the environment image into the action instruction prediction framework to obtain k execution actions and k execution times corresponding to the k execution actions;

[0078] If there are historical action prediction results, the control terminal 102 selects b historical action prediction results from each historical action prediction result as b target historical action prediction results; b is less than or equal to k-1; the historical action prediction result is an action prediction result generated by the action instruction prediction framework within a preset time range; the preset time range is a time range formed by subtracting k-1 preset times from the current time to the current time;

[0079] The control terminal 102 performs weighted calculation on the k execution actions and the execution actions included in the b target historical action prediction results, and determines the execution action with the highest weight value corresponding to each of the k execution moments;

[0080] The control terminal 102 uses the execution action with the highest weight value corresponding to each of the k execution moments as the action prediction result;

[0081] If there is no historical action prediction result, the control terminal 102 uses the k execution actions and the execution times corresponding to the k execution actions as the action prediction results.

[0082] The execution time required for each of the k execution actions is the same execution time n, and the preset time range is the time range from the current time minus n×(k-1) to the current time.

[0083] After the currently collected environment image is input into the action instruction prediction framework and k execution actions are obtained, it is determined whether there is an action prediction result generated within the preset time range. If so, the action prediction result generated within the preset time range is used as the historical action prediction result. From each historical action prediction result, b historical action prediction results are selected as b target historical action prediction results. The larger the value of b, the more accurate the action prediction result generated in the end. The maximum value of b is k-1.

[0084] According to the prediction results, the earlier the rule is generated, the higher the weight value is. i =e -mi The formula is used to set weight values ​​for the k execution actions predicted this time and the execution actions included in the b historical action prediction results. i It represents the weight value of the execution action predicted and generated at the i-th moment, m is a constant, the smaller m is, the faster the merging is, and i is the moment when the execution action is predicted. Based on the weighted calculation formula , perform weighted calculation on the k execution actions and the execution actions included in the b target historical action prediction results, determine the execution action with the highest weight value corresponding to each of the k execution moments, and use the execution action with the highest weight value corresponding to each of the k execution moments as the action prediction result. is the weight value for weighted calculation of the overlapping execution actions at time T=i+j among the prediction results of k execution actions and b target historical actions, w i [i+j (i) ] is the weight value of the execution action predicted at the i-th moment at the execution time T=i+j.

[0085] For easier understanding, the following example is given:

[0086] like Figure 7 As shown in the figure, since the preset time is the time required to perform an action, the action instruction prediction framework will generate an action prediction result based on the updated environment image every time an action is performed. Taking k=5, b=4, and n=1 second as an example, when T=i+4 seconds, the target historical action prediction results include the action prediction results generated at T=i seconds, T=i+1 seconds, T=i+2 seconds, and T=i+3 seconds. i The corresponding weight, i+5 (i+1) The corresponding weight, i+5 (i+2) The corresponding weight, i+5 (i+3) The corresponding weight, i+5 (i+4) The corresponding weight is input into the weighted calculation formula to obtain , to determine the execution action with the highest weight value from the five execution actions corresponding to the execution time T=i+5. According to the above scheme, continue to calculate the execution action with the highest weight value at T=i+6, the execution action with the highest weight value at T=i+7, the execution action with the highest weight value at T=i+8, and the execution action with the highest weight value at T=i+9. The execution action with the highest weight value from T=i+5 to T=i+9 is used as the action prediction result output at T=i+4. Thereby, the subsequent k-step execution actions are predicted at each time step, the operation stability of the operating robot arm 101 is improved, and the discrete switching of the intelligent system between the execution and observation environment is avoided due to the sudden acquisition of new environmental images by the intelligent system.

[0087] like Figure 8 As shown, in an optional embodiment, the above intelligent system further includes: a control robot arm 801;

[0088] When the control robot arm 801 is training the motion instruction prediction framework, the control robot arm 101 is controlled to install the training steel bars until the installation of the training steel bars is completed; the training steel bars are the steel bars used when training the motion instruction prediction framework;

[0089] The control terminal 102 stores the execution actions of the working robot arm 101 and the images collected by the working robot arm 101 during the training action instruction prediction framework according to the timestamp;

[0090] The control terminal 102 constructs an action instruction prediction framework based on the execution actions of the working robot arm 101, the images collected by the working robot arm 101, and the timestamp during the training of the action instruction prediction framework.

[0091] When training the action command prediction framework, the above intelligent system Figure 8As shown, it includes a control robot arm 801, a control terminal 102 and an operating robot arm 101. The control terminal 102 can be specifically an industrial computer, and the control robot arm 801 is a scaled-down version of the operating robot arm 101, so that the control robot arm 801 meets ergonomic requirements. The operating robot arm 101 and the control robot arm 801 are connected to the control terminal 102 via TCP and maintain real-time information exchange. The data transmission process of the control robot arm 801, the control terminal 102 and the operating robot arm 101 is as follows: Fig. 9 As shown. The demonstrator manually controls the movement of the manipulator arm 801, and the control terminal 102 obtains the joint state and force sensor data of the manipulator arm 801 in real time, and uses the spatial mapping method to control the working manipulator arm 101 in real time, so that the manipulator arm 801 controls the working manipulator arm 101. The working manipulator arm 101 will feed back the force conditions in the working process to the control terminal 102 in real time, and the control terminal 102 will transfer the feedback to the manipulator arm 801 in real time, so that the demonstrator can clearly feel the current working situation and judge the subsequent actions to be performed. After the demonstrator completes the installation of the training steel bars by the manipulator arm 801, the execution actions of the working manipulator arm 101 collected by the manipulator arm 801 during the installation of the training steel bars, the images collected by the working manipulator arm 101, and the timestamps corresponding to the execution actions and images are used as expert demonstration data.

[0092] It should be noted that when training the action command prediction framework, the image data collected by the control robot 801, the working robot 101 and the two RGB cameras are synchronously collected at a frequency of more than 50HZ according to the timestamp. The execution action of the working robot 101 includes the joint state of the working robot 101 during the installation of the training steel bar, the force sensor data, and the time sequence relationship of each joint state.

[0093] The control terminal 102 constructs the action instruction prediction framework based on the execution action of the working robot 101, the image collected by the working robot 101, and the timestamp during the training of the action instruction prediction framework. The construction process of the action instruction prediction framework includes training and testing the action instruction prediction model to obtain the action instruction prediction framework.

[0094] The training part is as follows Fig.10As shown, the joint state of the working robot arm 101 includes joint angles. The RGB image and the tuple of joint angles in the expert demonstration data are sampled, and the k-step action sequence after the current joint angle determined based on the temporal relationship of the joint state is used as the prediction target. The conditional variational autoencoder (CAVE) is used to infer the style variable s, and the joint angles are linearly mapped to obtain the embedded joint angles. The k-step action sequence is linearly mapped and positionally embedded to obtain the embedded action sequence. The randomly initialized learning weights, embedded joint angles, and embedded action sequence are used as the input of the transformer encoder to obtain the output of the transformer encoder. The output of the transformer encoder is linearly mapped using a linear network to predict the mean and variance of the s distribution, and the mean and variance of the s distribution are parameterized as a diagonal Gaussian distribution.

[0095] The joint angles and diagonal Gaussian distributions are linearly mapped, and the RGB images are linearly mapped and featurized. The linearly mapped joint angles and diagonal Gaussian distributions, as well as the linearly mapped and featurized RGB images are input into the transformer encoder. The "keys" and "values" output by the transformer encoder are used as the input of the cross layer of the transformer decoder. At the same time, the output of the transformer decoder after position embedding, the "queries", is used as the input of the first layer of the transformer decoder to obtain the predicted k-step action sequence.

[0096] It should be noted that during the training process, the embodiment of the present application uses mean square error as the loss function of the model.

[0097] Test part such as Fig.11As shown in the figure, the style variable s is set to the 0 vector, which is the average value of the style variables used during training. The joint angles and s are linearly mapped respectively, and the RGB image is linearly mapped and featurized. The linearly mapped joint angles and s, as well as the linearly mapped and featurized RGB image are input into the transformer encoder. The "keys" and "values" output by the transformer encoder are used as the input of the cross layer of the transformer decoder, and the output of the transformer decoder after position embedding, "queries", is used as the input of the first layer of the transformer decoder to obtain the predicted k-step action sequence. When the predicted k-step action sequence is the same as the actual action sequence, the action instruction prediction model is trained to obtain the action instruction prediction framework. Thus, by combining CVAE with transformer, the noise impact of expert demonstration data is reduced.

[0098] like Fig.12 As shown, an embodiment of the present application provides a method for automatically installing steel bars, the method is applied to an intelligent system for automatically installing steel bars, and the method includes:

[0099] S1201: Acquire an environmental image of the current location at a preset time interval;

[0100] S1202: When the environment image is acquired, the environment image is input into the action instruction prediction framework to generate an action prediction result; when the environment image includes steel bars that have not been installed, the action prediction result is used to indicate the execution action that needs to be performed; when the environment image includes steel bars that have been installed, or when the environment image does not include steel bars, the action prediction result is empty;

[0101] S1203: If the action prediction result is not empty, executing the execution action included in the action prediction result;

[0102] S1204: If the action prediction result is empty, the installation is terminated.

[0103] In an optional embodiment, before acquiring the environment image of the current location at every preset time, the method further includes:

[0104] When training the motion instruction prediction framework, the training steel bars are installed until the installation of the training steel bars is completed; the training steel bars are the steel bars used when training the motion instruction prediction framework;

[0105] The execution actions and collected images during the training action instruction prediction framework are stored according to the timestamps;

[0106] An action command prediction framework is constructed based on the executed actions, collected images, and timestamps during the training process of the action command prediction framework.

[0107] In an optional embodiment, the action prediction result includes k execution actions with a time sequence relationship, k is an integer greater than or equal to 2, and n is the time required to execute one of the k execution actions. When the preset time is k×n and the environment image is acquired, the acquired environment image is input into the action instruction prediction framework to generate the action prediction result, including:

[0108] Input the environment image into the action command prediction framework to obtain k execution actions and the execution time corresponding to each of the k execution actions;

[0109] The k execution actions and the execution times corresponding to the k execution actions are taken as action prediction results.

[0110] In an optional embodiment, the action prediction result includes k execution actions with a time sequence relationship, where k is an integer greater than or equal to 2; when the preset time is the time required to execute an execution action and an environmental image is acquired, the acquired environmental image is input into the action instruction prediction framework to generate an action prediction result, including:

[0111] Input the environment image into the action command prediction framework to obtain k execution actions and k execution times corresponding to the k execution actions;

[0112] If there are historical action prediction results, then b historical action prediction results are selected from each historical action prediction result as b target historical action prediction results; b is less than or equal to k-1; the historical action prediction result is the action prediction result generated by the action instruction prediction framework within a preset time range; the preset time range is the time range formed by subtracting k-1 preset times from the current time to the current time;

[0113] Perform weighted calculation on the k execution actions and the execution actions included in the prediction results of the b target historical actions, and determine the execution action with the highest weight value corresponding to each of the k execution moments;

[0114] The execution action with the highest weight value corresponding to each of the k execution moments is taken as the action prediction result;

[0115] If there is no historical action prediction result, the k executed actions and the execution times corresponding to the k executed actions are used as the action prediction results.

[0116] In an optional embodiment, when the environment image includes steel bars that have not been installed, the action prediction result is used to indicate an execution action that needs to be performed, including:

[0117] When the environment image includes steel bars that have not been installed, the execution action in the action prediction result is used to indicate the sliding distance of the sliding carriage, the telescopic height of the telescopic platform, the adjustment angles of each joint on the robotic arm, and the adjustment angle of the end effector.

[0118] In an optional embodiment, if the action prediction result is not empty, executing the execution action included in the action prediction result includes:

[0119] If the motion prediction result is not empty, the sliding distance of the sliding carriage, the telescopic height of the telescopic platform, the angles of each joint on the robot arm and the angle of the end effector are adjusted according to the motion prediction result.

[0120] The operating robot arm obtains the environmental image of the current location at preset intervals, and the control terminal inputs the environmental image obtained by the operating robot arm into the action instruction prediction framework to generate an action prediction result. When the environmental image includes steel bars that have not been installed, the action prediction result includes an execution action. When the environmental image includes steel bars that have been installed, or the environmental image does not include steel bars, the action prediction result is empty. If the action prediction result is not empty, the operating robot arm executes the execution action contained in the action prediction result. If the action prediction result is empty, the installation is terminated. In this way, the control terminal can automatically generate an execution action corresponding to the actual scene based on the environmental image collected by the operating robot arm based on the actual scene. Without creating a three-dimensional model, the operating robot arm can be controlled to automatically execute the execution action that conforms to the current scene, so that the steel bars can be automatically installed in the entire process in any scene, saving the time required to create a three-dimensional model, improving the installation quality and efficiency, and ensuring the construction quality.

[0121] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, any one of the above-mentioned methods for automatically installing steel bars is implemented.

[0122] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0123] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0124] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. An intelligent system for automatically installing steel bars, characterized in that: The intelligent system includes: a control terminal and an operating mechanical arm; The operating robot arm obtains an environmental image of the current location at a preset time interval; When the operating robot arm acquires the environment image, the control terminal inputs the environment image into the action instruction prediction framework in the control terminal to generate an action prediction result; when the environment image includes steel bars that have not been installed, the action prediction result includes an execution action; the execution action is used to indicate the action that the operating robot arm needs to perform; when the environment image includes steel bars that have been installed, or the environment image does not include steel bars, the action prediction result is empty; If the action prediction result is not empty, the working robot arm executes the execution action included in the action prediction result; If the action prediction result is empty, the installation is terminated; The action prediction result includes k execution actions with a time sequence relationship, where k is an integer greater than or equal to 2; when the preset time is the time required to execute an execution action, when the operating robot arm acquires the environment image, the control terminal inputs the environment image into the action instruction prediction framework to generate an action prediction result, including: The control terminal inputs the environment image into the action instruction prediction framework to obtain k execution actions and k execution times corresponding to the k execution actions; If there are historical action prediction results, the control terminal selects b historical action prediction results from each historical action prediction result as b target historical action prediction results; b is less than or equal to k-1; the historical action prediction result is the action prediction result generated by the action instruction prediction framework within the preset time; the preset time is the time range formed by the historical time obtained by subtracting k-1 preset times from the current time to the current time; The control terminal performs weighted calculation on the k execution actions and the execution actions included in the b target historical action prediction results to determine the execution action with the highest weight value corresponding to each of the k execution moments; The control terminal determines the execution action with the highest weight value corresponding to each of the k execution moments as the action prediction result; If the historical action prediction result does not exist, the control terminal uses the k execution actions and the execution times corresponding to the k execution actions as the action prediction result.

2. The intelligent system according to claim 1, characterized in that: The intelligent system also includes a control robot arm; When the control robot arm is training the motion instruction prediction framework, the operation robot arm controls the operation robot arm to install the training steel bars until the installation of the training steel bars is completed; the training steel bars are the steel bars used when training the motion instruction prediction framework; The control terminal stores the execution actions of the operating robot arm and the images collected by the operating robot arm during the training of the motion instruction prediction framework according to the timestamp; The control terminal constructs the motion instruction prediction framework based on the execution actions of the operating robot arm, the images collected by the operating robot arm, and the timestamp during the training of the motion instruction prediction framework.

3. The intelligent system according to claim 1, characterized in that: The action prediction result includes k execution actions with a time sequence relationship, where k is an integer greater than or equal to 2. When the preset time is k×n, n is the time required to execute one of the k execution actions. When the operating robot arm acquires the environment image, the control terminal inputs the environment image into the action instruction prediction framework to generate an action prediction result, including: The control terminal inputs the environment image into the action instruction prediction framework to obtain k execution actions and execution times corresponding to the k execution actions; The control terminal determines the k execution actions and the execution times corresponding to the k execution actions as the action prediction results.

4. The intelligent system according to claim 1, characterized in that: The operating robot arm includes: a building machine truss, a sliding carriage, a telescopic platform, a robot arm body and an end effector; the sliding carriage is used to slide on the building machine truss; the telescopic platform is located on the sliding carriage and is used to adjust the height of the robot arm body; the robot arm body is located on the telescopic platform; the end effector is connected to the robot arm body and is used to install steel bars; when the environment image includes steel bars that have not been installed, the action prediction result is used to indicate the execution action that the operating robot arm needs to perform, including: When the environmental image includes steel bars that have not been installed, the execution action in the action prediction result is used to indicate the sliding distance of the sliding carriage, the telescopic height of the telescopic platform, the joint adjustment angle on the robot arm body and the adjustment angle of the end effector.

5. The intelligent system according to claim 4, characterized in that: If the action prediction result is not empty, the operation robot arm performs the execution action included in the action prediction result, including: If the action prediction result is not empty, the operating robot arm adjusts the sliding distance of the sliding carriage, the telescopic height of the telescopic platform, the angle of the joint on the robot arm body and the angle of the end effector according to the action prediction result.

6. The intelligent system according to claim 4, characterized in that: The operating mechanical arm further comprises: a first image acquisition device and a second image acquisition device; the first image acquisition device is located on the telescopic platform, and the second image acquisition device is located on the mechanical arm body; the operating mechanical arm acquires an environmental image of the current location at a preset time interval, including: The operating robot arm controls the first image acquisition device and the second image acquisition device to acquire an environmental image of a current location at preset time intervals.

7. A method for automatically installing steel bars, characterized in that: The method is applied to an intelligent system for automatically installing steel bars, and the method comprises: Obtaining an environmental image of the current location at a preset time interval; When the environment image is acquired, the environment image is input into the action instruction prediction framework to generate an action prediction result; when the environment image includes steel bars that have not been installed, the action prediction result is used to indicate the execution action that needs to be performed; when the environment image includes steel bars that have been installed, or when the environment image does not include steel bars, the action prediction result is empty; If the action prediction result is not empty, executing the execution action included in the action prediction result; If the action prediction result is empty, the installation is terminated; The action prediction result includes k execution actions with a time sequence relationship, where k is an integer greater than or equal to 2; when the preset time is the time required to execute an execution action, when the environment image is acquired, the environment image is input into the action instruction prediction framework to generate an action prediction result, including: Inputting the environment image into the action instruction prediction framework to obtain k execution actions and k execution times corresponding to the k execution actions; If there are historical action prediction results, b historical action prediction results are selected from each historical action prediction result as b target historical action prediction results; b is less than or equal to k-1; the historical action prediction result is the action prediction result generated by the action instruction prediction framework within the preset time; the preset time is the time range formed by the historical time obtained by subtracting k-1 preset times from the current time to the current time; Performing weighted calculation on the k execution actions and the execution actions included in the b target historical action prediction results, and determining the execution action with the highest weight value corresponding to each of the k execution moments; Determine the execution action with the highest weight value corresponding to each of the k execution moments as the action prediction result; If the historical action prediction result does not exist, the k execution actions and the execution times corresponding to the k execution actions are used as the action prediction results.

8. The method according to claim 7, characterized in that Before acquiring the environment image of the current location at every preset time, the method further includes: When training the motion instruction prediction framework, installing the training steel bars until the installation of the training steel bars is completed; the training steel bars are the steel bars used when training the motion instruction prediction framework; Storing the executed actions and the collected images during the training of the action instruction prediction framework according to the timestamps; The action instruction prediction framework is constructed based on the execution actions in the process of training the action instruction prediction framework, the collected images, and the timestamp.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, wherein when the computer program is executed by a processor, the method for automatically installing steel bars as claimed in any one of claims 7 to 8 is implemented.