Building scene robot operation method based on large model technology

Abstract

The application discloses a kind of building scene robot operation methods based on large model technology, it is related to intelligent construction technology field.The application at least includes following steps: S1: by screening robot basic action and task, trajectory data are collected and robot instruction model is trained using Diffusion model, to build bottom instruction library, optimize robot action execution;S2: in combination with BIM information and industry knowledge, the workable area in building scene is identified, and specific work task set is generated by discretization work area and path planning algorithm.The application proposes a task operation method based on large model, after given task, the prior information of building scene is used, long-range task is decoupled into multiple simple subtasks, and the robustness of task operation is improved;At the same time, the application does not need to specify the behavior logic of robot, and the robot can infer the behavior logic and complete the task according to its own bottom skill, so that the generalization ability of robot task is improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent construction technology, specifically to a method for robot operation in building scenes based on large model technology. Background Technology

[0002] When existing construction robots perform tasks, after generating work tasks for the work area based on BIM information, the robot's action sequence and behavior logic need to be manually defined, which lacks autonomy.

[0003] Meanwhile, when the architectural environment and the given task change dynamically, the robot requires human intervention and reconfiguration to adapt to the new environment.

[0004] Task planning methods based on large language model technology often adopt an end-to-end approach, providing action sequences that conform to robot and environmental constraints.

[0005] However, existing robot task planning technologies based on large language models are often limited to specific robot tasks, have difficulty adapting to the environment, and lack effective use of building scene information, making it impossible to perform tasks accurately and efficiently.

[0006] Therefore, a new solution is needed to address the above problems. Summary of the Invention

[0007] The purpose of this invention is to provide a method for robot operation in building scenes based on large model technology, so as to solve the technical problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a method for robot operation in a building scene based on large model technology, comprising at least the following steps:

[0009] S1: By screening basic robot movements and tasks, collecting trajectory data and using the Diffusion model to train the robot instruction model, a low-level instruction library is built to optimize robot movement execution;

[0010] S2: Combining BIM information and industry knowledge, identify workable areas in the building scene, and generate a specific set of work tasks through discretized work areas and path planning algorithms;

[0011] S3: Utilize large language models to construct a task thinking chain;

[0012] S4: Given task prompts, use a large language model to generate action instructions for each task, use the SimCSE model to find the most similar skill in the skill base, and execute the skill.

[0013] S5: Corrects the robot's end effector trajectory to ensure the smoothness and safety of the robot's motion path, while adhering to the actuator's motion constraints.

[0014] Furthermore, S1 includes at least the following steps:

[0015] Based on the actual robot structure, basic robot movements and tasks are selected, and a low-level instruction library for the robot is constructed.

[0016] Collect and clean trajectory data of robotic arm movements or tasks in real or simulated scenarios. For each selected movement or task, train a robot instruction model using a Diffusion model. Where I i For the i-th underlying instruction, refer to the following formula:

[0017]

[0018] Where k is the iteration step, Let O be the robot action at time t, k-1 iterations. t For the robot's observation at time t, For instruction I i Language description, N(0,δ 2 I) represents an expected value of 0 and a variance of δ. 2 I is a Gaussian distribution. Let α, γ, and δ be the robot's starting and ending poses, respectively;

[0019] The loss function for the robot instruction model is:

[0020]

[0021] in For A t The logarithmic gradient, This is the predicted logarithmic gradient of the action.

[0022] Furthermore, the robot's underlying instruction library includes at least instructions for actuator actions or task actions.

[0023] Furthermore, the trajectory data includes at least robot pose and robot image data.

[0024] Furthermore, S2 includes at least the following steps:

[0025] By combining BIM information with prior industry knowledge, the operable areas of robots within the building scene can be identified.

[0026] For each work area S k,k∈[1,K], use discrete point path planning and task generation algorithm to generate a set of tasks in the task area, each task includes the task start point, the task end point and the task content;

[0027] The discrete point path planning and task generation algorithm includes at least the following steps:

[0028] For each work area S k Take discrete points uniformly on the surface, denoted as P = {P1, ..., P2}. i ,…,P M}, where M is the number of discrete points, P i For discrete point poses;

[0029] Connecting surface points using zig-zag or Hilbert curves, etc., then task T i ={P i ,P i+1 ,C}, where C is the task.

[0030] Furthermore, the operational thought chain in S3 includes at least trajectory information, action information, and historical information. The trajectory information represents the action that has been executed so far. The action information includes at least the robot's underlying instructions that can be used and the meaning of the instructions. The historical information includes at least the robot action information that was executed previously.

[0031] Furthermore, S4 includes at least the following steps:

[0032] First, the robot uses its own sensors to perceive the environment and determine the positional relationship between the robot and the work area;

[0033] For each action, geometric and dynamic functions are constructed based on the work point and work area to correct the smoothness and dynamic safety of the trajectory;

[0034] For each task T i The coordinates P of the work point are obtained through a three-dimensional sensor. i Coordinates p within the work area i According to P i+1 With P i The relative relationship is used to obtain P. i+1 Coordinates p within the work area i+1 The large model is used to break down the task into corresponding action instruction sets I′.

[0035] For each action instruction in the job instruction set, the logarithmic gradient of the action estimated by the robot instruction model satisfies:

[0036]

[0037] in

[0038] The robot's movements can be calculated using the following formula:

[0039]

[0040] Where β is a parameter.

[0041] Furthermore, S5 includes at least the following steps:

[0042] The calculated robot end effector trajectory e = (e1, ..., e j ,…,e N Make corrections;

[0043] Where: e1 = p i e N =p i+1 e j Let N be the j-th pose point, and N be the number of poses;

[0044] Therefore, the actuator action constraint function is as follows:

[0045]

[0046] Where, λ snap (e j ) for e j-1 to e j The smooth term of the trajectory, This means that in e j-1 to e j The distance at which the movement path violates the safety threshold of the obstacle.

[0047] Compared with the prior art, the beneficial effects of the present invention are:

[0048] This invention proposes a task operation method based on a large model. After a task is given, prior information about the building scene is used to decouple a long-term task from multiple simple sub-tasks, thereby improving the robustness of task operation.

[0049] Meanwhile, this invention eliminates the need for human intervention to specify the robot's behavioral logic. The robot can autonomously reason about behavioral logic and complete tasks based on its own underlying skills, thereby improving the robot's task generalization ability. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a schematic diagram of the execution flow of a single time step task in this invention;

[0052] Figure 2 This is a schematic diagram of the robot's underlying skill library according to the present invention;

[0053] Figure 3 This is a schematic diagram of the test area of ​​the present invention;

[0054] Figure 4 This is a schematic diagram of the measurement point sequence of the present invention;

[0055] Figure 5 This is a schematic diagram of the bolt fastening skill library of the present invention;

[0056] Figure 6 This is a schematic diagram of the bolt fastening skill library of the present invention;

[0057] Figure 7 This is a schematic diagram of the fastening bolts of the robot. Detailed Implementation

[0058] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0059] Example 1:

[0060] Please see Figure 1 and Figure 2 A method for robot operation in a building scene based on large model technology includes at least the following steps:

[0061] S1: By screening basic robot movements and tasks, collecting trajectory data and using the Diffusion model to train the robot instruction model, a low-level instruction library is built to optimize robot movement execution;

[0062] S2: Combining BIM information and industry knowledge, identify workable areas in the building scene, and generate a specific set of work tasks through discretized work areas and path planning algorithms;

[0063] S3: Utilize large language models to construct a task thinking chain;

[0064] S4: Given task prompts, use a large language model to generate action instructions for each task, use the SimCSE model to find the most similar skill in the skill base, and execute the skill.

[0065] S5: Corrects the robot's end effector trajectory to ensure the smoothness and safety of the robot's motion path, while adhering to the actuator's motion constraints.

[0066] S1 includes at least the following steps:

[0067] Based on the actual robot structure, basic robot movements and tasks are selected, and a low-level instruction library for the robot is constructed.

[0068] Collect and clean trajectory data of robotic arm movements or tasks in real or simulated scenarios. For each selected movement or task, train a robot instruction model using a Diffusion model. Where I i For the i-th underlying instruction, refer to the following formula:

[0069]

[0070] Where k is the iteration step, Let O be the robot action at time t, k-1 iterations. t For the robot's observation at time t, L Ii For instruction I i Language description, N(0,δ 2 I) represents an expected value of 0 and a variance of δ. 2 I is a Gaussian distribution. Let α, γ, and δ be the robot's starting and ending poses, respectively;

[0071] The loss function for the robot instruction model is:

[0072]

[0073] in For A t The logarithmic gradient, This is the predicted logarithmic gradient of the action.

[0074] The robot's underlying instruction library includes at least instructions for actuator actions or task actions.

[0075] Trajectory data includes at least robot pose and robot image data.

[0076] S2 includes at least the following steps:

[0077] By combining BIM information with prior industry knowledge, the operable areas of robots within the building scene can be identified.

[0078] For each work area S k ,k∈[1,K], use discrete point path planning and task generation algorithm to generate a set of tasks in the task area, each task includes the task start point, the task end point and the task content;

[0079] Discrete point path planning and task generation algorithms include at least the following steps:

[0080] For each work area S k Take discrete points uniformly on the surface, denoted as P = {P1, ..., P2}. i ,…,P M}, where M is the number of discrete points, P i For discrete point poses;

[0081] Connecting surface points using zig-zag or Hilbert curves, etc., then task T i ={P i ,P i+1 ,C}, where C is the task.

[0082] The operational thought chain in S3 includes at least trajectory information, action information, and historical information. Trajectory information represents the actions that have been executed so far. Action information includes at least the robot's underlying instructions that can be used and the meaning of those instructions. Historical information includes at least the robot's previously executed action information.

[0083] S4 includes at least the following steps:

[0084] First, the robot uses its own sensors to perceive the environment and determine the positional relationship between the robot and the work area;

[0085] For each action, geometric and dynamic functions are constructed based on the work point and work area to correct the smoothness and dynamic safety of the trajectory;

[0086] For each task T i The coordinates P of the work point are obtained through a three-dimensional sensor. i Coordinates p within the work area i According to P i+1 With P i The relative relationship is used to obtain P. i+1 Coordinates p within the work area i+1 The large model is used to break down the task into corresponding action instruction sets I′.

[0087] For each action instruction in the job instruction set, the logarithmic gradient of the action estimated by the robot instruction model satisfies:

[0088]

[0089] in

[0090] The robot's movements can be calculated using the following formula:

[0091]

[0092] Where β is a parameter.

[0093] S5 includes at least the following steps:

[0094] The calculated robot end effector trajectory e = (e1, ..., e j ,…,e N Make corrections;

[0095] Where: e1 = p i e N =p i+1 e j Let N be the j-th pose point, and N be the number of poses;

[0096] Therefore, the actuator action constraint function is as follows:

[0097]

[0098] Where, λ snap (e j ) for e j-1 to e j The smooth term of the trajectory, This means that in e j-1 to e j The distance at which the movement path violates the safety threshold of the obstacle.

[0099] Example 2:

[0100] Based on the above embodiment 1, a robot specifically applied to concrete strength testing is proposed;

[0101] See Figure 2 To set up the robot's underlying skill library;

[0102] See Figure 3 - Figure 4 Based on the BIM information, a test area is obtained as follows. According to existing specifications, 12 test points can be set for each test area, and the order of the test points is as follows.

[0103] After the strength testing robot moves to the test area, the position of each test point in the robot coordinate system is calculated based on the distance from the test point to the edge of the wall. According to the order of the test points, the large model gives the robot's action sequence and performs dynamic checks.

[0104] Example 3:

[0105] Based on the above embodiment 1, a specific application to a bolt fastening robot is proposed;

[0106] See Figure 5 - Figure 6 To set up the robot's underlying skill library;

[0107] See Figure 7 This combination enables the robot to tighten bolts.

[0108] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for robot operation in a building scene based on large model technology, characterized in that: At least the following steps are included: S1: By screening basic robot movements and tasks, collecting trajectory data and using the Diffusion model to train the robot instruction model, a low-level instruction library is built to optimize robot movement execution; S1 includes at least the following steps: Based on the actual robot structure, basic robot movements and tasks are selected, and a low-level instruction library for the robot is constructed. Collect and clean trajectory data of robotic arm movements or tasks in real or simulated scenarios. For each selected movement or task, train a robot instruction model using a Diffusion model. ,in For the i-th underlying instruction, the formula is as follows: in, For the iteration step, Let t be the robot's action at time k-1 iterations. For the robot's observation at time t, For instructions Language description, This indicates that the expected value is 0 and the variance is . Gaussian distribution, For the robot's starting and ending poses, For parameters; The loss function for the robot instruction model is: in for The logarithmic gradient, The predicted action log gradient; S2: Combining BIM information and industry knowledge, identify workable areas in the building scene, and generate a specific set of work tasks through discretized work areas and path planning algorithms; S3: Utilize large language models to construct a task thinking chain; S4: Given task prompts, use a large language model to generate action instructions for each task, use the SimCSE model to find the most similar skill in the skill base, and execute the skill. The S4 includes at least the following steps: First, the robot uses its own sensors to perceive the environment and determine the positional relationship between the robot and the work area; For each action, geometric and dynamic functions are constructed based on the work point and work area to correct the smoothness and dynamic safety of the trajectory; For each task The coordinates of the work point are obtained through a three-dimensional sensor. Coordinates within the work area ,according to and The relative relationship is obtained. Coordinates within the work area The large model is used to break down the task into corresponding action instruction sets. ; For each action instruction in the job instruction set, the logarithmic gradient of the action estimated by the robot instruction model satisfies: in , ; The robot's movements can be calculated using the following formula: in For parameters S5: Corrects the robot's end effector trajectory to ensure the smoothness and safety of the robot's motion path, while adhering to the actuator's motion constraints.

2. The method for robot operation in a building scene based on large model technology according to claim 1, characterized in that: The robot's underlying instruction library includes at least instructions for actuator actions or task actions.

3. The method for robot operation in a building scene based on large model technology according to claim 1, characterized in that: The trajectory data includes at least robot pose and robot image data.

4. The method for robot operation in a building scene based on large model technology according to claim 1, characterized in that: S2 includes at least the following steps: By combining BIM information with prior industry knowledge, the operable areas of robots within the building scene can be identified. For each work area The discrete point path planning and task generation algorithm is used to generate a set of tasks in the work area. Each task includes a start point, an end point and the task content. The discrete point path planning and task generation algorithm includes at least the following steps: For each work area Take discrete points uniformly on the surface, denoted as . M is the number of discrete points. For discrete point poses; Connecting surface points using zig-zag or Hilbert curves then the task... ,in For the task.

5. A method for robot operation in a building scene based on large model technology according to claim 1, characterized in that: The operational thought chain in S3 includes at least trajectory information, action information, and historical information. The trajectory information represents the action that has been executed so far. The action information includes at least the robot's underlying instructions that can be used and the meaning of the instructions. The historical information includes at least the robot action information that was executed previously.

6. The method for robot operation in a building scene based on large model technology according to claim 1, characterized in that: The S5 includes at least the following steps: The obtained robot end-effector trajectory Make corrections; in: , Let N be the j-th pose point, and N be the number of poses; Therefore, the actuator action constraint function is as follows: in, for arrive The smooth term of the trajectory, This means that in arrive The distance at which the movement path violates the safety threshold of the obstacle.

Images