Robot task execution code automatic generation method and related equipment

Abstract

The invention provides a robot task execution code automatic generation method and related equipment, and the method comprises the steps: inputting a task starting point, a task ending point and a target path point set into a multi-modal language generation model, and enabling the language generation model to generate a task execution code corresponding to a robot under a preset constraint condition by taking a primitive library as a benchmark; generating a task path simulation diagram corresponding to the task execution code; and when the path error between the task path simulation diagram and the target path point set is within the tolerance range, taking the current task execution code as the task execution code implemented by the robot. The task execution code corresponding to the robot is quickly generated through the language generation model, and verification is completed based on the path error between the task path simulation diagram and the target path point set, so that the accuracy of the task execution code is ensured, and the operation efficiency and safety of the robot are ensured.

Description

Technical Field

[0001] This invention relates to the field of robotics, and more specifically, to a method and related equipment for automatically generating robot task execution code. Background Technology

[0002] With the rapid advancement of industrialization, the demand for productivity enhancement is increasing daily, leading to an explosive growth in robots designed to boost productivity. These include handling robots, picking robots, food delivery robots, and tea-serving robots, among others. During operation, robots often need to complete tasks along pre-defined paths, making robot path planning a crucial aspect of their workflow. Once the robot path is determined, the challenge lies in quickly and accurately converting it into code that the robot can recognize and execute. This directly impacts the robot's operational efficiency and safety, and has become a significant challenge for those skilled in the art. Summary of the Invention

[0003] The purpose of this invention is to provide a method and related equipment for automatically generating robot task execution code, so as to improve the above-mentioned problems.

[0004] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows: In a first aspect, embodiments of the present invention provide a method for automatically generating robot task execution code, the method comprising: Input the task start point, task end point and target path point set into the multimodal language generation model. Under preset constraints, the language generation model generates the task execution code corresponding to the robot based on the primitive language library. The target path point set includes path nodes between the task start point and the task end point, and the primitive library includes various primitives corresponding to the robot's task execution. The primitives are used to indicate the robot's execution actions and action parameter values. Generate a task path simulation diagram corresponding to the task execution code; Obtain the path error between the task path simulation graph and the target path point set; When the path error is within the tolerable range, the current task execution code is used as the task execution code implemented by the robot.

[0005] Secondly, embodiments of the present invention provide an automatic robot task execution code generation device, the device comprising: The first processing unit is used to input the task start point, task end point and target path point set into the multimodal language generation model. Under preset constraints, the language generation model generates the task execution code corresponding to the robot based on the primitive language library. The target path point set includes path nodes between the task start point and the task end point, and the primitive library includes various primitives corresponding to the robot's task execution. The primitives are used to indicate the robot's execution actions and action parameter values. The first processing unit is also used to generate a task path simulation diagram corresponding to the task execution code; The first processing unit is also used to obtain the path error between the task path simulation map and the target path point set; The second processing unit is used to use the current task execution code as the task execution code implemented by the robot when the path error is within the tolerance range.

[0006] Thirdly, embodiments of the present invention provide a storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0007] Fourthly, embodiments of the present invention provide an electronic device, the electronic device comprising: a processor and a memory, the memory being used to store one or more programs; when the one or more programs are executed by the processor, the above-described method is implemented.

[0008] Compared to existing technologies, the present invention provides a method and related equipment for automatically generating robot task execution code. The method inputs the task start point, task end point, and target path point set into a multimodal language generation model. Under preset constraints, the language generation model generates task execution code corresponding to the robot, based on a primitive language library. It also generates a task path simulation diagram corresponding to the task execution code. When the path error between the task path simulation diagram and the target path point set is within a tolerable range, the current task execution code is used as the task execution code implemented by the robot. By rapidly generating task execution code corresponding to the robot through the language generation model and verifying the path error between the task path simulation diagram and the target path point set, the accuracy of the task execution code is ensured, thus protecting the robot's operational efficiency and safety.

[0009] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0012] Figure 2 This is one of the flowcharts illustrating the automatic generation method for robot task execution code provided in an embodiment of the present invention.

[0013] Figure 3 This is the second flowchart illustrating the automatic generation method for robot task execution code provided in this embodiment of the invention.

[0014] Figure 4 This is the third flowchart illustrating the automatic generation method for robot task execution code provided in this embodiment of the invention.

[0015] Figure 5 The fourth flowchart illustrates the automatic generation method for robot task execution code provided in this embodiment of the invention.

[0016] Figure 6 The fifth flowchart illustrates the automatic generation method for robot task execution code provided in this embodiment of the invention.

[0017] Figure 7 A schematic diagram of a robot task execution code automatic generation device provided in an embodiment of the present invention.

[0018] In the diagram: 10-Processor; 11-Memory; 12-Bus; 13-Communication interface; 501-First processing unit; 502-Second processing unit. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0020] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0021] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0022] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0023] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of this invention is usually placed when in use. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0024] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set" and "connection" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0025] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0026] This invention provides an electronic device, which can be a central control device for a robot, or a mobile phone, computer, or server device that is communicatively connected to the central control device. Please refer to... Figure 1 This is a schematic diagram of the structure of an electronic device. The electronic device includes a processor 10, a memory 11, and a bus 12. The processor 10 and the memory 11 are connected via the bus 12. The processor 10 is used to execute executable modules, such as computer programs, stored in the memory 11.

[0027] Processor 10 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the automatic generation method for robot task execution code can be completed through integrated logic circuits in the hardware or software instructions within processor 10. Processor 10 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0028] The memory 11 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage.

[0029] Bus 12 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. Figure 1 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus 12 or one type of bus 12.

[0030] The memory 11 is used to store programs, such as programs corresponding to a robot task execution code automatic generation device. The robot task execution code automatic generation device includes at least one software functional module that can be stored in the memory 11 in the form of software or firmware, or embedded in the operating system (OS) of the electronic device. Upon receiving an execution instruction, the processor 10 executes the program to implement the robot task execution code automatic generation method.

[0031] The electronic device provided in this embodiment of the invention may further include a communication interface 13. The communication interface 13 is connected to the processor 10 via a bus.

[0032] It should be understood that, Figure 1The structure shown is only a partial schematic diagram of the electronic device; the electronic device may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0033] The robot task execution code automatic generation method provided in this embodiment of the invention can be applied to, but is not limited to, [various applications]. Figure 1 For the specific process of the electronic devices shown, please refer to [link / reference]. Figure 2 The automatic generation method for robot task execution code includes: S201 to S206, which are described in detail below.

[0034] S201: Input the task start point, task end point and target path point set into the multimodal language generation model. Under preset constraints, the language generation model generates the task execution code corresponding to the robot based on the primitive language library.

[0035] The target path point set includes path nodes between the task start and end points. The primitive library includes various primitives corresponding to the robot's task execution. Primitives (the smallest action units) are used to indicate the robot's actions and action parameter values. The language generation model can be, but is not limited to, QWen3, chatgpt-4, etc. The task execution code can be, but is not limited to, Python code.

[0036] The preset constraints include path constraints, collision detection constraints, attitude consistency constraints, inverse kinematics constraints, and safety policy constraints, which are configured by the operator.

[0037] Path constraints reflect whether the trajectory during task execution meets requirements (e.g., maintaining horizontality during movement). Collision detection constraints ensure that the robot does not collide with other objects in the environment while performing the task. Posture consistency constraints ensure that the posture of the end effector (such as a gripper) remains consistent during movement, without sudden rotation or tilting. Inverse kinematics (IK) constraints ensure the rationality of the robot's joint angles, guaranteeing that the robot can actually execute the trajectory. Safety policy constraints trigger emergency stop conditions.

[0038] The following are some examples of possible forms of primitives, but the examples are not limited to these types.

[0039] The primitive MoveLinear(m, v_max, a_max, ε_p) represents linear movement, where the action parameters include the target point m, the maximum movement speed v_max, the maximum acceleration a_max, and the tolerance threshold ε_p for linear movement. The primitive RotateTo(R, ω_max, ε_R) represents attitude adjustment, where the action parameters include the target orientation R, the maximum turning angular velocity ω_max, and the tolerance threshold ε_R for attitude adjustment. The primitive Approach(m, v, ε_p) represents end-of-course deceleration adjustment, where the action parameters include the target point m, the approach velocity v, and the tolerance threshold ε_p for end-of-course deceleration adjustment. The primitive Align(n_r, n_o, ε_θ) represents the normal alignment action, where the action parameters include the robot end effector n_r, the target object normal n_o, and the tolerance threshold ε_θ. The primitive KeepUpright(θ_max) represents the trajectory-keeping action, where the action parameters include the upper limit of the tilt angle θ_max; The primitive Clearance(d_min) represents the action of maintaining a safe distance from the environment, and the action parameters include the minimum clearance d_min from obstacles; The primitives Grip(f) / Release() represent the grab / release action, where the action parameters include grab force / opening / closing width; The primitive Pour(α, t) represents the pouring action, where the action parameters include the tilt angle α and the duration t; The primitive Wait(Δt) represents a buffer action, where the action parameter includes the waiting time Δt; The primitive Impedance(Kd, Kp) represents compliance control during the contact period, and its action parameters include compliance parameters Kd and Kp.

[0040] S202, Generate a task path simulation diagram corresponding to the task execution code.

[0041] Because obstacle avoidance needs to be taken into account during execution, the task path simulation diagram is not a straight line connection between all adjacent points in the path. This may cause a deviation between the task path simulation diagram and the path nodes in the target path point set. Therefore, it is necessary to make corrections and confirmations.

[0042] S203, obtain the path error between the task path simulation map and the target path point set.

[0043] Optionally, the path node offset between each path node in the task path simulation graph and the path node corresponding to the target path point set is obtained, and the path error between the task path simulation graph and the target path point set is determined based on any one or more of the maximum value, average value, and median of the path node offset.

[0044] S204, determine whether the path error is within the tolerance range. If yes, proceed to S205; otherwise, proceed to S206.

[0045] S205, when the path error is within the tolerance range, the current task execution code is used as the task execution code implemented by the robot.

[0046] It should be understood that the task execution code can be executed when the path error is within the tolerance range, so the current task execution code is taken as the task execution code that the robot will ultimately implement.

[0047] S206, When the path error exceeds the tolerance range, determine the penalty coefficient corresponding to the language generation model based on the path error, so that the language generation model can be adjusted according to the penalty coefficient.

[0048] After the language generation model is adjusted, S201 is repeated to input the task start point, task end point and target path point set into the multimodal language generation model. Under preset constraints, the language generation model generates the task execution code corresponding to the robot based on the primitive language library.

[0049] The formula for the penalty coefficient is as follows:

[0050]

[0051] in, Indicates the penalty coefficient. This indicates the upper limit of the tolerance range. Indicates path error. This represents the slope coefficient (which determines the growth rate outside of ε, and its value can be between 10 and 50). This indicates the transition width (which can be, but is not limited to, 1 mm at position and 0.5 at angle). This represents the reciprocal of the penalty function.

[0052] The penalty function is designed as a piecewise mechanism, divided into a zero zone, a buffer zone, and a linear penalty interval. This applies when the path error is within the tolerance range. When the path error exceeds the tolerance range, corresponding to the zero zone, the penalty coefficient is 0. The generated result receives no penalty and is considered error-free. When the path error exceeds the tolerance range, the corresponding buffer is activated. When, it corresponds to the linear penalty interval.

[0053] Referring to the reciprocal of the penalty function, it can be seen that the above mechanism ensures that there is no penalty within the tolerance range, while smoothly transitioning to the linear penalty interval to avoid gradient explosion.

[0054] In the automatic generation method for robot task execution code provided in this embodiment of the invention, the task execution code corresponding to the robot is quickly generated through a language generation model, and the verification is completed based on the path error between the task path simulation diagram and the target path point set to ensure the accuracy of the task execution code and protect the robot's operating efficiency and safety.

[0055] Based on the preceding text, regarding the content in S201, this embodiment of the invention also provides an optional implementation method, please refer to the following text. S201 involves inputting the task start point, task end point, and target path point set into a multimodal language generation model. Under preset constraints, the language generation model generates task execution code corresponding to the robot based on a primitive language library, including: S201A and S201B, which are described in detail below.

[0056] S201A, the language generation model, under preset constraints, uses the primitive language library as a benchmark to generate sub-path task code corresponding to each segment sub-path.

[0057] Wherein, the first sub-path is the path from the starting point of the task to the first path node in the target path point set, the (j+1)th sub-path is the path from the jth path node in the target path point set to the (j+1)th path node, 1≤j≤J-1, J is the total number of path nodes in the target path point set, and the (J+1)th sub-path is the path from the jth path node in the target path point set to the end point of the task.

[0058] The following is an example of the code for the sub-path task corresponding to the (j+1)th sub-path: stage j+1 { MoveLinear(m=Gj+1, v=0.20, a=0.50, eps=ε_p) / / Move KeepUpright(θ_max=5°) / / Keep the kettle angle Clearance(d_min=5mm) / / Slow down when approaching the target point } S201B concatenates the sub-path task codes corresponding to each segment sub-path to obtain the task execution code corresponding to the robot.

[0059] By using primitives as the smallest unit for generating constraints, code reusability for the same tasks within the same scenario is improved. Furthermore, the primitive library can be expanded to include custom primitives to adapt to more specialized scenarios.

[0060] Please refer to Figure 3 Before inputting the task start point, task end point, and target path point set into the multimodal language generation model, the automatic generation method for robot task execution code also includes: S101 to S104, which are described in detail below.

[0061] S101 inputs a panoramic view of the robot's work site into the visual model so that it can generate multiple sets of key points based on multiple sets of random variable factors (including random seed, temperature and cue fine granularity).

[0062] The key point set includes path nodes between the task start point and the task end point. The number of path nodes is greater than or equal to 1. Taking the tea pouring task as an example, the task start point can be the point where the teapot handle is located, and the task end point can be the tea pouring position corresponding to the teacup. Gij represents the j-th path node in the i-th key point set.

[0063] The panoramic view of the work site can be, but is not limited to, RGB-D, and can be, but is not limited to, captured by a camera deployed on the robot's head. The visual model LVM can be, but is not limited to, DINOv2. Random variable factors include random seed, temperature, and cue granularity; temperature is a key parameter controlling the randomness and determinism of the model when generating text.

[0064] S102, construct the direct connection path corresponding to each set of key points.

[0065] A direct path is a path formed by connecting adjacent nodes in a set of key points with straight lines. S103, filter multiple sets of key points to obtain N sets of key points that meet the filtering rules.

[0066] When N≥1 or N>2, path switching is allowed midway, which will be discussed later. The selection criteria are that the paths do not intersect and the minimum gap between the paths is greater than the preset safety margin.

[0067]

[0068] in, This represents the direct path corresponding to the q-th set of key points. This represents the direct path corresponding to the p-th set of key points. express and The minimum gap between them 1 indicates a safety margin, which is ≥5 mm.

[0069] S104: Select any one set from the N sets of key points as the target path point set.

[0070] Please refer to Figure 4 In the process of robot executing task code, the automatic generation method of robot task execution code also includes: S211 and S212, as follows.

[0071] S211, when the robot reaches the kth path node in the target path point set, start timing. If the robot has not reached the (k+1)th path node after the kth time tolerance value has been exceeded, obtain the remaining arrival time corresponding to each candidate group.

[0072] The candidate group is any group other than the target path point set among the N key point sets, or any group among the N key point sets whose reachability of the (k+1)th path node has not been verified. Clearly, the key point sets that have already been used as the target path point set at the current stage have all had their (k+1)th path node verified as unreachable. 1 ≤ k ≤ J-1.

[0073] The formula for the remaining arrival time corresponding to the i-th candidate group is:

[0074] in, This represents the remaining arrival time for the i-th candidate group. When h = k + 1, This represents the path from the current position to the (k+1)th node in the i-th candidate group, where k+1 ≤ h. This represents the path from the k-th path node in the i-th candidate group to the (k+1)-th path node in the i-th candidate group, where k+1≤h≤J-1.

[0075] The formula for the k-th time tolerance value is:

[0076] in, This represents the k-th time tolerance value, that is, the time tolerance value corresponding to the k-th path node. This represents the path length between the k-th path node and the (k+1)-th path node in the target path node set. This indicates the robot's set or average moving speed. It is a time constant greater than or equal to 1, and its value can be, but is not limited to, 1.5.

[0077] S212, take the robot's current position as the new task starting point, and take the (k+1)th path node to the Jth path node in the candidate group with the shortest remaining arrival time as the new target path point set.

[0078] At this point, the task endpoint remains unchanged. S201 is executed repeatedly, inputting the task start point, task endpoint, and target path point set into the multimodal language generation model. Under preset constraints, the language generation model generates the task execution code corresponding to the robot based on the primitive language library.

[0079] In one alternative implementation, when executing S212, the first k path nodes in the N sets of key points can be deleted simultaneously.

[0080] In the robot task execution code automatic generation method provided in the embodiments of the present invention, by setting up candidate groups and switching paths when a certain path node is unreachable, the success rate of task execution is improved and the fault tolerance is greatly enhanced.

[0081] Please refer to Figure 5 In the process of robot executing task code, the automatic generation method of robot task execution code also includes: S221, S222 and S223, as follows.

[0082] S221, Obtain the real-time first-type state value according to the preset interval.

[0083] The first type of state value includes one or more of position error, obstacle gap, and normal error. The first type of state value and the second type of state value can be identified by the image obtained by the camera deployed on the robot's hand.

[0084] Position error represents the deviation between the robot's actual position and its expected position in the task path simulation diagram at the current time (the time of acquiring the first type of state value). The corresponding state threshold can be 3mm.

[0085] Normal error represents the deviation between the robot's actual normal and the expected normal in the task path simulation diagram at the current time, and its corresponding state threshold can be 2°.

[0086] The obstacle gap represents the shortest distance between the robot's actual location and the obstacle, and its corresponding state threshold can be 5mm.

[0087] S222: Based on the first type of state value and its corresponding state threshold, determine whether the conditions for continuing execution are met. If yes, continue; otherwise, execute S223.

[0088] Among them, the state threshold corresponding to the first type of state value is a fixed value.

[0089] The conditions for continuing execution include: the position error being less than the corresponding state threshold, the normal error being less than the corresponding state threshold, and the obstacle gap being greater than the corresponding state threshold.

[0090] S223, take the robot's current position as the new starting point of the task, and take the path nodes in the target path point set that are after the current position as the new target path point set.

[0091] Repeat step S201, inputting the task start point, task end point and target path point set into the multimodal language generation model. Under preset constraints, the language generation model generates the task execution code corresponding to the robot based on the primitive language library.

[0092] Please refer to Figure 6 In the process of executing the task code implemented by the robot, the automatic generation method of robot task execution code also includes: S231 to S235, as follows.

[0093] S231, acquire real-time second-type state values ​​at preset intervals, the second-type state values ​​include attitude error.

[0094] The attitude error represents the deviation between the robot's actual attitude at the current time and its expected attitude at the current time (the time of acquiring the second type of state value) in the task path simulation diagram. The initial dynamic state threshold can be 2°.

[0095] S232, determine whether the value of the second type of state is less than its corresponding dynamic state threshold. If yes, continue; if no, execute S233.

[0096] Among them, the state threshold corresponding to the second type of state value is a dynamically adjusted value.

[0097] S233, Adjust the dynamic state threshold based on the second type of state values ​​obtained during the observation period prior to the current time.

[0098] The observation period for the second type of state value can be 3 to 5 seconds, and the sampling interval can be 20-50 ms.

[0099] The formula for adjusting the dynamic state threshold is:

[0100] in, This represents the adjusted dynamic state threshold. Represents a constant. This represents the dynamic state threshold before adjustment. express The 90th percentile value, This represents the set of second-class state values ​​within the observation period.

[0101] S234, Based on the adjusted dynamic state threshold, determine whether the second type of state value is less than its corresponding dynamic state threshold. If yes, continue; if no, execute S235.

[0102] S235, take the robot's current position as the new starting point of the task, and take the path nodes in the target path point set that are after the current position as the new target path point set.

[0103] Repeat step S201, inputting the task start point, task end point and target path point set into the multimodal language generation model. Under preset constraints, the language generation model generates the task execution code corresponding to the robot based on the primitive language library.

[0104] By setting dynamic state thresholds, i.e. adjusting the error tolerance range, the convergence ability of the loss function corresponding to the large model is improved, the computational efficiency is increased, and the risk of overfitting is reduced, making the algorithm more adaptable to real-world application scenarios.

[0105] In one alternative implementation, after the task is completed, the safety correlation coefficient (the minimum distance between the number of collisions and environmental obstacles) is evaluated to determine if the target is reachable. The number of collisions and execution time are then recalculated, and the experience base is updated based on the task performance. The latest dynamic error value is recorded for the next round of error mitigation calculation in the language generation model.

[0106] Please see Figure 7 , Figure 7 The present invention provides an automatic robot task execution code generation device, which is optionally applied to the electronic device described above.

[0107] The automatic generation device for robot task execution code includes: a first processing unit 501 and a second processing unit 502.

[0108] The first processing unit 501 is used to input the task start point, task end point and target path point set into the multimodal language generation model. Under preset constraints, the language generation model generates the task execution code corresponding to the robot based on the primitive language library. The target path point set includes path nodes between the task start point and the task end point, and the primitive library includes various primitives corresponding to the robot's task execution. The primitives are used to indicate the robot's execution actions and action parameter values. The first processing unit 501 is also used to generate a task path simulation diagram corresponding to the task execution code; The first processing unit 501 is also used to obtain the path error between the task path simulation map and the target path point set; The second processing unit 502 is used to use the current task execution code as the task execution code implemented by the robot when the path error is within the tolerance range.

[0109] The second processing unit 502 can execute S205 as described above, and the first processing unit 501 can execute other steps in the above method embodiment.

[0110] It should be noted that the robot task execution code automatic generation device provided in this embodiment can execute the method flow shown in the above method flow embodiment to achieve the corresponding technical effects. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the above embodiments.

[0111] This invention also provides a storage medium storing computer instructions and programs, which, when read and run, execute the robot task execution code automatic generation method described above. The storage medium may include memory, flash memory, registers, or a combination thereof.

[0112] The following provides an electronic device, which can be a central control device for a robot, or a mobile phone, computer, or server device that communicates with the central control device. This electronic device, for example... Figure 1 As shown, the above-described method for automatically generating robot task execution code can be implemented. Specifically, the electronic device includes: a processor 10, a memory 11, and a bus 12. The processor 10 may be a CPU. The memory 11 is used to store one or more programs, which, when executed by the processor 10, execute the robot task execution code automatic generation method of the above embodiment.

[0113] In summary, the robot task execution code automatic generation method and related equipment provided by this invention input the task start point, task end point, and target path point set into a multimodal language generation model. Under preset constraints, the language generation model generates task execution code corresponding to the robot based on a primitive language library; it also generates a task path simulation diagram corresponding to the task execution code; and when the path error between the task path simulation diagram and the target path point set is within a tolerable range, the current task execution code is used as the task execution code implemented by the robot. By rapidly generating task execution code corresponding to the robot through the language generation model and verifying the path error between the task path simulation diagram and the target path point set, the accuracy of the task execution code is ensured, thus protecting the robot's operating efficiency and safety.

[0114] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0115] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for automatically generating robot task execution code, characterized in that, The method includes: Input the task start point, task end point and target path point set into the multimodal language generation model. Under preset constraints, the language generation model generates the task execution code corresponding to the robot based on the primitive language library. The target path point set includes path nodes between the task start point and the task end point, and the primitive library includes various primitives corresponding to the robot's task execution. The primitives are used to indicate the robot's execution actions and action parameter values. Generate a task path simulation diagram corresponding to the task execution code; Obtain the path error between the task path simulation graph and the target path point set; When the path error is within the tolerable range, the current task execution code is used as the task execution code implemented by the robot.

2. The automatic generation method for robot task execution code as described in claim 1, characterized in that, The method further includes: When the path error exceeds the tolerance range, a penalty coefficient corresponding to the language generation model is determined based on the path error, so that the language generation model can be adjusted according to the penalty coefficient; After the language generation model is adjusted, the task start point, task end point and target path point set are repeatedly input into the multimodal language generation model. Under preset constraints, the language generation model generates the task execution code corresponding to the robot based on the primitive language library.

3. The automatic generation method for robot task execution code as described in claim 1, characterized in that, The process involves inputting the task start point, task end point, and target path point set into a multimodal language generation model. Under preset constraints, the language generation model generates task execution code corresponding to the robot based on a primitive language library, including: Under preset constraints, the language generation model generates sub-path task codes corresponding to each sub-path based on the primitive language library. The first sub-path is the path from the task start point to the first path node in the target path point set, the (j+1)th sub-path is the path from the j-th path node in the target path point set to the (j+1)-th path node, 1≤j≤J-1, where J is the total number of path nodes in the target path point set, and the (J+1)th sub-path is the path from the j-th path node in the target path point set to the task end point. The sub-path task codes corresponding to each segment of the sub-path are concatenated to obtain the task execution code corresponding to the robot.

4. The automatic generation method for robot task execution code as described in claim 1, characterized in that, Before inputting the task start point, task end point, and target path point set into the multimodal language generation model, the method further includes: A panoramic view of the robot's work site is input into the visual model so that it generates multiple sets of key points based on multiple sets of random variable factors. The sets of key points include path nodes between the task start point and the task end point. Construct a direct connection path corresponding to each set of key points. The direct connection path is the path formed by connecting adjacent nodes in the set of key points with straight lines. Multiple sets of key points are filtered to obtain N sets of key points that meet the filtering rules, where N≥1. The filtering rules are that the paths do not intersect and the minimum gap between the paths is greater than a preset safety margin. Choose any one set from the N sets of key points as the target path point set.

5. The automatic generation method for robot task execution code as described in claim 4, characterized in that, The method further includes the following during the execution of task code by the robot: When the robot reaches the kth path node in the target path point set, the timer starts. If the k+1th path node is not reached after the kth time tolerance value is exceeded, the remaining arrival time corresponding to each candidate group is obtained. The candidate group is any group other than the target path point set in the N sets of key points, 1≤k≤J-1. Take the robot's current position as the new starting point of the task, and take the (k+1)th path node to the Jth path node in the candidate group with the shortest remaining arrival time as the new set of target path nodes. The task start point, task end point, and target path point set are repeatedly input into the multimodal language generation model. Under preset constraints, the language generation model generates the task execution code corresponding to the robot based on the primitive language library.

6. The method for automatically generating robot task execution code as described in claim 1, characterized in that, The method further includes the following during the execution of task code by the robot: The first type of state value is acquired in real time at a preset interval. The first type of state value includes any one or more of position error, obstacle gap, and normal error. Based on the first type of state value and its corresponding state threshold, determine whether the conditions for continuing execution are met; If not, the robot's current position is taken as the new task starting point, and the path nodes after the current position in the target path point set are taken as the new target path point set. The task starting point, task ending point, and target path point set are repeatedly input into the multimodal language generation model. Under preset constraints, the language generation model generates the corresponding task execution code for the robot based on the primitive language library.

7. The method for automatically generating robot task execution code as described in claim 1, characterized in that, The method further includes the following during the execution of task code by the robot: The second type of state value is acquired in real time at a preset interval, and the second type of state value includes attitude error; Determine whether the second type of state value is less than its corresponding dynamic state threshold; If not, the dynamic state threshold is adjusted based on the second type of state value obtained during the observation period prior to the current time. Based on the adjusted dynamic state threshold, determine whether the second type of state value is less than its corresponding dynamic state threshold; If not, the robot's current position is taken as the new task starting point, and the path nodes after the current position in the target path point set are taken as the new target path point set. The task starting point, task ending point, and target path point set are repeatedly input into the multimodal language generation model. Under preset constraints, the language generation model generates the corresponding task execution code for the robot based on the primitive language library.

8. A device for automatically generating robot task execution code, characterized in that, The device includes: The first processing unit is used to input the task start point, task end point and target path point set into the multimodal language generation model. Under preset constraints, the language generation model generates the task execution code corresponding to the robot based on the primitive language library. The target path point set includes path nodes between the task start point and the task end point, and the primitive library includes various primitives corresponding to the robot's task execution. The primitives are used to indicate the robot's execution actions and action parameter values. The first processing unit is also used to generate a task path simulation diagram corresponding to the task execution code; The first processing unit is also used to obtain the path error between the task path simulation graph and the target path point set; The second processing unit is used to use the current task execution code as the task execution code implemented by the robot when the path error is within the tolerance range.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1-7.

10. An electronic device, characterized in that, include: Processor and memory, the memory being used to store one or more programs; When the one or more programs are executed by the processor, the method as described in any one of claims 1-7 is implemented.

Images