Robot task execution adjustment method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a task execution adjustment method and device of a robot, an electronic device and a storage medium, the method comprising: in response to the robot running, determining an alignment result of first vision data of the robot and a control instruction sequence, the control instruction sequence being used to control the robot to execute a task; analyzing a task execution process of the robot according to the alignment result to obtain an item to be adjusted in the task execution process of the robot; querying knowledge information related to the item to be adjusted from a knowledge base according to information associated with the item to be adjusted; and generating a target adjustment scheme based on the knowledge information, the target adjustment scheme being used to adjust the item to be adjusted. The method can automatically identify the problem that needs to be adjusted in the task execution process of the robot according to the alignment result, and automatically generate an adjustment scheme, thereby realizing accurate adjustment of the robot task execution process in all dimensions.

Description

Technical Field

[0001] This disclosure relates to the field of equipment control, and in particular to a method, apparatus, electronic device, and storage medium for adjusting the task execution of a robot. Background Technology

[0002] Industrial robot path optimization is a core means to improve the efficiency of intelligent manufacturing production lines and reduce energy consumption. Currently, path optimization technologies are mainly divided into two categories: offline optimization based on code logic and online optimization based on sensor data. Offline optimization only analyzes the logic and parameters of the robot control code to propose path adjustment suggestions; online optimization relies on data from a single sensor (such as a position sensor). Both suffer from problems such as optimization suggestions being out of touch with the actual scenario, lack of reliable data support, and poor knowledge reusability. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, and storage medium for adjusting robot task execution, which can achieve accurate optimization of the robot task execution process in all dimensions.

[0004] The first aspect of this disclosure provides a method for adjusting the task execution of a robot. The method includes: in response to robot operation, determining an alignment result of the robot's first visual data and a sequence of control commands, wherein the sequence of control commands is used to control the robot to perform a task; analyzing the robot's task execution process based on the alignment result to obtain items to be adjusted during the task execution process; querying knowledge information related to the items to be adjusted from a knowledge base based on information associated with the items to be adjusted; and generating a target adjustment scheme based on the knowledge information, wherein the target adjustment scheme is used to adjust the items to be adjusted.

[0005] In some embodiments of this disclosure, the robot's task execution process is analyzed based on the alignment results to obtain the adjustment items to be adjusted during the task execution process. This includes: determining the efficiency parameters of the robot during the task execution process based on the alignment results, wherein the efficiency parameters include at least one of time efficiency parameters, energy consumption efficiency parameters, and accuracy parameters; analyzing the robot's task execution process based on the efficiency parameters to obtain the adjustment items to be adjusted during the task execution process; the adjustment items include at least one of explicit adjustment items and implicit adjustment items, wherein the explicit adjustment items are adjustment items associated with the control command sequence, and the implicit adjustment items are adjustment items associated with the robot's actual execution state and / or the robot's operating environment.

[0006] In some embodiments of this disclosure, the time efficiency parameters include at least one of the following: the execution time of a single action performed by the robot during task execution, the overall path loop time corresponding to the task execution process, the proportion of robot waiting time during task execution, and the redundant time consumed by the robot when performing actions during task execution; the energy efficiency parameters include at least one of the following: the loss of the robot's moving mechanism during task execution, the total energy consumption of the robot during task execution, and the proportion of energy consumption of the robot's invalid motion during task execution; the accuracy parameters include at least one of the following: the positional deviation of the robot's actions during task execution, the time deviation between the start of execution of the control command sequence and the start of the robot's actions, and the similarity between the action features corresponding to the control command sequence and the action features corresponding to the visual data.

[0007] In some embodiments of this disclosure, the robot's task execution process is analyzed based on efficiency parameters to obtain items to be adjusted during the task execution process, including: determining the deviation of the efficiency parameters from standard efficiency parameters; determining the priority of the efficiency parameters as a first priority in response to the deviation being greater than a first threshold; or determining the priority of the efficiency parameters as a second priority in response to the deviation being less than or equal to the first threshold and greater than or equal to a second threshold; or determining the priority of the efficiency parameters as a third priority in response to the deviation being less than the second threshold, wherein the first priority is higher than the second priority and the third priority is lower than the second priority; and analyzing the robot's task execution process according to the priority of the efficiency parameters to obtain items to be adjusted during the task execution process.

[0008] In some embodiments of this disclosure, the knowledge information related to the item to be adjusted includes at least one of the following: safety constraint information, path adjustment strategy, standard process parameters, and historical adjustment cases; based on the knowledge information, a target adjustment scheme is generated.

[0009] In some embodiments of this disclosure, the information associated with the item to be adjusted includes at least one of the following: the type of the item to be adjusted; the first visual data corresponding to the item to be adjusted; the control command sequence corresponding to the item to be adjusted; the efficiency parameter corresponding to the item to be adjusted; the priority of the item to be adjusted, wherein the priority of the item to be adjusted is related to the priority of the efficiency parameter corresponding to the item to be adjusted; the type of robot; and the operating environment of the robot.

[0010] In some embodiments of this disclosure, the target adjustment scheme includes at least one of the following: modifying the control command sequence; modifying the parameters of the target device, the target device including at least one of the following: a robot, a device in the robot's operating environment, or a device that collaborates with the robot during task execution; modifying the process parameters associated with the robot; and outputting prompt information to indicate at least one of the following: the control command sequence needs to be modified, the parameters of the target device need to be modified, or the process parameters associated with the robot need to be modified.

[0011] In some embodiments of this disclosure, the prompt information includes at least one of the following: information associated with the item to be adjusted; the adjusted control command sequence; the position of the adjusted control command sequence; the control command sequence before adjustment; the basis for modification; and the expected adjustment effect.

[0012] In some embodiments of this disclosure, the method further includes: generating task requirement information based on information associated with the item to be adjusted; generating prompt words based on at least one of the task requirement information, information in a knowledge base, and information in a rule base, wherein the rule base includes rules for generating control instruction sequences; and calling the target model to generate an adjusted control instruction sequence based on the prompt words and structured constraint information.

[0013] In some embodiments of this disclosure, the method further includes: acquiring multiple technical documents related to the task execution process, the multiple technical documents including at least one of the following: a robot programming language instruction reference manual, a robot operation manual, a detailed explanation of robot control instructions, an application skills document, a signal configuration instruction document, and a tool coordinate setting instruction document; and constructing a knowledge base based on the multiple technical documents.

[0014] In some embodiments of this disclosure, a knowledge base is constructed based on multiple technical documents, including: constructing a skill definition file, which is used to define the programming process specifications for the robot; and constructing a knowledge base based on the skill definition file and multiple technical documents.

[0015] In some embodiments of this disclosure, the method further includes: obtaining coding rule information of the robot, the coding rule information including rules for generating control instruction sequences; and constructing a rule base based on the coding rule information.

[0016] In some embodiments of this disclosure, the encoding rule information includes at least one of the following: security priority rules; reliability rules; maintainability rules; efficiency rules; program structure specifications; data type classification; naming specifications; motion control instruction format specifications; input and output signal specifications; security protection specifications; and error handling specifications.

[0017] In some embodiments of this disclosure, the structured constraint information includes at least one of the following: loop logic constraint information; speed optimization constraint information; control instruction sequence conciseness constraint information; safe motion path constraint information; variable scope constraint information; coordinate system constraint information; error handling constraint information; and control instruction sequence output format constraint information.

[0018] In some embodiments of this disclosure, the method further includes at least one of the following: obtaining information from a locally stored knowledge base; obtaining information from the knowledge base via a command-line tool.

[0019] In some embodiments of this disclosure, the method further includes: generating a call log, which includes at least one of the following: information about the target model, the time of calling the target model, information related to the prompt words, and an adjusted sequence of control instructions.

[0020] In some embodiments of this disclosure, the method further includes displaying an adjusted sequence of control commands on a display interface.

[0021] In some embodiments of this disclosure, the method further includes: automatically saving the adjusted control command sequence; or saving the adjusted control command sequence in response to a user's save command.

[0022] In some embodiments of this disclosure, the method further includes: in response to a user's confirmation of the adjusted control command sequence, controlling the robot to perform a task based on the adjusted control command sequence.

[0023] In some embodiments of this disclosure, in response to a user's confirmation of the adjusted control command sequence, the robot is controlled to perform a task based on the adjusted control command sequence, including: performing a security check on the adjusted control command sequence; and if the security check passes, in response to a user's confirmation of the adjusted control command sequence, the robot is controlled to perform a task based on the adjusted control command sequence.

[0024] In some embodiments of this disclosure, the method further includes: during the process of controlling the robot to perform a task based on the adjusted control command sequence, collecting second visual data of the robot's operation and execution logs of the adjusted control command sequence; determining the adjustment effect of the item to be adjusted based on the second visual data and execution logs; and processing at least one of the information associated with the item to be adjusted, the information of the target adjustment scheme, and the adjustment effect into an adjustment case and saving it to a knowledge base.

[0025] In some embodiments of this disclosure, determining the alignment result of the robot's first visual data and control command sequence includes: performing motion detection on the first visual data to identify at least one event log corresponding to the first visual data; parsing the control command sequence to obtain at least one control command log corresponding to the control command sequence; and aligning the at least one event log and the at least one control command log according to the time information and motion characteristics of the at least one event log and the time information and motion characteristics of the at least one control command log to obtain an alignment result.

[0026] A second aspect of this disclosure provides a robot task execution adjustment device, comprising: a determining module, configured to determine, in response to robot operation, an alignment result of first visual data of the robot and a sequence of control commands, the control command sequence being used to control the robot to execute a task; an analyzing module, configured to analyze the robot's task execution process based on the alignment result to obtain items to be adjusted during the task execution process; a querying module, configured to query knowledge information related to the items to be adjusted from a knowledge base based on information associated with the items to be adjusted; and a processing module, configured to generate a target adjustment scheme based on the knowledge information, the target adjustment scheme being used to adjust the items to be adjusted.

[0027] In some embodiments of this disclosure, the analysis module is further configured to: determine the efficiency parameters of the robot during task execution based on the alignment results, wherein the efficiency parameters include at least one of time efficiency parameters, energy efficiency parameters, and accuracy parameters; analyze the robot's task execution process based on the efficiency parameters to obtain the adjustment items to be adjusted during the task execution process; the adjustment items include at least one of explicit adjustment items and implicit adjustment items, wherein the explicit adjustment items are adjustment items associated with the control command sequence, and the implicit adjustment items are adjustment items associated with the robot's actual execution state and / or the robot's operating environment.

[0028] In some embodiments of this disclosure, the knowledge information related to the item to be adjusted includes at least one of the following: safety constraint information, path adjustment strategy, standard process parameters, and historical adjustment cases; based on the knowledge information, a target adjustment scheme is generated.

[0029] A third aspect of this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the methods described in the first aspect of this disclosure.

[0030] A fourth aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described in the first aspect of this disclosure.

[0031] In summary, the robot task execution adjustment method proposed in this disclosure can identify items to be adjusted during task execution based on the alignment results of the robot's visual data and control command execution sequence. It accurately determines the areas to be adjusted due to unreasonable control command sequence settings, actual operating environment, and deviations in robot execution performance, thereby improving the comprehensiveness and accuracy of item identification. Subsequently, targeted adjustment schemes are generated for the accurately identified items, enabling automatic adjustment of the robot's task execution process in actual operating scenarios and improving the efficiency and reliability of task execution process adjustment.

[0032] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0034] Figure 1 A flowchart illustrating a robot task execution adjustment method provided in this embodiment of the disclosure. Figure 1 ; Figure 2 A flowchart illustrating a robot task execution adjustment method provided in this embodiment of the disclosure. Figure 2 ; Figure 3 A flowchart illustrating a robot task execution adjustment method provided in this embodiment of the disclosure. Figure 3 ; Figure 4 A flowchart illustrating a robot task execution adjustment method provided in this embodiment of the disclosure. Figure 4 ; Figure 5 This is a schematic diagram of the structure of a robot task execution adjustment device provided in an embodiment of the present disclosure; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0035] Embodiments of this disclosure are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0036] Industrial robot path optimization is a core means to improve the efficiency of intelligent manufacturing production lines and reduce energy consumption. Currently, path optimization technologies are mainly divided into two categories: offline optimization based on code logic and online optimization based on sensor data. Offline optimization only analyzes the logic and parameters of the robot control code to propose path adjustment suggestions; online optimization relies on data from a single sensor (such as a position sensor). Both suffer from problems such as optimization suggestions being out of touch with the actual scenario, lack of reliable data support, and poor knowledge reusability.

[0037] Therefore, to solve the above problems, this disclosure proposes a method for adjusting the task execution of a robot. The solution of this disclosure can be executed by an electronic device, such as a robot. The specific content of the method is as follows.

[0038] Figure 1 A flowchart illustrating a robot task execution adjustment method provided in this embodiment of the disclosure. Figure 1 .like Figure 1 As shown, the method may include the following steps.

[0039] Step 101: Determine the alignment result of the robot's first visual data and control command sequence during operation.

[0040] In some embodiments, in response to robot operation, the alignment result of the robot's first visual data and control command sequence is determined.

[0041] In some embodiments, the robot is an industrial robot, such as a robot used for tasks such as welding, assembly, handling, palletizing, spraying, gluing, grinding, polishing, or sorting.

[0042] In some embodiments, the visual data may be video data of the robot's operation, such as video of the robot's robotic arm. Optionally, it may be video data of the robot's task execution process collected by an image acquisition device deployed in the robot's work area. It may include visual data of the robot body, visual data of the robot's active joints, visual data of the robot's end effector, visual data of the robot's operating environment, etc.

[0043] In some embodiments, the sequence of control instructions is used to control the operation of the robot. The sequence of control instructions may be, for example, control code, instruction code, etc., used to control the operation of the robot.

[0044] Among them, the control command sequence is used to control the robot to perform tasks; the robot's task execution process can also be the process of the robot performing tasks, such as the handling task process, the assembly task process, etc.

[0045] In some embodiments, determining the alignment result of the robot's first visual data and control command sequence during operation includes: performing motion detection on the first visual data to identify at least one event log corresponding to the first visual data; parsing the control command sequence to obtain at least one control command log corresponding to the control command sequence; and aligning the at least one event log and the at least one control command log according to the time information and motion characteristics of the at least one event log and the time information and motion characteristics of the at least one control command log to obtain an alignment result.

[0046] In some embodiments, when aligning visual data and control command sequences, the visual data and control command sequences can be analyzed (processed / parsed) first. For example, the first visual data can be analyzed first to extract the event anchors contained in the first visual data. Each event anchor corresponds to an event, and the event type includes at least one of the following: robot moves to a safe position (Robot_Safe_Pos); robot moves to a picking position (Move_To_Pick); robot performs a gripping action (Robot_Grasp); robot leaves the picking position (Move_From_Pick); robot moves to a placing position (Move_To_Place); robot leaves the placing position (Move_From_Place); robot performs an end effector rotation (Rotate_End_Effector).

[0047] Specifically, an event log is generated for each event anchor point. The event log includes the type of the event mentioned above, and may also include timestamps and time spans. For example, if the robot moves to a safe position between video frame 10 and video frame 20, the event log may include the acquisition timestamp of video frame 10, the acquisition timestamp of video frame 20, the time span between video frame 10 and video frame 20, and so on.

[0048] In some embodiments, the control instruction sequence can be analyzed (processed / parsed) to obtain control instruction anchors. For example, the control instruction code can be parsed to extract executable code blocks and generate structured code data containing instruction type, core parameters, line number range, and execution time window as control instruction anchors.

[0049] In some embodiments, after determining the event anchor and the control command anchor, hierarchical anchor matching can be performed to match the event anchor and the control command anchor. For example, matching can be performed first based on the time of the event anchor and the control command anchor, matching event anchors and control command anchors that are close in time. Then, matching can be performed based on the action characteristics of the event anchor and the control command anchor, matching event anchors and control command anchors with the same action characteristics, and finally obtaining the alignment result. The alignment result can include the correspondence between the event anchor and the control command anchor, and can also include the event log information corresponding to the time anchor, and can include the information of the control command log corresponding to the control command anchor. The control command log can include, for example, the control command sequence (or code block) corresponding to the control command anchor, such as the line number, range, etc. of the corresponding code. The control command log can also include the execution timestamp, time span, confidence of alignment between the event anchor and the control command anchor, action characteristics, etc., corresponding to the control command sequence.

[0050] In some embodiments, determining the alignment result of the first visual data and control command sequence of the robot during operation includes: synchronizing the first visual data and control command sequence with timestamps; and determining the alignment result of the first visual data and control command sequence of the robot during operation.

[0051] The time-stamp synchronization of the first visual data and the control command sequence can be achieved by synchronizing the time of the device acquiring the first visual data and the device acquiring the control command sequence; or by synchronizing the acquisition timestamp of the first visual data and the execution timestamp of the control command sequence according to a unified time base; optionally, the time-stamp synchronization of the first visual data and the control command sequence can be based on the Network Time Protocol (NTP). When aligning the first visual data and the control command sequence, they can be aligned based on the synchronized time information.

[0052] Optionally, the alignment results may also include production line process parameters, robot equipment parameters, etc. Among them, production line process parameters refer to parameters related to the production process of the production line in the robot's operating environment. Production line process parameters may include, for example, the speed of the production line and the layout of the production line. Robot equipment parameters refer to performance parameters related to the robot body and its associated equipment. Robot equipment parameters may include, for example, the movement speed of the robotic arm, joint limits, load capacity, etc.

[0053] Step 102: Based on the alignment results, analyze the robot's task execution process to obtain the items to be adjusted during the task execution process.

[0054] In some embodiments, after obtaining the alignment result, the robot's task execution process can be analyzed based on the alignment result to obtain the items to be adjusted during the task execution process. The items to be adjusted refer to the operational links or parameters that need to be adjusted when the robot deviates from the expected operating state during the process of executing the task under the control of the sequence of control instructions.

[0055] Optionally, the items to be adjusted may include: items caused by unreasonable instruction logic or parameter settings in the control instruction sequence; items caused by the robot's actual execution state and / or operating environment. For example, motion interference, action timing problems, action deviation, position deviation, invalid motion, etc., that occur during the actual execution of the robot; and items caused by unreasonable equipment parameters, such as unreasonable robot equipment parameter settings, unreasonable parameters of process equipment in the robot's operating environment (e.g., excessively high production line speed), and may also include unreasonable parameter settings of other equipment cooperating with the robot, etc., which are not limited in this application.

[0056] Step 103: Based on the information associated with the item to be adjusted, query the knowledge base for knowledge information related to the item to be adjusted.

[0057] In some embodiments, after determining the information associated with the item to be adjusted, knowledge information related to the item to be adjusted can be queried from a knowledge base. This knowledge information can be used to assist in determining the target adjustment scheme. The knowledge base includes at least one of the following: safety constraint information, path adjustment strategies, standard process parameters, and historical adjustment cases.

[0058] Step 104: Generate a target adjustment plan based on knowledge information.

[0059] In some embodiments, after acquiring knowledge information, a target adjustment plan can be generated based on the knowledge information. The knowledge information may include at least one of the following: safety constraint information, path adjustment strategy, standard process parameters, and historical adjustment cases.

[0060] In some embodiments, the target adjustment scheme is used to adjust the item to be adjusted. After the item to be adjusted is determined, a target adjustment scheme can be generated to adjust the item to be adjusted. Furthermore, a targeted adjustment scheme can be generated according to the specific type of the item to be adjusted.

[0061] For example, for items requiring adjustment due to unreasonable instruction logic or parameter settings in the control instruction sequence, modification schemes for the control instruction sequence can be generated, such as outputting the adjusted control instruction sequence; for items requiring adjustment due to the robot's actual execution state and / or operating environment, adjustment suggestions can be output, such as adjusting the robot's running trajectory, adjusting the robot's action timing, etc.; for items requiring adjustment due to unreasonable equipment parameters, prompt messages can be output, prompting staff to modify the corresponding parameters, etc.

[0062] In summary, the above embodiments of this disclosure can identify items that need adjustment during the robot's task execution process based on the alignment results of the robot's visual data and control command sequence. This accurately determines the areas that need adjustment due to unreasonable control command sequence settings, actual operating environment, or deviations in robot execution performance, thus improving the comprehensiveness and accuracy of item identification. Subsequently, targeted adjustment schemes are generated for the accurately identified items, enabling automatic adjustment of the robot's task execution process in the robot's actual operating scenario, and improving the efficiency and reliability of task execution process adjustments.

[0063] Figure 2 A flowchart illustrating a robot task execution adjustment method provided in this embodiment of the disclosure. Figure 2 .like Figure 2 As shown, based on Figure 1 The illustrated embodiment shows that the method includes the following steps.

[0064] Step 201: Based on the alignment results, determine the efficiency parameters of the robot during task execution.

[0065] In some embodiments, after obtaining the alignment result, the robot's task execution process can be analyzed based on the alignment result to determine the efficiency parameters in the task execution process.

[0066] In some embodiments, the efficiency parameter is used to reflect the path execution efficiency of the robot during task execution, and the efficiency parameter includes at least one of time efficiency parameter, energy efficiency parameter and accuracy parameter.

[0067] In some embodiments, the time efficiency parameters include at least one of the following: the execution time of a single action performed by the robot during task execution, the overall path loop time corresponding to the task execution process, the proportion of robot waiting time during task execution, and the redundant time consumed by the robot when performing actions during task execution.

[0068] Among them, single action execution time is the time spent by the robot to perform a single action, such as the time to perform a movement or a grasp; overall path loop time refers to the total time taken by the robot from the starting point to the end point of a task, which is the time required for the robot to perform a complete task; waiting time percentage can be the ratio of the waiting time of the robot during the entire task execution process or during a single task execution process to the total time of the entire task execution process (or the total time of a single task execution); motion redundancy time refers to the extra time spent by the robot in the actual execution of actions, such as the difference between the actual execution time and the theoretical execution time of an action.

[0069] In some embodiments, the energy efficiency parameters include at least one of the following: the loss of the robot's moving mechanism during task execution, the total energy consumption of the robot during task execution, and the proportion of energy consumption from the robot's ineffective movements during task execution.

[0070] Among them, the moving mechanism is, for example, the joint of the robot; the loss of the moving mechanism refers to the energy consumed by the motors of each joint of the robot during the task execution; the total energy consumption of the robot during the task execution refers to the total energy consumption of the robot to complete the entire work cycle, which can be the total energy consumption of the robot to complete one task; the energy consumption ratio of the robot's ineffective motion refers to the proportion of the energy consumed by the robot due to ineffective motion to the total energy consumption.

[0071] In some embodiments, the accuracy parameters include at least one of the following: positional deviation of the robot's actions during task execution, time deviation between the start of execution of the control command sequence and the start of the robot's actions, and similarity between the action features corresponding to the control command sequence and the action features actually executed by the robot.

[0072] Among them, position deviation refers to the deviation between the actual position reached by the robot's end effector and the expected position (e.g., the target position set in the control command); the time deviation between the start of the control command sequence and the start of the robot's action refers to the difference between the start timestamp of the control command sequence and the timestamp of the robot's actual start of the action in the collected visual data; the similarity between the action features corresponding to the control command sequence and the action features actually executed by the robot refers to the similarity between the expected action features corresponding to the control command sequence (e.g., expected motion trajectory, expected velocity curve, expected posture change, etc.) and the actual action features extracted from the visual data.

[0073] Step 202: Analyze the robot's task execution process based on efficiency parameters to obtain the items to be adjusted during the task execution process.

[0074] In some embodiments, the adjustment item includes at least one of explicit adjustment items and implicit adjustment items, wherein the explicit adjustment item is the adjustment item caused by the control instruction sequence, and the implicit adjustment item is the adjustment item caused by the robot's actual execution state and / or the robot's operating environment.

[0075] For example, explicit adjustment items are those caused by unreasonable instruction logic or parameter settings in the control instruction sequence, such as redundant instructions, unreasonable path point settings, excessively long waiting times, and unreasonable instruction timing (e.g., redundant path points in MoveL instructions, excessively long waiting times in WaitDI instructions). Implicit adjustment items are those caused by the robot's actual execution state and / or operating environment, such as motion interference, action timing problems, action deviations, positional deviations, and invalid movements (e.g., excessive rotation when the robotic arm picks up materials, positional offsets when unloading materials). Implicit adjustment items can also include those caused by unreasonable equipment parameters, such as unreasonable robot equipment parameter settings, unreasonable parameters of process equipment in the robot's operating environment (e.g., excessively high production line speed), and unreasonable parameter settings of other equipment cooperating with the robot.

[0076] In some embodiments, the robot's task execution process is analyzed based on efficiency parameters to obtain the adjustment items to be adjusted during the task execution process, including: determining the deviation between the efficiency parameters and standard efficiency parameters; determining the priority of the efficiency parameters as a first priority in response to the deviation being greater than a first threshold; or determining the priority of the efficiency parameters as a second priority in response to the deviation being less than or equal to the first threshold and greater than or equal to a second threshold; or determining the priority of the efficiency parameters as a third priority in response to the deviation being less than the second threshold, wherein the first priority is higher than the second priority and the third priority is lower than the second priority; and analyzing the robot's task execution process according to the priority of the efficiency parameters to obtain the adjustment items to be adjusted during the task execution process.

[0077] The standard efficiency parameter can be an industry standard value or obtained from a knowledge base. After determining the robot's actual efficiency parameter, the deviation between the actual efficiency parameter and the standard efficiency parameter can be determined (e.g., by calculating the difference or deviation rate). Then, multiple efficiency parameters can be categorized based on the deviation. For example, efficiency parameters with a deviation rate greater than 20% can be assigned high priority, those with a deviation rate between 10% and 20% medium priority, and those with a deviation rate less than 10% low priority. Adjustment items can then be determined in descending order of priority. For instance, the adjustment items corresponding to the highest priority efficiency parameter are adjusted first, and the adjustment items corresponding to the third priority efficiency parameter are adjusted last. In other words, the adjustment items corresponding to efficiency parameters with larger deviations can be determined first. The adjustment items corresponding to an efficiency parameter refer to the main reasons causing the deviation of that efficiency parameter. Therefore, the priority of the efficiency parameter is the same as the priority of its corresponding adjustment item. When adjusting, the higher-priority adjustment items can be adjusted first.

[0078] In summary, the above embodiments of this application, by identifying the items to be adjusted by the robot during task execution based on the alignment results, can achieve the fusion of multimodal data including visual data, control command sequences, execution logs, and process parameters. This allows for the analysis of both code logic and actual actions, enabling multi-dimensional and comprehensive analysis of the robot path. It can identify deviations between code execution and actual actions, as well as problems caused by unreasonable control command sequences and issues arising from the robot's actual execution state and operating environment. This improves the comprehensiveness and accuracy of the identified items to be adjusted.

[0079] Figure 3 A flowchart illustrating a robot task execution adjustment method provided in this embodiment of the disclosure. Figure 3 .like Figure 3 As shown, based on Figure 1 The illustrated embodiment shows that the method includes the following steps.

[0080] Step 301: Based on the information associated with the item to be adjusted, query the knowledge information related to the item to be adjusted from the knowledge base.

[0081] In some embodiments, after determining the information associated with the item to be adjusted, knowledge information related to the item to be adjusted can be queried from a knowledge base. This knowledge information can be used to assist in determining the target adjustment scheme. The knowledge base includes at least one of the following: safety constraint information, path adjustment strategies, standard process parameters, and historical adjustment cases.

[0082] In some embodiments, the information associated with the item to be adjusted includes at least one of the following: the type of the item to be adjusted; the first visual data corresponding to the item to be adjusted; the control instruction sequence corresponding to the item to be adjusted; the efficiency parameter (or the impact index of the item to be adjusted) corresponding to the item to be adjusted; the priority of the item to be adjusted, wherein the priority of the item to be adjusted is related to the priority of the efficiency parameter corresponding to the item to be adjusted; the type of robot; and the operating environment of the robot.

[0083] The type of the item to be adjusted (or the bottleneck type) can be either an explicit item to be adjusted or an implicit item to be adjusted. For example, it can be a sequence of video frames (video range) associated with the item to be adjusted in the first visual data corresponding to the item to be adjusted. If the item to be adjusted is a deviation in the robot's grasping action, then the first visual data corresponding to the item to be adjusted can be a sequence of video frames related to the grasping action in the collected first visual data. The control instruction sequence corresponding to the item to be adjusted is similar to the first visual data corresponding to the item to be adjusted. For example, if the item to be adjusted is a deviation in the robot's grasping action, then the control instruction sequence corresponding to the item to be adjusted can be a code block, code anchor point, code line number, etc. that controls the robot to perform the grasping action.

[0084] Among them, the priority of the item to be adjusted is related to the priority of the efficiency parameter corresponding to the item to be adjusted. For example, the priority of the item to be adjusted is the same as the priority of the efficiency parameter corresponding to the item to be adjusted. The robot's operating environment can also be a process scenario, such as dispensing, handling, assembly, etc.

[0085] In some embodiments, knowledge information can be obtained by searching the knowledge base based on the information associated with the item to be adjusted. For example, the type of the item to be adjusted, the type of robot, the operating environment of the robot, etc. can be used as keywords to perform vector similarity retrieval in the knowledge base, and information with high similarity can be retrieved as knowledge information.

[0086] Optionally, the retrieved knowledge information can be filtered, for example, content that does not match the current robot type or operating environment can be excluded.

[0087] Step 302: Generate a target adjustment plan based on knowledge information.

[0088] In some embodiments, after acquiring knowledge information, a target adjustment plan can be generated based on the knowledge information. The knowledge information may include at least one of the following: safety constraint information, path adjustment strategy, standard process parameters, and historical adjustment cases.

[0089] In some embodiments, the target adjustment scheme includes at least one of the following: modifying the control command sequence; modifying the parameters of the target device, the target device including at least one of the following: a robot, a device in the robot's operating environment, or a device that collaborates with the robot during task execution; modifying the process parameters associated with the robot; and outputting a prompt message indicating that at least one of the following is required: the control command sequence needs to be modified, the parameters of the target device need to be modified, or the process parameters associated with the robot need to be modified.

[0090] For example, when the item to be adjusted is caused by an unreasonable control command sequence, a target adjustment plan can be generated to modify the control command sequence. At this time, the adjusted control command sequence can also be output, or a new control command sequence can be directly issued to the robot to adjust the robot's control command sequence logic. For example, the target adjustment plan can be output to delete redundant path points, shorten waiting time, adjust command timing, and modify motion parameters (speed / acceleration).

[0091] In some embodiments, the target adjustment scheme may involve adjusting the parameters of the target device. For example, if the item to be adjusted is due to unreasonable device parameters, the parameters of the target device can be modified. For instance, if the operating environment is an assembly scenario, the device in the robot's operating environment is a conveyor belt device; if the operating environment is a warehouse handling scenario, the device cooperating with the robot can be a sorting device, a lifting device, etc. For example, the target adjustment scheme may involve adjusting the robotic arm's motion trajectory, adjusting the action timing, or eliminating motion interference.

[0092] For example, when the adjustment is needed because the parameters of the process equipment in the robot's operating environment are unreasonable, the target adjustment plan can include modifying the process parameters associated with the robot, such as the conveyor belt speed, material handling position, and material unloading position. For instance, the target adjustment plan could be to adjust the workpiece placement position, adjust the production line layout, or adjust the timing of multi-robot collaboration.

[0093] In some embodiments, the target adjustment scheme may further include outputting prompt information. For example, when modifying the control command sequence, a prompt message may be displayed indicating that there is a problem with the current control command sequence and it needs to be adjusted. The prompt message may also include the adjusted control command sequence. For example, the method further includes displaying the adjusted control command sequence on a display interface. The display interface may be the display interface of an electronic device, or the display interface of a device used to control a robot. If the robot has a display screen, the prompt message may be displayed on the robot's display screen, or the adjusted control command sequence may be displayed on the display interface of a user interaction device.

[0094] The output prompt information can also be output in other ways, such as through audio output, vibration signal output, indicator light output, etc., and this application does not limit this. The prompt information may also include the adjusted parameters of the target equipment, the adjusted process parameters, etc., and this application does not limit this. Optionally, after determining the target adjustment scheme, the control command sequence, the parameters of the target equipment, the process parameters, etc., can be modified directly based on the target adjustment scheme, or the target adjustment scheme can be output (e.g., output prompt information including the content of the target adjustment scheme), and modifications can be made based on the target adjustment scheme after the staff confirms the target adjustment scheme; or the target adjustment scheme can be output (e.g., output prompt information including the content of the target adjustment scheme), and the staff can manually modify it according to the target adjustment scheme, and this application does not limit this.

[0095] For example, the method also includes: responding to the user's confirmation of the adjusted control command sequence, and controlling the robot to run based on the adjusted control command sequence. That is, the adjusted control command sequence can be displayed on the display interface, and after the staff confirms the adjusted control command sequence, the robot can be controlled according to the adjusted control command sequence, or the adjusted control command sequence can be sent to the robot control cabinet, and then the robot can run (or perform tasks) based on the adjusted control command sequence.

[0096] In some embodiments, in response to a user's confirmation of the adjusted control command sequence, controlling the robot to run based on the adjusted control command sequence includes: performing a security check on the adjusted control command sequence; and if the security check passes, controlling the robot to run based on the adjusted control command sequence in response to the user's confirmation of the adjusted control command sequence.

[0097] In other words, the automatically generated and adjusted control command sequence can be safety verified to ensure that it complies with industry safety rules and robot equipment parameter requirements, avoiding problems such as exceeding movement limits and equipment interference. After the safety verification is passed, the adjusted control command sequence can be sent to the robot control cabinet, and then the robot can run (or perform tasks) based on the adjusted control command sequence.

[0098] In some embodiments, the prompt information may further include at least one of the following: information associated with the item to be adjusted; the adjusted control command sequence; the position of the adjusted control command sequence; the control command sequence before adjustment; the basis for modification; and the expected adjustment effect.

[0099] The prompt information may also include the position of the adjusted control command sequence and the control command sequence before the adjustment, so as to inform the staff which control command sequences in which positions need to be modified in this modification, so that the user can compare and confirm what kind of modification has been made to the control command sequence.

[0100] The prompt information may also include the basis for modification, which may include information, knowledge information, etc. related to the above-mentioned items to be adjusted; the expected adjustment effect may be, for example, a reduction in execution time by X% and a reduction in energy consumption by Y%.

[0101] In some embodiments, the method further includes: during the process of controlling the robot to run based on the adjusted control command sequence, collecting second vision data of the robot's operation and execution logs of the adjusted control command sequence; determining the adjustment effect based on the second vision data and execution logs; and processing at least one of the information associated with the item to be adjusted, the information of the target adjustment scheme, and the adjustment effect into a historical adjustment case and saving it to a knowledge base.

[0102] In other words, after each adjustment is completed, the cases and effects of this adjustment can be stored in the knowledge base, enabling iterative updates of the knowledge base. When making adjustments later, the accuracy of subsequent optimizations can be improved.

[0103] In some embodiments, during the process of controlling the robot to run based on the adjusted control command sequence, the second visual data of the robot's operation and the execution log of the adjusted control command sequence are collected. At this time, the collected second visual data and the execution log of the adjusted control command sequence can be used to determine whether the adjustment effect meets the expected effect. If the expected effect is not met, the alignment result is determined based on the re-collected visual data and control command sequence, and the items to be adjusted are determined based on the alignment result. The items to be adjusted of the robot are adjusted again until the performance of the robot after adjustment meets the expected requirements.

[0104] In summary, the above embodiments of this disclosure, by querying relevant knowledge information from the knowledge base based on the information associated with the item to be adjusted to generate a target adjustment scheme, can provide corresponding knowledge information for different types of items to be adjusted, so that the generated adjustment scheme can be adapted to different items to be adjusted, thereby improving the effectiveness of the generated target adjustment scheme. Furthermore, by performing security verification and user confirmation on the target adjustment scheme, it can be ensured that the target adjustment scheme complies with safety specifications, and the safety and standardization of robot operation during the adjustment process can be guaranteed.

[0105] Figure 4 A flowchart illustrating a robot task execution adjustment method provided in this embodiment of the disclosure. Figure 4 .like Figure 4 As shown, based on Figure 1The illustrated embodiment shows that the method includes the following steps.

[0106] Step 401: Generate task requirement information based on the information associated with the items to be adjusted.

[0107] In some embodiments, when it is necessary to modify the control command sequence, task requirement information can be generated based on the information associated with the item to be adjusted. The task requirement information can be a control command sequence modification requirement, such as a requirement to delete redundant path points, a requirement to shorten the waiting time, a requirement to adjust the command timing, a requirement to modify motion parameters, etc.

[0108] Step 402: Generate prompt words based on at least one of the following: task requirement information, information in the knowledge base, and information in the rule base.

[0109] In some embodiments, the rule base includes rules for controlling the generation of instruction sequences, such as encoding rules, program structure specifications, data type specifications, security protection specifications, and so on.

[0110] In some embodiments, the method further includes: acquiring multiple technical documents related to the task execution process, the multiple technical documents including at least one of the following: a robot programming language instruction reference manual, a robot operation manual, a detailed explanation of robot control instructions, an application skills document, a signal configuration instruction document, and a tool coordinate setting instruction document; and constructing a knowledge base based on the multiple technical documents.

[0111] The programming language instruction reference manual provides complete syntax definitions, parameter descriptions, return value types, and usage guidelines for available instructions in robot programming languages. It ensures that the generated control instruction sequences conform to robot programming requirements in terms of instruction format and parameters. For example, for industrial robot control instructions, the reference manual can define the number of parameters, parameter types, value ranges, and optional parameters for each motion instruction (such as linear motion instructions or joint motion instructions).

[0112] The robot operation manual includes basic operating procedures for the robot, such as startup and shutdown procedures, safety precautions, and switching between different modes. The robot operation manual helps ensure that the generated control command sequences can be safely executed in real-world operating environments.

[0113] Among them, the detailed documentation on control instructions can be used to provide detailed explanations of each control instruction in a programming language, including the function description of the instruction, applicable scenarios, the specific meaning of the parameters and suggestions for their values, how to combine instructions, common programming errors and how to avoid them, etc.

[0114] The application skills document can be used to record practical techniques and best practice cases in the robot programming process, such as programming experience in specific process scenarios, path planning methods and precautions, recommended settings for velocity and acceleration parameters, and programming templates for various tasks. This document can also record experience from the past programming process.

[0115] The signal configuration specification document provides configuration standards for input and output signals between the robot and other devices, including the definition of digital signals (Digital Input (DI) / Digital Output (DO)) and analog signals (Analog Input (AI) / Analog Output (AO)), the naming rules for signal names, the method of allocating signal addresses, and the logical relationship between signals and the actions to be performed.

[0116] The tool coordinate setting documentation can be used to indicate content related to the robot tool coordinate system, such as the concept of the tool coordinate system, the calibration steps of the tool center point, the setting format of tool coordinate data, and the method of switching coordinate systems between different tools.

[0117] In some embodiments, a knowledge base can be constructed based on the aforementioned technical documents and other relevant technical documents. Optionally, the knowledge base can be constructed based on multiple technical documents, including: constructing a skill definition file, which defines the programming process specifications for the robot; and constructing the knowledge base based on the skill definition file and multiple technical documents. In other words, when constructing the knowledge base, skills can also be defined to define the workflow specifications for robot programmers, including input / output requirements and quality standards for code generation.

[0118] In some embodiments, the method further includes at least one of the following: obtaining information from a locally stored knowledge base; obtaining information from the knowledge base via a command-line tool. In other words, a dual-path loading mechanism can be used to obtain knowledge base content. When a knowledge base is stored locally, information from the locally stored knowledge base can be directly obtained, such as reading skill definition files, API reference documents, and program example files directly from the local file system and concatenating them into a structured context string. When a knowledge base is not stored locally, skill definitions can be remotely loaded via a command-line tool, ensuring availability in both offline and online scenarios.

[0119] In some embodiments, when knowledge base information is obtained, it can be stored as a skill context string (skill_context) as input data for subsequent prompt word construction.

[0120] In some embodiments, the method further includes: acquiring coding rule information of the robot, the coding rule information including rules for generating control instruction sequences; and constructing a rule base based on the coding rule information.

[0121] In some embodiments, the encoding rule information includes at least one of the following: security priority rules; reliability rules; maintainability rules; efficiency rules; program structure specifications; data type classification; naming conventions; motion control instruction format specifications; input and output signal specifications; security protection specifications; and error handling specifications.

[0122] Among them, the safety priority rule refers to the safety constraints that must be followed during code generation to ensure that the generated control command sequence will not cause harm or damage to personnel, equipment, or the environment when controlling the robot. The safety priority rule means that when other rules conflict with the safety priority rule, the safety priority rule prevails. The safety priority rule can include at least one of the following: absolute safety rule, emergency stop reachability rule, zero-collision design rule, and speed grading rule.

[0123] Among them, the absolute safety rule can mean that the generated control command sequence ensures the safe operation of the robot in any operating scenario, avoiding damage to the robot itself, surrounding equipment, or personnel; the emergency stop reachability rule can mean that during the execution of the generated control commands, the robot can respond to the emergency stop signal at any position and state, and stop all movement within the required safe time; the zero collision design rule can mean that when the robot runs according to the path and trajectory corresponding to the generated control commands, the robot will not collide with surrounding equipment, devices, or other robots; the speed grading rule can mean that the generated control commands need to meet the speed limits of different motion stages. For example, a higher speed can be used in safe working conditions such as no-load and large-scale transfer, while a lower speed can be used in working conditions that require precise control, such as picking up and placing materials, precise positioning, and contact operations.

[0124] In some embodiments, reliability rules refer to reliability criteria that need to be followed during the generation of control instruction sequences, in order to ensure that the generated control instruction sequences can be executed stably over long periods of time and under various operating scenarios.

[0125] Reliability rules can include at least one of the following: Detection-first rule, which requires checking whether relevant conditions are met before any movement or operation (e.g., checking the material arrival signal before picking up materials, checking the workstation readiness signal before unloading materials) to avoid task failure due to unmet conditions; Status confirmation rule, which requires confirming the result of each critical action (e.g., checking whether the gripping device is clamped after picking up materials, checking whether the workpiece is placed in place after unloading materials) to ensure successful execution before proceeding to the next step; Fault-tolerant design rule, which requires the control command sequence to include handling logic for abnormal situations, setting backup handling methods for possible signal timeouts, communication interruptions, sensor malfunctions, etc., so that the robot can safely stop or resume when an abnormality occurs; Timeout protection rule, which sets timeout limits for operations that require waiting for external signals (e.g., setting timeout parameters for the WaitDI command), automatically triggering the timeout handling program when the signal does not arrive within the specified time to prevent the robot from waiting indefinitely.

[0126] In some embodiments, maintainability rules are used to ensure that the generated sequence of control instructions can be easily read, modified, and extended manually.

[0127] The maintainability rules can include at least one of the following: Modular design rules refer to decomposing complex tasks into several single-function program modules (PROCs), each responsible for an independent function (such as a material picking module, a material placing module, a palletizing calculation module, etc.), which facilitates individual modification and testing. When a function needs to be adjusted, only the corresponding module needs to be modified, without requiring a global modification to the entire program; Clear naming rules indicate that program module names, variable names, constant names, etc., should have a clear naming convention, which can distinguish different modules, variables, constants, etc. by naming; Sufficient commenting rules refer to adding comments to the generated control command sequences, which can help other maintenance personnel understand the logic and intent of the generated control command sequences; Log output rules refer to the control command sequences needing to include log output logic, which can output logs when the control command sequences are executed, making it easy to understand the current execution progress, and at the same time, the problem location can be quickly located through the logs when a fault occurs.

[0128] In some embodiments, efficiency rules are used to ensure that the generated sequence of control instructions has high operating efficiency.

[0129] The efficiency rules can include at least one of the following: the idle-travel fast movement rule, which means that the robot uses a higher speed to move during idle travel (such as large-scale transfer movements without load, workpiece, or contact requirements); the precision slow movement rule, which means that the robot can use a lower speed to ensure execution accuracy during precise positioning movements (such as picking up and placing materials, positioning, contact operations, and other positions that require precise control); the path optimization rule, which means that the robot's movement path should be as short as possible; and the parallel execution rule, which means that when the robot needs to perform multiple independent operations, if conditions permit, the operations can be executed in parallel through reasonable instruction arrangement (such as controlling the action of the picking and placing device at the same time during the robot's movement) to reduce the overall cycle time.

[0130] In some embodiments, program structure specifications refer to the program organization structure specifications that must be followed during code generation. Program structure specifications may include at least one of the following: module type and file format specifications, such as .modx format and .sysx format; minimal program structure specifications, such as specifying which components a complete program module must include (such as MODULE declaration, BEGIN marker, one or more PROC and ENDMODULE end markers) to ensure the structural integrity of the generated control instruction sequence; and main program structure specifications, such as specifying that the main program module should be organized according to a standard template of four stages: startup check, system initialization, main loop, and cleanup exit, to ensure that the program has complete lifecycle management logic.

[0131] In some embodiments, data type classification refers to the data type definitions and classification specifications that must be followed during the generation of control instruction sequences, to ensure the correctness of variable declarations and data type usage in the generated control instruction sequences. Data type classification may include at least one of the following: value data types, geometric data types, and configuration data types.

[0132] In some embodiments, naming conventions refer to the naming rules that need to be followed during code generation. The naming conventions may include at least one of the following: constant naming conventions (e.g., all uppercase or camelCase); variable naming conventions (e.g., starting with a lowercase letter and using camelCase); and permanent data naming conventions (e.g., adding a prefix according to function).

[0133] In some embodiments, the motion control instruction format specification refers to the format and parameter structure specification of the motion control instructions that must be followed during the generation of the control instruction sequence, to ensure that the generated motion instructions meet the robot's requirements in terms of syntax and parameters. Among these, motion control instruction format specifications include, for example, four types of motion instructions and their standard formats: MoveJ / MoveL / MoveC / MoveAbsJ.

[0134] In some embodiments, the input and output signal specifications refer to the definitions and operational rules of input and output signals that must be followed during the generation of control instruction sequences. These specifications may include naming and operational instructions for six signal categories: DI, DO, AI, AO, Group Input (GI), and Group Output (GO).

[0135] In some embodiments, the security protection specification refers to the security protection function specification that must be followed during the generation of the control instruction sequence, which is used to ensure that the generated control instruction sequence contains security protection logic, such as emergency stop system TRAP interrupt handling, area protection WorldZone restriction interrupt handling, area protection WorldZone restriction, etc.

[0136] In some embodiments, error handling specifications refer to the error handling mechanisms that must be followed during the generation of control instruction sequences to ensure that the generated control instruction sequences can respond promptly when encountering errors or abnormal situations. Examples of error handling specifications include a retry policy of 3 times, UNDO handle handling, and ERROR handler mechanisms.

[0137] In some embodiments, after establishing the aforementioned rule base (encoding rule base), the rule base content can be merged with the knowledge base content to serve as the domain context input for the prompt words. Optionally, the task requirement information, the information in the knowledge base, and the information in the rule base can be integrated into structured prompt words that the target model can understand, where the target model can be an LLM model. The target model can be used not only to generate the adjusted control instruction sequence but also to generate the control instruction sequence before adjustment. When generating the control instruction sequence before adjustment, the task requirement information can be the robot task requirement described by the user in natural language. In this case, the robot task requirement described by the user in natural language (e.g., picking up bottles from the conveyor belt and placing them into a 3x3 tray grid) can be used as the primary content of the prompt words, clearly defining the code generation goal of the LLM. Merging the rule base content and the knowledge base content as the domain context input for the prompt words can provide the LLM with professional knowledge support for ABB robot programming, including API references, program examples, and templates.

[0138] Step 403: The target model is invoked to generate an adjusted sequence of control instructions based on the prompt words and structured constraint information.

[0139] In some embodiments, structured constraint information can be used to impose structural constraints on the LLM code generation process. The structured constraint information includes at least one of the following: loop logic constraint information; speed optimization constraint information; control instruction sequence conciseness constraint information; safe motion path constraint information; variable scope constraint information; coordinate system constraint information; error handling constraint information; and control instruction sequence output format constraint information.

[0140] In some embodiments, loop logic constraint information, for example, specifies the use of a WHILE loop instead of a FOR loop, waiting for the DIPZ signal before each material pick-up and waiting for the DIXZ signal before each material unloading, to implement event-driven piece-by-piece processing logic; speed optimization constraint information, for example, specifies the definition of high-speed data (v7500:=[7500,500,5000,1000]) and low-speed data (v500:=[500,50,5000,1000]), and sets AccSet80,80 and VelSet100,7500 at program startup, using high speed for fast movement. Low-speed loading and unloading for precise positioning; concise control command sequence constraints, such as specifying a maximum of 3 to 4 PROCs (Main, PickBottle, PlaceBottle), prohibiting excessive modularization; safe motion path constraints, such as specifying the use of the Offs() function to dynamically generate intermediate points, adopting a three-stage safe path of high-speed approach → low-speed precise positioning → high-speed lifting; variable scope constraints, such as specifying that cross-PROC shared variables such as currentX, currentY, and totalCount must be declared as LOCAL VAR at the MODULE level, prohibiting declaration within PROCs, and ensuring that the PlaceBottle PROC is accessible; coordinate system constraints, such as specifying the use of the tool0 system default tool and base coordinate system, simplifying simulation environment configuration; error handling constraints, such as specifying the addition of an ERROR handler at the end of the Main PROC to handle ERR_WAIT_MAXTIME signal timeout exceptions and execute Stop; control command sequence output format constraints, such as specifying that generated code does not contain comments, TPWrite only outputs strings, and defined signals (DIPZ / DIXZ / DOPZ) cannot be redefined.

[0141] In some embodiments, when invoking the target model to generate an adjusted sequence of control instructions based on prompt words and structured constraint information, multi-model initialization can be performed first. The interface differences between different large language model providers are encapsulated by the model invocation module, and a unified invocation interface is used to establish connections with the servers of each model provider. During invocation, model routing is achieved through the model provider identification information carried in the HTTP request header, routing the request to the target model server specified by the user. This shields the access differences between different models, allowing upper-layer code to achieve unified invocation of multiple large language models without needing to concern itself with the API details of specific model providers.

[0142] Next, message assembly can be performed. The pre-defined role setting information and the complete prompt words constructed in the above embodiment are assembled into a message list. The role setting information is used to define the role of the large language model, for example, setting the model role as a professional industrial robot control code programming expert, so that the model understands and generates in this role throughout the dialogue. The complete prompt words contain user task requirements, information from the knowledge base, information from the rule base, and structured constraint information. The role setting information and prompt words are assembled into a message list according to the message format required by the target model, serving as input for model invocation.

[0143] Finally, model invocation and code generation are performed. A message list is sent to the target model server through a unified calling interface. After receiving the message, the target model understands the user's intent based on the task requirements in the prompt words, and generates a sequence of control instructions that meets the requirements based on information from the knowledge base, rule base, and structured constraint information.

[0144] In some embodiments, the method further includes: generating a call log, which includes at least one of the following: information about the target model, the time of calling the target model, information related to the prompt words, and an adjusted sequence of control instructions. In other words, after the model call is completed, relevant information about this call can be recorded, including the call time, the name of the target model, the model provider, the length of the prompt words, etc., and the complete prompt words and the model response can be saved as separate files. A hash value is calculated on the response content for quality traceability, so as to facilitate subsequent quality traceability of the generated results.

[0145] In some embodiments, the method further includes: automatically saving the adjusted control command sequence; or saving the adjusted control command sequence in response to a user's save command. In other words, after generating the adjusted control command sequence, the adjusted control command sequence can be automatically stored, or it can be stored in response to a user's needs. When the user manually saves the sequence, a file selection dialog box can be provided, allowing the user to specify the save path and filename.

[0146] In summary, the above embodiments of this application construct a knowledge base based on technical documents, construct a rule base based on coding rule information, and then automatically generate control instruction sequences based on the information in the knowledge base, the information in the rule base, and the constraint information. This can ensure that the generated control instruction sequences conform to industrial robot programming rules and industry standards, improve coding efficiency, and reduce the technical threshold and labor costs of industrial robot programming.

[0147] The technical solutions of this disclosure will be further described in detail below with reference to specific application embodiments.

[0148] This disclosure also provides a robot path intelligent optimization method and system, the core application scenario of which is the optimization of the path efficiency of the robotic arm of an industrial robot in an automotive parts production line.

[0149] This technical solution is applied to automotive parts welding and assembly production lines. The intelligent path optimization system described in this application is deployed on-site. The system interfaces with the control cabinet of the industrial robot arm (reading the robot's control code and execution logs) and a high-definition camera (capturing video of the robot arm's operation). The system achieves spatiotemporal alignment of video and code, path efficiency analysis, bottleneck identification, and automatically generates path optimization suggestions and code modification plans. Technicians can view the optimization suggestions through the system terminal and send the adjusted code to the robot control cabinet with a single click. The robot arm then executes the optimized path, resulting in improved production efficiency and reduced energy consumption.

[0150] Scenario Description: Multiple industrial robot arms in a production line collaboratively perform welding operations on automotive parts. Each robot arm is equipped with a high-definition camera to capture the entire welding process. The optimization system is deployed on the production line's central control server and communicates with all robot arm control cabinets and cameras via industrial Ethernet. The system automatically parses the industrial robot control code and running video of each robot arm, performs spatiotemporal alignment, analyzes the path execution efficiency of each robot arm, and identifies bottlenecks such as "redundant material handling paths, unreasonable welding action timing, and excessive waiting time." The system combines an industry knowledge base for automotive parts welding to generate personalized optimization suggestions and code modification plans for each robot arm. After review by the central control technicians, the optimized code is distributed to the robot control cabinets, and the robot arms execute the optimized paths, resulting in reduced operation time for individual robot arms and improved overall production line efficiency.

[0151] This solution can also be extended to scenarios such as robot material handling path optimization in the 3C electronics industry, robot fault review and maintenance optimization, and intelligent improvement of production processes. The core logic is to use this technology to integrate multimodal data and knowledge base to achieve intelligent analysis and optimization of robot paths.

[0152] The robot path intelligent optimization method of this application relies on five core modules: video parsing, code parsing, knowledge base management, spatiotemporal alignment, and intelligent optimization. The overall process is executed in the logical order of multimodal data input → data parsing and spatiotemporal alignment → path efficiency quantitative analysis → bottleneck identification → knowledge base retrieval → optimization suggestion generation → optimization effect verification. Each step is described in detail below.

[0153] Step 1: Input multimodal data.

[0154] The system data input module executes the process, and the input data includes: high-definition video of the robot running (resolution ≥1080P, frame rate ≥30fps), covering the complete working path of the robot; robot control code (such as industrial robot control code) and code execution log, including information such as instruction execution time, line number, and parameters; production line process parameters and robot equipment parameters (such as robotic arm movement speed, joint limits, and load capacity).

[0155] All input data is uploaded in the system's preset format, and timestamps are synchronized (based on the NTP protocol) to ensure data consistency.

[0156] Step 2: Data parsing and spatiotemporal alignment.

[0157] The system, consisting of a video parsing module, a code parsing module, and a spatiotemporal alignment module, parses multimodal data into structured data and achieves precise spatiotemporal alignment between video events and code instructions, providing a data foundation for subsequent analysis.

[0158] First, the video parsing module analyzes the input video, extracting eight core standardized video events and generating a video event log containing event type, timestamp, and time span. Next, the code parsing module analyzes the input industrial robot control code, extracting executable code blocks and generating structured code data containing instruction type, core parameters, line number range, and execution time window. Finally, the spatiotemporal alignment module employs an "anchor point matching hierarchical algorithm" to achieve precise spatiotemporal alignment between video events and code instructions, generating standardized alignment results containing core fields such as "video_time_range, code_block, confidence, and inferred_actions," with an alignment accuracy ≥90%. After alignment, a video-code spatiotemporal alignment result dataset can be output. This dataset contains alignment information for all valid anchor point pairs and supports bidirectional traceability.

[0159] Step 3: Quantitative analysis of path efficiency.

[0160] Executed by the intelligent optimization module, a quantitative analysis index system can be established to conduct multi-dimensional quantitative evaluation of the robot's path execution efficiency and identify potential optimization points.

[0161] Specifically, based on the spatiotemporal alignment result dataset, the core indicators of robot path execution are extracted, and a path efficiency quantitative indicator system is constructed, which includes three types of indicators: time indicators, such as single action execution time, overall path loop time, waiting time ratio, and action redundancy time; energy consumption indicators, such as energy consumption of each joint of the robotic arm, overall path energy consumption, and the proportion of ineffective motion energy consumption; and accuracy indicators, such as action position deviation, time deviation between instruction execution and actual action, and feature similarity.

[0162] Next, each indicator is quantitatively calculated to generate a path efficiency quantitative analysis report, which marks the actual value, industry standard value (retrieved from the knowledge base), and deviation rate of each indicator. Finally, the indicators are graded and evaluated, with indicators with a deviation rate >20% marked as high-priority optimization points, those with a deviation rate between 10% and 20% marked as medium-priority optimization points, and those with a deviation rate <10% as normal indicators.

[0163] Step 4: Path bottleneck identification.

[0164] Executed by the intelligent optimization module, which combines quantitative analysis results with video / code analysis, explicit and implicit bottlenecks in the robot path are identified, and optimization directions are clarified.

[0165] First, explicit bottlenecks can be identified, such as redundant instructions, unreasonable path point settings, excessively long waiting times, and unreasonable instruction timings in the code, based on code analysis (e.g., redundant path points in the MoveL instruction, excessively long waiting times in the WaitDI instruction). Next, implicit bottlenecks can be identified, such as motion interference, unreasonable action timing, positional deviations, and invalid movements in the robot's actual actions, based on video analysis (e.g., unnecessary rotations during material picking, and positional offsets during material placement). Finally, by combining spatiotemporal alignment results, a bottleneck-corresponding code / action relationship can be established, meaning each bottleneck corresponds to a specific code segment and video action segment, providing precise location for subsequent optimization. After identifying bottlenecks, a path bottleneck identification report can be output, including bottleneck type, location (code line number / video time range), impact metrics, priority, and other information.

[0166] Step 5: Knowledge base retrieval and matching.

[0167] Executed by the knowledge base management module, it can provide industry knowledge and best practice support for bottleneck optimization, improving the universality and reliability of optimization suggestions.

[0168] First, the core information from the bottleneck identification report (such as bottleneck type, robot type, and process scenario) is used as search keywords to perform vector similarity retrieval in the knowledge base. The knowledge base stores four types of knowledge: industry safety rules, best practices for path optimization, process standards, and typical cases. After retrieval, the knowledge fragments with the highest matching degree to the current bottleneck and typical optimization cases can be returned. Then, the retrieved knowledge is filtered to exclude content that does not match the current robot equipment and process scenario, generating knowledge base matching results to provide a basis for generating optimization suggestions.

[0169] Step 6: Optimization suggestions and code modification plan generation.

[0170] Executed by the intelligent optimization module, it first combines path bottleneck identification reports, path efficiency quantitative analysis reports, and knowledge base matching results to generate targeted optimization suggestions for each bottleneck. The optimization suggestions are divided into three categories: code modification suggestions, such as deleting redundant path points, shortening waiting time, adjusting instruction timing, and modifying motion parameters (speed / acceleration); motion adjustment suggestions, such as optimizing the robotic arm's motion trajectory, adjusting motion timing, and eliminating motion interference; and process improvement suggestions, such as adjusting workpiece placement, optimizing production line layout, and adjusting multi-robot collaboration timing.

[0171] Based on the code modification suggestions, a standardized code modification plan can then be automatically generated, including the specific line numbers of the modified code, the original code content, the modified code content, and the basis for the modification. Finally, the optimization suggestions and code modification plan are subjected to safety verification to ensure that the adjusted code complies with industry safety rules and robot equipment parameter requirements, avoiding problems such as exceeding movement limits and equipment interference. After generating optimization suggestions, a robot path intelligent optimization report can be output, including bottleneck identification results, optimization suggestions, code modification plan, and expected optimization effects (such as a reduction in execution time by X% and a reduction in energy consumption by Y%).

[0172] Step 7: Optimize and verify the results and iterate.

[0173] The optimization is executed by the intelligent optimization module, or the optimization effect can be verified on-site. For example, technicians distribute the code modification plan generated by the system to the robot control cabinet, and the robot executes the optimized path. The system re-collects the robot's optimized running video and code execution logs, repeats steps 2-5, and generates a quantitative analysis report on the efficiency of the optimized path. The path efficiency indicators before and after optimization are compared to verify whether the optimization effect has met expectations. If it has met expectations, the optimization is completed. If it has not met expectations, based on the post-optimization analysis results, the system returns to step 4 for secondary bottleneck identification and generates iterative optimization suggestions. Finally, the optimization cases and effects can be stored in the knowledge base to achieve iterative updates of knowledge and improve the accuracy of subsequent optimizations.

[0174] The intelligent robot path optimization system of this application is a hardware and software combination system for implementing the above methods. It includes five core functional modules, which work together to achieve full-process automation of multimodal data fusion, spatiotemporal alignment, path analysis, and intelligent optimization. The module functions and connections are as follows: Video parsing module: hardware consists of a high-definition camera and a video processing chip; software implements video frame preprocessing, motion detection, event parsing, and video anchor point generation, outputting structured video event logs and communicating bidirectionally with the spatiotemporal alignment module; Code parsing module: hardware consists of an industrial control computer / server; software implements parsing of robot control code, such as industrial robot control code, extracting executable code, generating code anchor points, and establishing indexes, outputting structured code data and communicating bidirectionally with the spatiotemporal alignment module; Knowledge base management module: based on a vector database and a Large Language Model (LLM). The system is structured as follows: Hardware: Storage server; Software: Knowledge import, vector index construction, knowledge retrieval, and iterative knowledge updates; Storage industry security rules, best practices, process standards, and typical cases; Bidirectional communication with the intelligent optimization module. Spatiotemporal alignment module: Hardware: Data processing server; Software: Time base calibration of video and code anchor points, hierarchical anchor point matching, and post-processing verification; Output: Video and code spatiotemporal alignment result datasets; Bidirectional communication with the video parsing module, code parsing module, and intelligent optimization module. Intelligent optimization module: Hardware: High-performance computing server; Software: Path efficiency quantification analysis, bottleneck identification, optimization suggestion generation, code modification scheme generation, and optimization effect verification; Bidirectional communication with the spatiotemporal alignment module and knowledge base management module; One-way communication with the robot control cabinet for code distribution and data retrieval.

[0175] The overall system architecture is as follows: Data input module (camera, robot control cabinet) → Video parsing module / code parsing module → Spatiotemporal alignment module → Intelligent optimization module Knowledge base management module → Result output module (terminal, robot control cabinet).

[0176] This approach establishes a standardized quantitative indicator system through path efficiency quantification analysis, transforming fuzzy path efficiency into quantifiable metrics. This solves the problem of unquantifiable optimization effects and provides objective evidence for bottleneck identification and optimization effectiveness verification. The path bottleneck identification step combines code and video analysis to identify both explicit bottlenecks in the code and implicit bottlenecks in actual actions, overcoming the limitation of related technologies that can only analyze code and achieving full-dimensional bottleneck identification. The optimization suggestion and code modification scheme generation step automatically generates targeted optimization suggestions and standardized code modification schemes based on multimodal data and a knowledge base, reducing manual intervention and solving the problems of low optimization efficiency and poor implementation, thus improving optimization efficiency and implementation success rate. The spatiotemporal alignment step achieves precise correlation between video and code, upgrading path analysis from pure code analysis to correlation analysis of code and actual actions, solving the core problem of optimization suggestions being disconnected from actual scenarios.

[0177] In summary, the examples disclosed above, by integrating the spatiotemporal alignment results of video and code, achieve correlation analysis between code instructions and actual actions, enabling optimization suggestions to be derived from actual robot operation data, thus solving the problem of optimization suggestions being disconnected from the actual scenario. Simultaneously, by integrating multimodal data such as video, code, execution logs, and process parameters, it analyzes both code logic and actual actions, achieving multi-dimensional analysis of robot paths. This allows for the identification of deviations between code execution and actual actions, and further, through video analysis, identifies implicit optimization items in actual actions, overcoming the limitation of only identifying explicit code problems and achieving comprehensive and refined path optimization. Furthermore, this application, by building a knowledge base management module, integrates safety constraint information, optimization strategy information, process parameter benchmarks, and historical optimization cases, providing reliable knowledge basis for optimization suggestions, avoiding the limitations of relying solely on human experience, and improving the universality and reliability of optimization suggestions. At the same time, it establishes a quantitative index system for path efficiency, quantitatively comparing indicators such as execution time, energy consumption, and action redundancy before and after optimization, making the optimization effect quantifiable and evaluable. Building upon this foundation, this application achieves automated parsing, alignment, analysis, and optimization suggestion generation for multimodal data, along with standardized code modification schemes, reducing manual intervention and effectively shortening the path optimization cycle. Furthermore, this application stores each optimization case and its effects in a knowledge base, enabling iterative knowledge updates. With the accumulation of cases, optimization accuracy and efficiency continuously improve, forming a virtuous cycle of optimization, accumulation, and further optimization. In summary, this application effectively shortens the robot's single-path cycle time, improves the overall production line efficiency, reduces robot energy consumption, and simultaneously reduces equipment wear caused by ineffective movements, extending equipment lifespan and lowering maintenance costs.

[0178] This disclosure also provides a method for automatically generating industrial robot code based on domain knowledge injection. Industrial robot programming is a core component of intelligent manufacturing production line deployment. Currently, industrial robots use specialized programming languages, whose syntax, motion instructions, and I / O control methods differ significantly from general-purpose programming languages, requiring programmers to possess both robot process knowledge and programming skills. Currently, industrial robot control code writing in the industry mainly relies on experienced engineers completing it manually, resulting in the following issues: First, there is a shortage of robot programming talent, and the training period is long; second, manual programming is inefficient, with a medium-complexity material handling program typically requiring 2 to 4 hours; third, code quality is highly dependent on individual experience, with programs written by different engineers showing significant differences in safety and standardization. With the development of large language model technology, AI code generation has made significant progress in the field of general programming. However, in the highly specialized field of industrial robots, general-purpose LLMs lack the syntactic constraints, motion control specifications, and safety protection knowledge of industrial robot-specific programming languages. Directly generated code often does not conform to industry standards and cannot be directly deployed and used.

[0179] The core application scenario of this method is the automatic generation of material handling programs for industrial robot robotic arms. This technical solution is applied to industrial robot robotic arm production lines in industries such as automotive parts, 3C electronics, and food and beverage. An industrial control computer is deployed on-site, and programmers input robot task requirements (such as picking up bottles from a conveyor belt and placing them into a 3x3 pallet grid) through a graphical interface. The system automatically calls a large language model combined with a domain knowledge base to generate control code that conforms to the specific coding standards for industrial robots. The generated code includes complete motion control instructions, I / O signal operations, safety protection logic, and error handling mechanisms. After review by the programmer, it can be directly downloaded to the robot control cabinet for execution.

[0180] Scenario Description: In the production line, a robotic arm performs bottle picking and placing tasks. The industrial control computer provides a code generation entry point through a graphical interface. The programmer selects the target LLM model, inputs the task description and process parameters, and the system automatically loads the programming knowledge base (30 manuals / tutorials) and coding rule base. By constraining the LLM with structured prompts, it generates a RAPID program containing three standard PROCs: MainModule, PickBottle, and PlaceBottle. The generation time is about 30 seconds. After the code is confirmed by the programmer, it is downloaded to the robot control cabinet through RobotStudio.

[0181] The solution proposed in this application can also be applied to scenarios of automatic generation of robot welding programs, automatic generation of robot gluing programs, and automatic generation of robot palletizing programs. The core logic is to generate control code that conforms to industry standards through domain knowledge injection and structured prompt word constraint LLM.

[0182] This application provides a method for automatic code generation of industrial robots based on domain knowledge injection. The overall process is executed in the logical order of knowledge base construction and loading, coding rule base construction and loading, structured prompt word construction, model calling and code generation, and code output and storage. Each step is described in detail below.

[0183] Step 1: Domain knowledge base construction and dynamic loading. This step provides professional knowledge support in the field of industrial robot programming for code generation. The main execution unit is the knowledge base loading module.

[0184] Operation 1: Build a domain knowledge base, which includes technical documents in the field of industrial robot programming, including programming language instruction reference manuals, robot operation manuals, detailed explanations of commonly used instructions, practical application techniques, signal configuration instructions, tool coordinate setting instructions, and other technical documents, covering the entire programming process, including motion control, signal operation, safety protection, and program debugging.

[0185] Step 2: Build a skill definition file to define the workflow specifications for robot programmers, including the input and output requirements and quality standards for code generation.

[0186] Operation 3: Obtain knowledge base content through a dual-path loading mechanism. Priority 1: Read skill definition files, reference documents, and program example files directly from the local file system and concatenate them into a structured context string; Priority 2: If the local files do not exist, load the skill definitions remotely via command-line tools to ensure availability in both offline and online scenarios.

[0187] Operation 4: Store the loaded knowledge base content as a skill context string (skill_context) as input data for subsequent prompt word construction.

[0188] Step 2: Construction and loading of the coding rule base. This step provides coding specification constraints for the robot programming language for code generation. The main execution body is the rule base loading module.

[0189] Operation 1: Build a coding rule base and define four core principles: safety priority principle (absolute safety, emergency stop reachability, zero collision design, speed grading), reliability principle (detection first, status confirmation, fault-tolerant design, timeout protection), maintainability principle (modular design, clear naming, sufficient comments, log output), and efficiency principle (quick movement on idle routes, precise slow movement, path optimization, parallel execution).

[0190] Operation 2: Define the program structure specifications, including module types and file formats (.modx / .sysx), minimal program structure (MODULE-BEGIN-PROC-ENDPROC-ENDMODULE standard template), and main program standard template (startup check → system initialization → main loop → cleanup and exit four-stage structure).

[0191] Operation 3: Define data type classification (value data, geometric data, configuration data), naming conventions (constants are all uppercase or camelCase, variables are lowercase with camelCase at the beginning, permanent data are prefixed by function), motion control conventions (four types of motion commands: MoveJ / MoveL / MoveC / MoveAbsJ and standard format), and input / output signal conventions (six types of signals: DI / DO / AI / AO / GI / GO, and their naming and operation instructions).

[0192] Operation 4: Define security protection specifications (interruption handling, area restriction) and error handling specifications (retry strategy, rollback handling, exception handling mechanism).

[0193] Operation 5: Merge the content of the encoding rule base with the content of the knowledge base, and use them together as the domain context input for the prompt words.

[0194] Step 3: Structured Prompt Construction. This step integrates user needs, domain knowledge, and coding constraints into structured prompts that the large language model can understand. The main execution unit is the prompt construction module, and the construction logic is divided into three layers.

[0195] First layer: Task requirement layer. The user's robot task requirements, described in natural language, are used as the primary content of the prompt words, clearly defining the code generation goal of the large language model.

[0196] The second layer is the domain knowledge layer. The skill context loaded in step 1 is used as a knowledge base (OpenSkill Knowledge Base) to inject prompt words, providing professional knowledge support for robot programming for the large language model, including API reference documents, program examples, and templates.

[0197] The third layer: the structured constraint layer. This layer embeds several key design requirements into the prompt words, imposing structural constraints on the code generation process of the large language model. Specifically, it includes: Constraint 1 - Loop Logic Constraint: Specifies the use of a WHILE loop instead of a FOR loop, waiting for the corresponding input signal before each specific action to implement event-driven, item-by-item processing logic; Constraint 2 - Speed ​​Optimization Constraint: Specifies the definition of high-speed data (v7500 := [7500, 500, 5000, 1000]) and low-speed data (v500 := [500, 50, 5000, 1000]), and sets acceleration limits (AccSet 80, 80) and speed limits (VelSet) at program startup. Constraints 100, 7500), high speed for rapid movement, low speed for precise positioning; Constraint 3 - Code Conciseness Constraint: A maximum of 3 to 4 program modules (Main, PickBottle, PlaceBottle) are allowed, prohibiting excessive modularization; Constraint 4 - Safe Motion Path Constraint: The offset function (Offs()) is used to dynamically generate intermediate points, employing a three-stage safe path of "high-speed approach → low-speed precise positioning → high-speed lifting"; Constraint 5 - Variable Scope Constraint: Variables shared across program modules (currentX, currentY, totalCount) must be declared as local variables (LOCAL VAR) at the module level, prohibiting their declaration within program modules to ensure accessibility by all program modules; Constraint 6 - Coordinate System Constraint: The system default tool (tool0) and base coordinate system are used to simplify environment configuration; Constraint 7 - Error Handling Constraint: Exception handling is added at the end of the main program module to handle signal timeout exceptions and halt execution; Constraint 8 - Output Format Constraint: Generated code must not contain comments, output instructions must only output specified types of content, and defined signals (DIPZ / DIXZ / DOPZ) cannot be redefined.

[0198] Step 4: Large Language Model Invocation and Code Generation. This step uses a unified model interface to invoke the large language model to generate code. The main execution components are the model invocation module and the code generation module.

[0199] Operation 1: Multi-model initialization, encapsulates the interface differences of different large language model providers, and realizes model routing switching through unified interface calls.

[0200] Operation 2: Message assembly. Assemble the system messages (setting the role of the large language model as a professional industrial robot control code programming expert) and the complete prompt words constructed in step 3 into a message list.

[0201] Operation 3: Large language model invocation, sending a message list to the model interface, the large language model generates robot control code based on domain knowledge and structured constraints in the prompt words.

[0202] Operation 4: Log the call, record the call time, model name, provider, and prompt word length, and save the complete prompt word and the large language model response as separate text files. Calculate the hash value of the response content for quality traceability.

[0203] Step 5: Code output and storage. This step delivers the code generated by the large language model to the user and persists it in storage. The main execution unit is the result output module.

[0204] Operation 1: Code Display. The generated code is displayed in real time through the code output area of ​​the graphical user interface, supporting streaming block-by-block output.

[0205] Operation 2: Automatic saving. After generation, the code will be automatically saved to the preset directory. The file name will be a standard module file containing timestamp information and will use a standard character encoding format.

[0206] Operation 3: Manual save. A file selection dialog box is provided, allowing users to specify the save path and file name. The file format is the standard module format of the target robot control program (such as .modx format).

[0207] The structured prompt word construction step (step 3) is the core step to solve the problem that the code generated by the general large language model does not conform to industry standards. Through a three-layer structure (task requirements + domain knowledge + multiple constraints), industrial programming knowledge is systematically injected into the context of the large language model, which breaks through the limitation of the general large language model lacking domain knowledge.

[0208] Domain knowledge base construction and loading steps (step 1): This is the foundational step of the entire method. Through the accumulation of knowledge from multiple technical documents and the dual-path loading mechanism, it provides professional knowledge support for code generation that is not available in the training data of the general large language model.

[0209] The coding rule base construction and loading step (step 2) is a key step to ensure code security and standardization. Through the constraints of four core principles and program structure specifications, it ensures that the generated code meets the security standards for industrial deployment.

[0210] In summary, the embodiments disclosed above significantly improve the conformity rate of generated code module structure, variable declaration, and motion instruction format to industry standards by injecting target programming manuals and coding rules as context constraints. This is far higher than the conformity rate of code directly generated by general-purpose large language models. Simultaneously, the generated code automatically includes safety logic such as exception handling, safe motion paths, and speed grading, eliminating the potential safety risks of code generated by general-purpose large language models and meeting industrial deployment safety standards. Furthermore, by solidifying prompt word templates and multiple structured constraints, the consistency rate of code structure generated multiple times for the same task requirements is significantly improved, effectively reducing the workload of code review and correction. In terms of efficiency, this application reduces code writing time from hours to seconds, and the generated results can be deployed with minimal manual correction, significantly lowering the technical threshold and labor costs of industrial robot programming. Moreover, the knowledge base and rule base are organized in file form, supporting continuous expansion (adding programming manuals and process documents). The quality of large language model generation automatically improves after new knowledge is injected, forming an accumulative domain knowledge system and avoiding the repetitive organization of knowledge from scratch for each programming iteration.

[0211] Figure 5 This is a schematic diagram of the structure of a robot task execution adjustment device 500 provided in an embodiment of this disclosure. Figure 5 As shown, the device includes: a determining module 510, used to determine the alignment result of the robot's first visual data and the control command sequence in response to robot operation, the control command sequence being used to control the robot to perform a task; an analysis module 520, used to analyze the robot's task execution process based on the alignment result to obtain the items to be adjusted during the task execution process; a query module 530, used to query knowledge information related to the items to be adjusted from a knowledge base based on the information associated with the items to be adjusted; and a processing module 540, used to generate a target adjustment scheme based on the knowledge information, the target adjustment scheme being used to adjust the items to be adjusted.

[0212] In some embodiments of this disclosure, the analysis module is further configured to determine the efficiency parameters of the robot during task execution based on the alignment results. The efficiency parameters include at least one of time efficiency parameters, energy efficiency parameters, and accuracy parameters. The robot's task execution process is analyzed based on the efficiency parameters to obtain the adjustment items to be adjusted during the task execution process. The adjustment items include at least one of explicit adjustment items and implicit adjustment items, wherein the explicit adjustment items are adjustment items associated with the control command sequence, and the implicit adjustment items are adjustment items associated with the robot's actual execution state and / or the robot's operating environment.

[0213] In some embodiments of this disclosure, the time efficiency parameters include at least one of the following: the execution time of a single action performed by the robot during task execution, the overall path loop time corresponding to the task execution process, the proportion of robot waiting time during task execution, and the redundant time consumed by the robot when performing actions during task execution; the energy efficiency parameters include at least one of the following: the loss of the robot's moving mechanism during task execution, the total energy consumption of the robot during task execution, and the proportion of energy consumption of the robot's invalid motion during task execution; the accuracy parameters include at least one of the following: the positional deviation of the robot's actions during task execution, the time deviation between the start of execution of the control command sequence and the start of the robot's actions, and the similarity between the action features corresponding to the control command sequence and the action features actually performed by the robot.

[0214] In some embodiments of this disclosure, the analysis module is further configured to determine the deviation of the efficiency parameter from the standard efficiency parameter; in response to the deviation being greater than a first threshold, determine the priority of the efficiency parameter as a first priority; or in response to the deviation being less than or equal to the first threshold and greater than or equal to a second threshold, determine the priority of the efficiency parameter as a second priority; or in response to the deviation being less than the second threshold, determine the priority of the efficiency parameter as a third priority, wherein the first priority is higher than the second priority and the third priority is lower than the second priority; and analyze the robot's task execution process according to the priority of the efficiency parameter to obtain the items to be adjusted by the robot during the task execution process.

[0215] In some embodiments of this disclosure, the knowledge information related to the item to be adjusted includes at least one of the following: safety constraint information, path adjustment strategy, standard process parameters, and historical adjustment cases; based on the knowledge information, a target adjustment scheme is generated.

[0216] In some embodiments of this disclosure, the information associated with the item to be adjusted includes at least one of the following: the type of the item to be adjusted; the first visual data corresponding to the item to be adjusted; the control command sequence corresponding to the item to be adjusted; the efficiency parameter corresponding to the item to be adjusted; the priority of the item to be adjusted, wherein the priority of the item to be adjusted is related to the priority of the efficiency parameter corresponding to the item to be adjusted; the type of robot; and the operating environment of the robot.

[0217] In some embodiments of this disclosure, the target adjustment scheme includes at least one of the following: modifying the control command sequence; modifying the parameters of the target device, the target device including at least one of the following: a robot, a device in the robot's operating environment, or a device that collaborates with the robot during task execution; modifying the process parameters associated with the robot; and outputting prompt information to indicate at least one of the following: the control command sequence needs to be modified, the parameters of the target device need to be modified, or the process parameters associated with the robot need to be modified.

[0218] In some embodiments of this disclosure, the prompt information also includes at least one of the following: information associated with the item to be adjusted; the adjusted control command sequence; the position of the adjusted control command sequence; the control command sequence before adjustment; the basis for modification; and the expected adjustment effect.

[0219] In some embodiments of this disclosure, the processing module is further configured to generate task requirement information based on the information associated with the item to be adjusted; generate prompt words based on at least one of the task requirement information, information in the knowledge base, and information in the rule base, wherein the rule base includes rules for generating control instruction sequences; and call the target model to generate the adjusted control instruction sequence based on the prompt words and structured constraint information.

[0220] In some embodiments of this disclosure, the processing module is also used to acquire multiple technical documents related to the task execution process. The multiple technical documents include at least one of the following: a robot programming language instruction reference manual, a robot operation manual, a detailed explanation of robot control instructions, an application skills document, a signal configuration instruction document, and a tool coordinate setting instruction document; and to build a knowledge base based on the multiple technical documents.

[0221] In some embodiments of this disclosure, the processing module is also used to construct a skill definition file, which defines the programming process specifications for the robot; and to construct a knowledge base based on the skill definition file and multiple technical documents.

[0222] In some embodiments of this disclosure, the processing module is further configured to acquire coding rule information of the robot, including rules for generating control instruction sequences; and to construct a rule base based on the coding rule information.

[0223] In some embodiments of this disclosure, the encoding rule information includes at least one of the following: security priority rules; reliability rules; maintainability rules; efficiency rules; program structure specifications; data type classification; naming specifications; motion control instruction format specifications; input and output signal specifications; security protection specifications; and error handling specifications.

[0224] In some embodiments of this disclosure, the structured constraint information includes at least one of the following: loop logic constraint information; speed optimization constraint information; control instruction sequence conciseness constraint information; safe motion path constraint information; variable scope constraint information; coordinate system constraint information; error handling constraint information; and control instruction sequence output format constraint information.

[0225] In some embodiments of this disclosure, the processing module is also used to obtain information from a locally stored knowledge base; and to obtain information from the knowledge base via a command-line tool.

[0226] In some embodiments of this disclosure, the processing module is also used to generate a call log, which includes at least one of the following: information about the target model, the time of calling the target model, information related to the prompt words, and the adjusted sequence of control instructions.

[0227] In some embodiments of this disclosure, the processing module is also configured to display the adjusted sequence of control commands on a display interface.

[0228] In some embodiments of this disclosure, the processing module is also configured to automatically save the adjusted control command sequence; or to save the adjusted control command sequence in response to a user's save command.

[0229] In some embodiments of this disclosure, the processing module is further configured to control the robot to perform tasks based on the adjusted control command sequence in response to a user's confirmation operation of the adjusted control command sequence.

[0230] In some embodiments of this disclosure, the processing module is further configured to perform a security check on the adjusted control command sequence; if the security check passes, in response to the user's confirmation operation on the adjusted control command sequence, the robot is controlled to perform the task based on the adjusted control command sequence.

[0231] In some embodiments of this disclosure, the processing module is further configured to collect second visual data of the robot's operation and execution log of the adjusted control command sequence during the process of controlling the robot to perform a task based on the adjusted control command sequence; determine the adjustment effect of the item to be adjusted based on the second visual data and the execution log; process at least one of the information associated with the item to be adjusted, the information of the target adjustment scheme, and the adjustment effect into a historical adjustment case and save it to the knowledge base.

[0232] In some embodiments of this disclosure, the determining module is further configured to perform motion detection on the first visual data, identify at least one event log corresponding to the first visual data; parse the control command sequence to obtain at least one control command log corresponding to the control command sequence; and align the at least one event log and the at least one control command log according to the time information and motion features of the at least one event log and the time information and motion features of the at least one control command log to obtain an alignment result.

[0233] In some embodiments of this disclosure, the determining module is further configured to timestamp and synchronize the first visual data and the control command sequence; and to determine the alignment result of the first visual data and the control command sequence when the robot is running.

[0234] In summary, the robot task execution adjustment device 500 can identify items that need adjustment during task execution based on the alignment results of the robot's visual data and control command execution sequence. It can accurately determine the areas that need adjustment due to unreasonable control command sequence settings, actual operating environment, or deviations in robot execution performance, thereby improving the comprehensiveness and accuracy of item identification. Then, it generates targeted adjustment schemes for the accurately identified items, enabling automatic adjustment of the robot's task execution process in actual operating scenarios, and improving the efficiency and reliability of task execution process adjustment.

[0235] The methods and apparatus provided in the embodiments of this application have been described above. To implement the functions of the methods provided in the embodiments of this application, the electronic device may include a hardware structure and software modules, and may implement the above functions in the form of a hardware structure, software modules, or a hardware structure plus software modules. One of the above functions may be executed in the form of a hardware structure, software modules, or a hardware structure plus software modules.

[0236] Figure 6 This is a block diagram illustrating an electronic device 600 for implementing the above-described method according to an exemplary embodiment. For example, the electronic device 600 may be a mobile phone, computer, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, robot, etc.

[0237] Reference Figure 6 The electronic device 600 may include one or more of the following components: a processing component 602, a memory 604, a power supply component 606, a multimedia component 608, an audio component 610, an input / output (I / O) interface 612, a sensor component 614, and a communication component 616.

[0238] Processing component 602 typically controls the overall operation of electronic device 600, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 602 may include one or more processors 620 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 602 may include one or more modules to facilitate interaction between processing component 602 and other components. For example, processing component 602 may include a multimedia module to facilitate interaction between multimedia component 608 and processing component 602.

[0239] Memory 604 is configured to store various types of data to support the operation of electronic device 600. Examples of such data include instructions for any application or method operating on electronic device 600, contact data, phonebook data, messages, pictures, videos, etc. Memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0240] Power supply component 606 provides power to various components of electronic device 600. Power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 600.

[0241] Multimedia component 608 includes a screen that provides an output interface between electronic device 600 and user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 608 includes a front-facing camera and / or a rear-facing camera. When electronic device 600 is in an operating mode, such as a shooting mode or video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0242] Audio component 610 is configured to output and / or input audio signals. For example, audio component 610 includes a microphone (MIC) configured to receive external audio signals when electronic device 600 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 604 or transmitted via communication component 616. In some embodiments, audio component 610 also includes a speaker for outputting audio signals.

[0243] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0244] Sensor assembly 614 includes one or more sensors for providing state assessments of various aspects of electronic device 600. For example, sensor assembly 614 may detect the on / off state of electronic device 600, the relative positioning of components such as the display and keypad of electronic device 600, changes in position of electronic device 600 or a component of electronic device 600, the presence or absence of user contact with electronic device 600, orientation or acceleration / deceleration of electronic device 600, and temperature changes of electronic device 600. Sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 614 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0245] Communication component 616 is configured to facilitate wired or wireless communication between electronic device 600 and other devices. Electronic device 600 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G LTE, 5G NR (NewRadio), or combinations thereof. In one exemplary embodiment, communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 616 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0246] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0247] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, which can be executed by a processor 620 of an electronic device 600 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0248] Embodiments of this disclosure also provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described in the above embodiments of this disclosure.

[0249] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0250] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in at least one embodiment or example.

[0251] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0252] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having at least one wiring (control method), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0253] It should be understood that various parts of the embodiments of this disclosure can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0254] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0255] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a single processing module, or each unit can exist physically separately, or two or more units can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The aforementioned storage medium can be a read-only memory, a hard disk, or an optical disk, etc.

[0256] Although embodiments of the present disclosure have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A method for adjusting the task execution of a robot, characterized in that, The method includes: In response to the operation of the robot, an alignment result of the robot's first visual data and a sequence of control commands is determined, the sequence of control commands being used to control the robot to perform a task; Based on the alignment results, the task execution process of the robot is analyzed to obtain the items to be adjusted during the task execution process. Based on the information associated with the item to be adjusted, query the knowledge base for knowledge information related to the item to be adjusted; Based on the knowledge information, a target adjustment plan is generated, which is used to adjust the item to be adjusted.

2. The method according to claim 1, characterized in that, The step involves analyzing the robot's task execution process based on the alignment result to obtain the items to be adjusted during the task execution process, including: Based on the alignment result, the efficiency parameters of the robot during task execution are determined, and the efficiency parameters include at least one of time efficiency parameters, energy consumption efficiency parameters, and accuracy parameters. The robot's task execution process is analyzed based on the efficiency parameters to obtain the adjustment items for the robot during the task execution process; The items to be adjusted include at least one of explicit items to be adjusted and implicit items to be adjusted, wherein the explicit items to be adjusted are those associated with the control instruction sequence, and the implicit items to be adjusted are those associated with the actual execution state of the robot and / or the operating environment of the robot.

3. The method according to claim 2, characterized in that, The time efficiency parameters include at least one of the following: the execution time of a single action performed by the robot during the task execution process, the overall path loop time corresponding to the task execution process, the proportion of the robot's waiting time during the task execution process, and the redundant time consumed by the robot when performing actions during the task execution process; The energy efficiency parameters include at least one of the following: the loss of the robot's moving mechanism during the task execution, the total energy consumption of the robot during the task execution, and the proportion of energy consumption from the robot's ineffective movements during the task execution; The accuracy parameters include at least one of the following: the positional deviation of the robot's actions during the task execution process, the time deviation between the start of the control command sequence and the start of the robot's actions, and the similarity between the action features corresponding to the control command sequence and the action features corresponding to the visual data.

4. The method according to claim 2, characterized in that, The step of analyzing the robot's task execution process based on the efficiency parameters to obtain the adjustment items for the robot during task execution includes: Determine the deviation between the efficiency parameter and the standard efficiency parameter; In response to the deviation being greater than a first threshold, the priority of the efficiency parameter is determined to be the first priority; or In response to the deviation being less than or equal to the first threshold and greater than or equal to the second threshold, the priority of the efficiency parameter is determined to be the second priority; or In response to the deviation being less than the second threshold, the priority of the efficiency parameter is determined to be the third priority, where the first priority is higher than the second priority and the third priority is lower than the second priority; The robot's task execution process is analyzed according to the priority of the efficiency parameters to obtain the items to be adjusted in the task execution process.

5. The method according to claim 1, characterized in that, The knowledge information related to the item to be adjusted includes at least one of the following: safety constraint information, path adjustment strategy, standard process parameters, and historical adjustment cases.

6. The method according to claim 1, characterized in that, The information associated with the item to be adjusted includes at least one of the following: The type of the item to be adjusted; The first visual data corresponding to the item to be adjusted; The sequence of control instructions corresponding to the item to be adjusted; The efficiency parameter corresponding to the item to be adjusted; The priority of the item to be adjusted is related to the priority of the efficiency parameter corresponding to the item to be adjusted; The type of robot; The operating environment of the robot.

7. The method according to claim 1, characterized in that, The target adjustment scheme includes at least one of the following: Modify the control command sequence; Modify the parameters of the target device, wherein the target device includes at least one of the following: the robot, the device in the robot's operating environment, and the device that collaborates with the robot during the task execution process; Modify the process parameters associated with the robot; Output a prompt message, which indicates at least one of the following: the control command sequence needs to be modified, the parameters of the target device need to be modified, or the process parameters associated with the robot need to be modified.

8. The method according to claim 7, characterized in that, The prompt message includes at least one of the following: The information associated with the item to be adjusted; Adjusted control command sequence; The position of the adjusted control command sequence; The sequence of control commands before adjustment; Basis for modification; Expected adjustment effect.

9. The method according to claim 8, characterized in that, The method further includes: Based on the information associated with the items to be adjusted, generate task requirement information; Based on at least one of the task requirement information, information in the knowledge base, and information in the rule base, a prompt word is generated, wherein the rule base includes the generation rules for the control instruction sequence; The target model is invoked to generate the adjusted control instruction sequence based on the prompt words and structured constraint information.

10. The method according to claim 9, characterized in that, The method further includes: Obtain multiple technical documents related to the task execution process, including at least one of the following: the robot's programming language instruction reference manual, the robot's operation manual, the robot's control instruction detailed explanation document, application skills document, signal configuration instruction document, and tool coordinate setting instruction document; The knowledge base is constructed based on the aforementioned technical documents.

11. The method according to claim 10, characterized in that, The construction of the knowledge base based on the plurality of technical documents includes: Construct a skill definition file, which is used to define the programming process specifications for the robot; The knowledge base is constructed based on the skill definition file and the multiple technical documents.

12. The method according to claim 9, characterized in that, The method further includes: Obtain the robot's encoding rule information, which includes the generation rules for the control command sequence; The rule base is constructed based on the encoding rule information.

13. The method according to claim 12, characterized in that, The encoding rule information includes at least one of the following: Safety first rule; Reliability rules; Maintainability rules; Efficiency rules; Program structure specifications; Data type classification; Naming conventions; Motion control command format specifications; Input and output signal specifications; Safety protection standards; Error handling guidelines.

14. The method according to claim 9, characterized in that, The structured constraint information includes at least one of the following: Loop logic constraint information; Speed ​​optimization constraint information; The control command sequence is concise and includes constraint information. Safe motion path constraint information; Variable scope constraint information; Coordinate system constraint information; Error handling constraint information; The output format constraint information of the control instruction sequence.

15. The method according to claim 9, characterized in that, The method further includes at least one of the following: Retrieve information from the locally stored knowledge base; Information from the knowledge base can be obtained using command-line tools.

16. The method according to claim 9, characterized in that, The method further includes: Generate a call log, which includes at least one of the following: information about the target model, the time when the target model was called, information related to the prompt words, and the adjusted sequence of control instructions.

17. The method according to claim 9, characterized in that, The method further includes: The adjusted control command sequence is displayed on the display interface.

18. The method according to claim 9, characterized in that, The method further includes: Automatically save the adjusted control command sequence; or In response to the user's save command, the adjusted control command sequence is saved.

19. The method according to claim 9, characterized in that, The method further includes: In response to the user's confirmation of the adjusted control command sequence, the robot is controlled to perform the task based on the adjusted control command sequence.

20. The method according to claim 19, characterized in that, The step of responding to the user's confirmation of the adjusted control command sequence and controlling the robot to perform tasks based on the adjusted control command sequence includes: The adjusted control command sequence is subjected to security verification; If the security check passes, in response to the user's confirmation of the adjusted control command sequence, the robot is controlled to perform the task based on the adjusted control command sequence.

21. The method according to claim 19, characterized in that, The method further includes: During the process of controlling the robot to perform tasks based on the adjusted control command sequence, the second vision data of the robot's operation and the execution log of the adjusted control command sequence are collected. Based on the second visual data and the execution log, determine the adjustment effect of the item to be adjusted; At least one of the following: the information associated with the item to be adjusted, the information of the target adjustment scheme, and the adjustment effect, is processed into an adjustment case and saved to the knowledge base.

22. The method according to claim 1, characterized in that, The alignment result of determining the robot's first visual data and control command sequence includes: Action detection is performed on the first visual data to identify at least one event log corresponding to the first visual data; The control command sequence is parsed to obtain at least one control command log corresponding to the control command sequence; Based on the time information and action characteristics of the at least one event log and the time information and action characteristics of the at least one control command log, the at least one event log and the at least one control command log are aligned to obtain the alignment result.

23. A task execution adjustment device for a robot, characterized in that, include: A determination module is configured to, in response to the operation of the robot, determine the alignment result of the robot's first visual data and a sequence of control commands, the sequence of control commands being used to control the robot to perform a task; An analysis module is used to analyze the robot's task execution process based on the alignment results, and to obtain the items to be adjusted by the robot during the task execution process; The query module is used to query knowledge information related to the item to be adjusted from the knowledge base based on the information associated with the item to be adjusted. The processing module is used to generate a target adjustment plan based on knowledge information, and the target adjustment plan is used to adjust the item to be adjusted.

24. The apparatus according to claim 23, characterized in that, The analysis module is also used for: Based on the alignment result, the efficiency parameters of the robot during task execution are determined, and the efficiency parameters include at least one of time efficiency parameters, energy consumption efficiency parameters, and accuracy parameters. The robot's task execution process is analyzed based on the efficiency parameters to obtain the adjustment items for the robot during the task execution process; The items to be adjusted include at least one of explicit items to be adjusted and implicit items to be adjusted, wherein the explicit items to be adjusted are those associated with the control instruction sequence, and the implicit items to be adjusted are those associated with the actual execution state of the robot and / or the operating environment of the robot.

25. The apparatus according to claim 23, characterized in that, The knowledge information related to the item to be adjusted includes at least one of the following: safety constraint information, path adjustment strategy, standard process parameters, and historical adjustment cases.

26. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-22.

27. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-22.

Images