Graphical programming compiling system, method and device for cooperative mechanical arm

Abstract

The invention relates to a graphical programming compiling system, method and device for a collaborative mechanical arm. The system comprises a user terminal, a compiler, an actuator and a collaborative mechanical arm, the user terminal responds to code writing operation of a user on a human-computer interaction interface and / or dragging operation of graphic elements, program data is generated, and the program data is issued to the compiler; the compiler converts the program data into a cooperative mechanical arm control instruction, an external equipment control instruction and a logic control instruction; the executor executes the logic control instruction and sends the cooperative mechanical arm control instruction and the external equipment control instruction to the cooperative mechanical arm and the external equipment respectively; the mechanical arm and the external equipment are cooperated to execute matched tasks, and first task feedback information and second task feedback information are generated and fed back to the actuator; and the executor generates an execution result according to the feedback information and pushes the execution result to the user terminal. By adopting the method, the use difficulty of the cooperative mechanical arm is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of collaborative robot control, and particularly to a graphical programming compilation method, device, computer device, computer-readable storage medium, and computer program product for a collaborative robotic arm. Background Art

[0002] With the rapid development of the robotic arm industry, collaborative robotic arms are gradually emerging from the exclusive domain of industrial robotic arms and are widely used in large and small factories, stores, and even educational institutions. Collaborative robotic arms not only simplify the complexity of traditional industrial robotic arms in terms of operation, programming, and deployment, but also play an irreplaceable role in fields such as welding, palletizing, physiotherapy, education, and CNC (Computer Numerical Control) loading and unloading by virtue of their human-robot collaboration advantages.

[0003] However, there are many cases in the existing technology and the market where the programming interfaces of collaborative robotic arms are not very user-friendly for non-professional users. The graphical programming tools provided in the existing technology are difficult to cover the logical requirements in the actual scenarios of collaborative robotic arms, resulting in a relatively high usage threshold for collaborative robotic arms. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide a graphical programming compilation system for a collaborative robotic arm that can reduce the usage difficulty of the collaborative robotic arm, as well as a graphical programming compilation method, device, computer device, computer-readable storage medium, and computer program product for the collaborative robotic arm.

[0005] In a first aspect, the present application provides a graphical programming compilation system for a collaborative robotic arm, including a user terminal, a compiler, an executor, and a collaborative robotic arm:

[0006] The user terminal generates program data in response to the user's code writing operation on the human-computer interaction interface and / or the dragging operation of graphical elements, and sends the program data to the compiler;

[0007] The compiler converts the program data into an instruction set and sends the instruction set to the executor. The instruction set includes collaborative robotic arm control instructions, external device control instructions, and logic control instructions;

[0008] The executor executes the logic control instructions, sends the collaborative robotic arm control instructions to the collaborative robotic arm, and sends the external device control instructions to the external device;

[0009] The collaborative robotic arm executes the tasks matching the collaborative robotic arm control instructions, generates first task feedback information, and sends the first task feedback information to the executor;

[0010] The external device executes a task that matches the external device control instruction, generates second task feedback information, and sends the second task feedback information to the actuator;

[0011] The actuator generates an execution result based on the first task feedback information and the second task feedback information, and pushes the execution result to the user terminal.

[0012] In one embodiment, the graphical element includes a teaching point; the user terminal determines the target point of the collaborative robotic arm and generates target point data in response to the user's dragging operation on the teaching point in the human-machine interaction interface.

[0013] In one embodiment, the graphical element further includes a logic instruction; the user terminal generates logic flow data for defining the target task execution of the collaborative robotic arm in response to the user's dragging operation on the logic instruction in the human-machine interaction interface.

[0014] In one embodiment, the user terminal generates teaching control instruction data in response to the user's parameter configuration of the predefined teaching template instruction.

[0015] In one embodiment, the user terminal obtains the code data written by the user in response to the user's code writing operation; the program data includes at least one of the target point data, task flow data, teaching control instruction data, and code data.

[0016] In one embodiment, the user terminal is further configured to store the program data according to a preset data structure format to obtain a program data file, and send the program data file to the compiler.

[0017] In one embodiment, the compiler converts the program data into an instruction set in Python format according to the preset compilation rules.

[0018] In one embodiment, after the collaborative robotic arm executes the task corresponding to the collaborative robotic arm control instruction, it generates position information, attitude information, torque information, and speed information, integrates the position information, attitude information, torque information, and speed information into feedback information, and feeds back the execution status and feedback information of the task to the actuator. The first task feedback information includes the execution status and feedback information of the task;

[0019] After the external device executes the task corresponding to the external device control instruction, it feeds back the execution status and task feedback information of the task to the actuator. The second task feedback information includes the execution status and task feedback information of the task.

[0020] In one embodiment, the actuator is further configured to adjust the control logic of the collaborative robotic arm control instruction and return the adjusted instruction to the collaborative robotic arm when the first task feedback information indicates that the task execution is abnormal;

[0021] In the case where the execution of the second task feedback information representation task is abnormal, adjust the control logic of the external device control instruction, and return the adjusted instruction to the external device.

[0022] In a second aspect, the present application further provides a collaborative robotic arm graphical programming compilation method, including:

[0023] In response to a user's code writing operation on the human-machine interaction interface and / or a drag operation of graphical elements, generate program data;

[0024] Based on preset compilation rules, convert the program data into a Python format instruction set, and the instruction set includes robotic arm control instructions, external device control instructions, and logic control instructions;

[0025] Execute the logic control instructions, control the collaborative robotic arm according to the robotic arm control instructions, and control the external device according to the external device control instructions;

[0026] In the case of receiving task feedback information from the collaborative robotic arm and / or the external device, generate an execution result according to the task feedback information, and push the execution result to the user terminal.

[0027] In a third aspect, the present application further provides a collaborative robotic arm graphical programming compilation device, including:

[0028] A program data generation module, configured to generate program data in response to a user's code writing operation on the human-machine interaction interface and / or a drag operation of graphical elements;

[0029] A compilation module, configured to convert the program data into a Python format instruction set based on preset compilation rules, and the instruction set includes robotic arm control instructions, external device control instructions, and logic control instructions;

[0030] A control module, configured to execute the logic control instructions, control the collaborative robotic arm according to the robotic arm control instructions, and control the external device according to the external device control instructions;

[0031] A feedback module, configured to generate an execution result according to the task feedback information in the case of receiving task feedback information from the collaborative robotic arm and / or the external device, and push the execution result to the user terminal.

[0032] In a fourth aspect, the present application further provides a computer device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the steps in any one of the above-mentioned embodiments of the collaborative robotic arm graphical programming compilation method are implemented.

[0033] Fifth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the embodiments of any one of the above collaborative robotic arm graphical programming compilation methods are implemented.

[0034] Sixth aspect, the present application further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps in the embodiments of any one of the above collaborative robotic arm graphical programming compilation methods are implemented.

[0035] For the above collaborative robotic arm graphical programming compilation system, on the one hand, it innovatively combines the dual programming modes of graphical programming and third-party programming language expansion, and is applied to collaborative robotic arms. By the user's code writing operation on the human-machine interaction interface and / or the dragging operation of graphical elements, program data for controlling the collaborative robotic arm is generated, greatly reducing the usage threshold and programming difficulty of controlling the collaborative robotic arm. For users without a deep programming background, they can quickly achieve robotic arm control and task arrangement by dragging the graphical elements of the human-machine interaction interface. For professionals or those with complex application requirements, they can meet the complex logic requirements in the actual scenario through graphical programming combined with a small amount of code writing, improving the adaptability to complex application scenarios and functional scalability, thus taking into account the control requirements of different users and being conducive to improving the popularity and application scope of the system; on the other hand, the compiler converts the program data into robotic arm control instructions, external device control instructions, and logic control instructions, so that the actuator, according to the logic control instructions, sends the robotic arm control instructions to the collaborative robotic arm and sends the external device control instructions to the external device, supporting the flexible design of using the collaborative robotic arm as the core control unit to coordinate external devices to jointly complete the tasks corresponding to the instructions and cooperate with other systems, thereby improving the adaptability of the system to various application requirements of single-device control and multi-device linkage and being conducive to enhancing the collaborative ability and task execution efficiency of the system; on the further hand, through the feedback and closed-loop control mechanism among the human-machine interaction interface, collaborative robotic arm, external device, and actuator, it is conducive to real-time monitoring of the task execution situation and ensuring the high efficiency and safety of the system.

[0036] The above collaborative robotic arm graphical programming compilation method, device, computer device, computer-readable storage medium, and computer program product innovatively combine a graphical programming method and a dual programming mode of third-party programming language extension for the control of the collaborative robotic arm. In response to the user's code writing operation and / or dragging operation of graphical elements on the human-machine interaction interface, program data is generated. Based on preset compilation rules, the program data is converted into an instruction set in the target programming language format to control the collaborative robotic arm and its external devices, reducing the programming difficulty of the collaborative robotic arm, improving the simplicity of user robotic arm control and task arrangement, comprehensively considering users with different usage requirements, enabling function customization according to actual needs, and improving the adaptability to complex scenario logic control. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for describing the embodiments of the present application or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other related drawings can be obtained based on these drawings.

[0038] Figure 1 It is a structural block diagram of a collaborative robotic arm graphical programming compilation system in an embodiment;

[0039] Figure 2 It is a schematic flowchart of a collaborative robotic arm graphical programming compilation system in an embodiment;

[0040] Figure 3 It is an application environment diagram of a collaborative robotic arm graphical programming compilation method in an embodiment;

[0041] Figure 4 It is a schematic flowchart of a collaborative robotic arm graphical programming compilation method in an embodiment;

[0042] Figure 5 It is a structural block diagram of a collaborative robotic arm graphical programming compilation device in an embodiment;

[0043] Figure 6 It is an internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] In order to make the objectives, technical solutions, and advantages of the present application clearer, the following further details the present application in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] In an exemplary embodiment, as Figure 1As shown in the figure, a collaborative robotic arm graphical programming compilation system is provided, including a user terminal 110, a compiler 120, an actuator 130, and a collaborative robotic arm 140, where:

[0046] The user terminal 110 generates program data in response to the user's code writing operation on the human-machine interface and / or the dragging operation of graphical elements, and sends the program data to the compiler 120.

[0047] Among them, the graphical elements include predefined components or modules, such as movement commands, grasping actions, conditional judgments, and loop structures.

[0048] In practical applications, the user can select the required graphical elements in the human-machine interface, drag and drop the graphical elements to the working area of the human-machine interface, and adjust the positions of the graphical elements through the dragging operation to configure the required program. To meet complex application requirements, such as scenarios that require external device interaction or have complex logical processes, an extended function module for a third-party programming language is also provided in the human-machine interface. The user can write code in the human-machine interface through the user terminal using a third-party programming language (such as Python) to add code blocks to meet the application requirements. The user terminal generates the program data created by the user in response to the code writing operation and / or the dragging operation of graphical elements in the human-machine interface, and sends the program data to the compiler.

[0049] The compiler 120 converts the program data into an instruction set and sends the instruction set to the actuator 130. The instruction set includes collaborative robotic arm control instructions, external device control instructions, and logical control instructions.

[0050] Among them, the logical control instructions can include logical branch judgments, conditional loops, etc. The robotic arm control instructions can include instructions related to the basic movement and composite movement of the collaborative robotic arm. The peripheral control instructions can include instructions related to peripheral interactions such as TCP (Transmission Control Protocol) communication and MODBUS protocol control.

[0051] In practical applications, it can be to pre-set the mapping rules between the graphical elements in the human-machine interface and the Python code, and convert the program data into instructions in the Python code format through the mapping rules. Specifically, it can be to define a corresponding Python code segment for each graphical element in the human-machine interface and construct a mapping table. After receiving the program data, the compiler parses the program data constructed by the user in the human-machine interface, identifies the logical relationship, the program data for controlling the collaborative robotic arm, and the program data for controlling external devices, and respectively converts them into logical control instructions, external device control instructions, and collaborative robotic arm control instructions in the Python code format according to the pre-constructed mapping table.

[0052] The actuator 130 executes the logic control instruction, sends the collaborative robot arm control instruction to the collaborative robot arm 140, and sends the external device control instruction to the external device.

[0053] Among them, the external device may include but is not limited to sensors and actuators, etc.

[0054] In practical applications, the actuator completes the logic control according to the logic conditions of the logic control instruction. The actuator sends the collaborative robot arm control instruction to the robot arm in real time through the communication interface with the collaborative robot arm (such as TCP / IP) to complete the operation of the target point and the execution of the task process. The operation and coordination of external devices such as sensors and actuators are realized through the communication protocol with the external device (such as TCP, MODBUS, etc.).

[0055] The collaborative robot arm 140 executes the task matching the collaborative robot arm control instruction, generates the first task feedback information, and sends the first task feedback information to the actuator 130.

[0056] Among them, the first task feedback information is used to characterize the task execution situation, and may include system status update information, position information, torque data, etc.

[0057] In practical applications, the collaborative robot arm receives the collaborative robot arm control instruction, executes the matching task, such as moving to a specific position, grasping an object, etc. During the task execution process, the system status update information, position information, and torque data are sent to the actuator.

[0058] The external device executes the task matching the external device control instruction, generates the second task feedback information, and sends the second task feedback information to the actuator 130.

[0059] Among them, the second task feedback information is used to characterize the task execution situation, and may include device status update information and execution data, etc.

[0060] In practical applications, the external device receives the external device control instruction, executes the matching task, such as reading sensor data, etc. During the task execution process, the device status update and execution data (such as if the external device includes a sensor, the sensor data is fed back) are sent to the actuator.

[0061] Through the above design of dual-mode programming, compilation, and instruction execution, the limitations of function expansion are broken through, and the flexibility and scalability of the collaborative robotic arm system are improved. Specifically, the user can use the collaborative robotic arm as the core master control unit to coordinate external devices including sensors, actuators, etc. through predefined interfaces and protocols to jointly complete task scheduling in complex scenarios. The collaborative robotic arm can also be used as an auxiliary unit, scheduled by a more advanced master control program, and efficiently cooperate with other systems through an open API architecture. Thus, through the dual programming mode, it is ensured that the system can adapt to various application requirements from single-device control to multi-device linkage, expanding the usage scenarios of the collaborative robotic arm.

[0062] The actuator 130 generates an execution result based on the first task feedback information and the second task feedback information, and pushes the execution result to the user terminal 110.

[0063] In practical applications, the actuator evaluates whether the task execution status is normal based on the first task feedback information and the second task feedback information. After the task is completed, it summarizes the log messages of the events during the task and the final result to generate an execution result, and pushes the execution result to the user side through a communication channel (such as WebSocket, REST API, etc.).

[0064] In this embodiment, on the one hand, innovatively combining the dual programming modes of graphical programming method and the extension of third-party programming languages, it is applied to the collaborative robotic arm. Through the code writing operation of the user on the human-computer interaction interface and / or the dragging operation of graphical elements, program data for controlling the collaborative robotic arm is generated, greatly reducing the usage threshold and programming difficulty of controlling the collaborative robotic arm. For users without a deep programming background, they can quickly realize robotic arm control and task arrangement by dragging the graphical elements of the human-computer interaction interface. For professionals or those with complex application requirements, they can meet the complex logic requirements in the actual scenario through graphical programming combined with a small amount of code writing, improving the adaptability to complex application scenarios and functional scalability, thus taking into account the control requirements of different users and being conducive to improving the popularity and application scope of the system. On the other hand, the compiler converts the program data into robotic arm control instructions, external device control instructions, and logic control instructions, so that the actuator, according to the logic control instructions, sends the robotic arm control instructions to the collaborative robotic arm and sends the external device control instructions to the external device, supporting the flexible design of taking the collaborative robotic arm as the core control unit to coordinate external devices to jointly complete the tasks corresponding to the instructions and cooperate with other systems, thereby improving the adaptability of the system to various application requirements of single-device control and multi-device linkage and being conducive to enhancing the collaborative ability and task execution efficiency of the system. On the other hand, through the feedback and closed-loop control mechanism among the human-computer interaction interface, the collaborative robotic arm, the external device, and the actuator, it is conducive to real-time monitoring of the task execution situation and ensuring the high efficiency and safety of the system.

[0065] In an exemplary embodiment, the graphical element includes a teaching point; the user terminal 110 determines the target point of the collaborative robotic arm 140 and generates target point data in response to the user's dragging operation of the teaching point in the human-computer interaction interface.

[0066] In practical applications, the user can click and drag the teaching point on the interface to the desired position and click save to complete the setting of moving the collaborative robotic arm to the target point. A series of key points can also be set through multiple drags to form a complete motion path. The user terminal determines the target point and its position in response to the dragging operation of the teaching point in the human-computer interaction interface and generates moving control program data. The target point can be one or more.

[0067] In other embodiments, when the user drags the teaching point, the human-computer interaction interface can display the posture and path of the collaborative robotic arm in real time to facilitate the user to anticipate the action effect of the setting.

[0068] In this embodiment, providing an intuitive setting of the target position of the collaborative robotic arm on the human-computer interaction interface simplifies the programming process of controlling the collaborative robotic arm and is conducive to the user's intuitive understanding and optimization of the action path of the robotic arm.

[0069] In an exemplary embodiment, the graphical element further includes a logic instruction; the user terminal 110 generates logic flow data for defining the execution of a target task by the collaborative robotic arm 140 in response to a user's drag operation on the logic instruction in the human-computer interaction interface.

[0070] Among them, the logic instructions include but are not limited to start, end, conditional judgment, loop structure, etc.

[0071] In practical applications, the human-computer interaction interface provides graphical logic instructions. The user can select the required logic instructions from the human-computer interaction interface and drag them to the work area, adjust the positions of the logic instructions by dragging in the work area, configure the attributes for the selected logic instructions, and construct a complete task logic flow. Specifically, click on the logic instruction to open its property panel and set the attributes. Exemplarily, such as parameters like target coordinates, speed, acceleration in a movement command; comparison conditions in a conditional branch, etc.

[0072] In this embodiment, by constructing a logic control structure through the drag-and-drop method of graphical logic instructions on the human-computer interaction interface, programming is abstracted into visual elements, reducing the programming threshold for controlling the collaborative robotic arm and enabling users without a deep programming background to create complex automation programs.

[0073] In an exemplary embodiment, the user terminal 110 generates teaching control instruction data in response to a user's parameter configuration of a predefined teaching template instruction.

[0074] Among them, the predefined teaching template instruction encapsulates the logic and parameter settings of a specific task or operation of the collaborative robotic arm. Such as pick-and-place and material handling, etc.

[0075] In practical applications, specific operation logics and parameter settings can be pre-encapsulated according to some common functions of the collaborative robotic arm to construct multiple teaching template instructions, enabling users to implement complex functions without having to deeply understand programming details, but only by selecting a suitable template and making a small amount of configuration. Exemplarily, a predefined teaching template instruction for pick-and-place encapsulates the complete process of picking up an object from one position and placing it at another position, including steps such as moving to the pick-up point, grasping the object, moving to the placement point, putting down the object, and returning to the starting position. The corresponding configuration items can include pick-up point coordinates, placement point coordinates, grasping force, waiting time between grasping and placing actions, etc. The user selects a suitable teaching template instruction, configures the parameters of the configuration items of the teaching template instruction, and the user terminal generates a corresponding instruction sequence in response to the user's parameter configuration of the teaching template instruction to obtain teaching control instruction data.

[0076] In this embodiment, through predefined teaching template instructions, users can quickly get started with complex tasks. By parameter configuration, the control flexibility and the adaptability to different application scenario requirements are taken into account. While simplifying the programming process of the collaborative robot arm, the predefined teaching template instructions are beneficial to ensure the accuracy and safety of task execution. In addition, each template can be adjusted according to specific requirements to meet special application requirements.

[0077] In an exemplary embodiment, the user terminal 110 obtains the code data written by the user in response to the user's code writing operation; the program data includes at least one of target point position data, task flow data, teaching control instruction data, and code data.

[0078] In practical applications, considering some application scenarios that require complex algorithm support or interaction with other external devices, some function libraries or modules can be pre-encapsulated, such as functions like TCP / IP communication, MODBUS protocol processing, and thread management. Users can insert Python code blocks through specific nodes or components in the human-machine interaction interface, write a small amount of Python code, and process complex calculations, data processing, or communicate with other devices through predefined interfaces.

[0079] In this embodiment, an innovative dual programming mode that combines graphical programming with third-party language extension is adopted, breaking through the functional limitations of graphical programming, improving the scalability of the system, and being beneficial to improving the adaptability and flexibility to complex application scenario requirements.

[0080] In an exemplary embodiment, the user terminal 110 is further configured to store the program data according to a preset data structure format to obtain a program data file, and send the program data file to the compiler 120.

[0081] In this embodiment, the preset data structure formats include formats such as JSON (JavaScript Object Notation), CVS (Comma-Separated Values), and XML (Extensible Markup Language).

[0082] In practical applications, after the user finishes programming through the human-machine interaction interface, the user terminal stores the generated program data in JSON format. The data in JSON format contains instructions for controlling the collaborative robot arm, the logical flow of the task, and relevant information for controlling external devices. Obtain the program data file storing the program data, and send the program data file to the compiler. The JSON format has good compatibility, is easy to parse, and the compact format is beneficial to reducing data transmission time.

[0083] In this embodiment, by storing program data in a preset data structure format, the program data can have good readability and structural characteristics, which is beneficial to improving the efficiency of compiling and executing the program data.

[0084] In an exemplary embodiment, the compiler 120 converts the program data into an instruction set in Python format according to preset compilation rules.

[0085] Among them, the preset compilation rules include the conversion logic between programming elements and the Python language.

[0086] In practical applications, it can be that the compiler parses the program data in JSON format through preset compilation rules, identifies logical structures such as branch judgments (such as if-else) and conditional loops (such as for, while) therein, and generates Python-format logical control instructions; identifies program data for controlling the collaborative robotic arm to perform basic movements (such as moving to a specified position) and compound movements (such as a series of continuous actions), and generates Python-format robotic arm control instructions; identifies program data for interacting with external devices, and generates Python-format external device control instructions, such as TCP communication, MODBUS protocol control, etc. instructions.

[0087] In this embodiment, by the compiler converting the program data into an instruction set in Python format according to preset compilation rules, it is beneficial to ensure a seamless transition from graphical programming to actual execution.

[0088] In an exemplary embodiment, after the collaborative robotic arm 140 executes the task corresponding to the collaborative robotic arm control instruction, it generates position information, attitude information, torque information, and speed information, integrates the position information, attitude information, torque information, and speed information into feedback information, and feeds back the execution status of the task and the feedback information to the actuator 130. The first task feedback information includes the execution status of the task and the feedback information.

[0089] Among them, the position information may include the position coordinates of the end effector of the collaborative robotic arm in three-dimensional space; the attitude information may include the angle information of the end effector of the collaborative robotic arm. The execution status of the task may include whether the task is completed.

[0090] In practical applications, after the collaborative robotic arm executes the task corresponding to the collaborative robotic arm control instruction, it obtains position information, attitude information, torque information, and speed information, integrates them into feedback information, and returns the execution status of the task and the feedback information to the actuator.

[0091] After the external device executes the task corresponding to the external device control instruction, it feeds back the execution status of the task and the task feedback information to the actuator 130. The second task feedback information includes the execution status of the task and the task feedback information.

[0092] In practical applications, after the external device (such as a sensor) executes the task corresponding to the external device control instruction, it returns the execution status of the task and the task feedback information (such as sensor data) to the actuator.

[0093] In this embodiment, after the collaborative robotic arm and the external device execute the task, they feed back task-related information to the actuator, which is convenient for monitoring the task execution situation.

[0094] In an exemplary embodiment, the actuator 130 is further configured to adjust the control logic of the collaborative robotic arm control instruction and return the adjusted instruction to the collaborative robotic arm 140 when the first task feedback information indicates that the task execution is abnormal.

[0095] When the second task feedback information indicates that the task execution is abnormal, adjust the control logic of the external device control instruction and return the adjusted instruction to the external device.

[0096] In practical applications, design a feedback and closed-loop control mechanism, such as Figure 2 As shown, after the actuator receives the task feedback information, it determines whether the predetermined action is completed according to the task execution status of the task feedback information. When the first task feedback information indicates that the task execution is abnormal (such as the predetermined action is not completed), it can adjust the control logic of the collaborative robotic arm control instruction according to the predefined rules and logical conditions, such as adjusting the speed or force parameters of the robotic arm, retrying the failed operation, etc., and return the adjusted instruction to the collaborative robotic arm. When the second task feedback information indicates that the task execution is abnormal, such as the external device interaction fails, it can adjust the control logic of the external device control instruction, such as re-interacting, etc., and return the adjusted instruction to the external device. The actuator returns the execution result to the user terminal after the task execution is completed.

[0097] In this embodiment, the actuator adjusts the control logic in real time through the task feedback information, which is beneficial to improving the efficiency and safety of the system.

[0098] Based on the same inventive concept, the collaborative robotic arm graphical programming compilation method provided by the embodiments of the present application can be applied to an application environment such as Figure 3 As shown. Among them, the user terminal 102 communicates with the collaborative robotic arm 104 and the external device 106 through the network respectively.

[0099] Specifically, it can be that the user performs code writing operations on the human-computer interaction interface through the user terminal 102, and / or drag-and-drop operations on graphical elements. The user terminal responds to the user's code writing operations on the human-computer interaction interface, and / or drag-and-drop operations on graphical elements, and generates program data. Secondly, based on preset compilation rules, the program data is converted into an instruction set in Python format. The instruction set includes robotic arm control instructions, external device control instructions, and logic control instructions. Then, the logic control instructions are executed, the collaborative robotic arm 104 is controlled according to the robotic arm control instructions, and the external device 106 is controlled according to the external device control instructions. Finally, in the case of receiving feedback information from the collaborative robotic arm and / or the external device, an execution result is generated according to the feedback information, and the execution result is pushed to the user terminal 102.

[0100] Among them, the user terminal 102 can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc.

[0101] In an exemplary embodiment, as Figure 4 shown, a method for graphical programming and compilation of a collaborative robotic arm is provided. Taking the method applied to the Figure 3 user terminal 102 therein as an example for illustration, it includes the following S100 to S400. Among them:

[0102] S100, in response to the user's code writing operations on the human-computer interaction interface, and / or drag-and-drop operations on graphical elements, generate program data.

[0103] Among them, the graphical elements can include teaching points and logic instructions.

[0104] In practical applications, the user can click and drag the teaching points on the interface to the desired positions, and then click to save to complete the setting of the collaborative robotic arm to move to the target points. In response to the user's dragging operation of the teaching points on the human-machine interaction interface, movement control program data is generated. The user can select the required logical instructions from the human-machine interaction interface and drag and drop them into the work area, adjust the positions of the logical instructions by dragging in the work area, configure attributes for the selected logical instructions, and construct a complete task logic process. In response to the user's dragging operation of the logical instructions on the human-machine interaction interface, task flow data is generated. The user can also insert Python code blocks through specific nodes or components in the human-machine interaction interface, write a small amount of Python code, and process complex calculations, data processing, or communicate with other devices through predefined interfaces. In response to the user's code writing operation on the human-machine interaction interface, code data is obtained. The program data includes at least one of target point data, task flow data, and code data.

[0105] In other embodiments, the program data further includes teaching control instruction data. Based on some common functions of the collaborative robotic arm, specific operation logics and parameter settings are encapsulated in advance to construct multiple teaching template instructions. The user selects a suitable teaching template instruction and configures parameters for the configuration items of the teaching template instruction. In response to the user's parameter configuration of the teaching template instruction, a corresponding instruction sequence is quickly generated to obtain the teaching control instruction data.

[0106] S200, based on preset compilation rules, convert the program data into a Python-format instruction set. The instruction set includes robotic arm control instructions, external device control instructions, and logical control instructions.

[0107] Among them, the preset compilation rules include the conversion logic between programming elements and the Python language.

[0108] In practical applications, it can be to parse the JSON-format program data through the preset compilation rules, identify logical structures such as branch judgments (such as if-else) and conditional loops (such as for, while) therein, and generate Python-format logical control instructions; identify the program data for controlling the collaborative robotic arm to perform basic movements (such as moving to a specified position) and composite movements (such as a series of continuous actions), and generate Python-format robotic arm control instructions; identify the program data for interacting with external devices, and generate Python-format external device control instructions, such as TCP communication, MODBUS protocol control, etc.

[0109] S300, execute the logical control instructions, control the collaborative robotic arm according to the robotic arm control instructions, and control the external devices according to the external device control instructions.

[0110] In practical applications, logical control is completed according to the logical conditions of logical control instructions. The control instructions of the collaborative robotic arm are sent to the robotic arm in real time through the communication interface with the collaborative robotic arm (such as TCP / IP) to complete the operation of the target point and the execution of the task process. The operations and coordination of external devices such as sensors and actuators are realized through the communication protocols with external devices (such as TCP, MODBUS, etc.).

[0111] S400, in the case of receiving the task feedback information of the collaborative robotic arm and / or external device, generates an execution result according to the task feedback information and pushes the execution result to the user terminal.

[0112] Among them, the task feedback information may include feedback information related to the execution status of the task.

[0113] In practical applications, according to the task feedback information, it is evaluated whether the task execution status is normal. If the feedback information indicates that the task execution is abnormal, the control logic of the collaborative robotic arm control instruction and / or external device control instruction is adjusted according to the preset rules, and the corresponding collaborative robotic arm and / or external device are controlled according to the adjusted control instructions. After the task is completed, the log messages and final results of the events during the task are summarized to generate an execution result, and the execution result is pushed to the user terminal. The pushing method may include pushing the execution result in the notification bar of the user terminal, displaying the execution result in the form of a full-screen or semi-screen pop-up window, giving a prompt through a specific sound or vibration mode, and giving a visual prompt through the flashing of an indicator light, etc. It can be understood that the method of pushing the execution result may be any one of the foregoing methods or a combination of any multiple methods, which is not limited herein.

[0114] In the above-mentioned graphical programming compilation method for collaborative robotic arms, for the control of collaborative robotic arms, a dual programming mode that innovatively combines graphical programming with the extension of third-party programming languages is adopted. In response to the user's code writing operation and / or dragging operation of graphical elements on the human-machine interface, program data is generated. Based on the preset compilation rules, the program data is converted into an instruction set in the target programming language format to control the collaborative robotic arm and its external devices, reducing the programming difficulty of the collaborative robotic arm, improving the simplicity of user robotic arm control and task arrangement, comprehensively considering users with different usage requirements, enabling function customization according to actual needs, and improving the adaptability to complex scenario logical control.

[0115] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are sequentially shown as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless specifically stated herein, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turns with at least a part of other steps or steps or stages in other steps.

[0116] In an exemplary embodiment, as Figure 5 shown, a collaborative robotic arm graphical programming compilation device 600 is provided, including: a program data generation module 610, a compilation module 620, a control module 630, and a feedback module 640, where:

[0117] The program data generation module 610 is configured to generate program data in response to a user's code writing operation on the human-machine interface and / or a dragging operation of graphical elements.

[0118] The compilation module 620 is configured to convert the program data into an instruction set in Python format based on preset compilation rules. The instruction set includes robotic arm control instructions, external device control instructions, and logic control instructions.

[0119] The control module 630 is configured to execute logic control instructions, control the collaborative robotic arm according to the robotic arm control instructions, and control external devices according to the external device control instructions.

[0120] The feedback module 640 is configured to generate an execution result according to the task feedback information and push the execution result to the user terminal when receiving the task feedback information of the collaborative robotic arm and / or external devices.

[0121] Each module in the above-mentioned collaborative robotic arm graphical programming compilation device 600 can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in the processor of the computer device in hardware form or be independent of it, or be stored in the memory of the computer device in software form, so as to facilitate the processor to call and execute the operations corresponding to the above-mentioned modules.

[0122] In an exemplary embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 6As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a collaborative robotic arm graphical programming compilation method.

[0123] Those skilled in the art can understand that Figure 6 the structure shown in [the figure] is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different component layout.

[0124] In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the steps in any one of the above-mentioned embodiments of the collaborative robotic arm graphical programming compilation method are implemented.

[0125] In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, the steps in any one of the above-mentioned embodiments of the collaborative robotic arm graphical programming compilation method are implemented.

[0126] In an embodiment, a computer program product is provided, including a computer program. When the computer program is executed by the processor, the steps in any one of the above-mentioned embodiments of the collaborative robotic arm graphical programming compilation method are implemented.

[0127] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0128] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, artificial intelligence (AI) processors, etc., without limitation.

[0129] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in the present application.

[0130] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A collaborative robot arm graphical programming compilation system, characterized in that: The system includes a user terminal, a compiler, an actuator and a collaborative robot arm: The user terminal generates program data in response to a user's code writing operation on a human-computer interaction interface and / or a dragging operation on a graphic element, and sends the program data to the compiler; The compiler converts the program data into an instruction set, and sends the instruction set to the actuator, wherein the instruction set includes collaborative robot arm control instructions, external device control instructions, and logic control instructions; The actuator executes the logic control instruction, sends the collaborative robot arm control instruction to the collaborative robot arm, and sends the external device control instruction to the external device; The collaborative robot arm executes a task matching the collaborative robot arm control instruction, generates first task feedback information, and sends the first task feedback information to the executor; The external device executes a task matching the external device control instruction, generates second task feedback information, and sends the second task feedback information to the executor; The executor generates an execution result according to the first task feedback information and the second task feedback information, and pushes the execution result to the user terminal.

2. The system according to claim 1, characterized in that The graphic elements include teaching points; The user terminal determines the target point of the collaborative robot arm and generates target point data in response to the user's dragging operation on the teaching point in the human-computer interaction interface.

3. The system according to claim 2, characterized in that The graphic element also includes a logic instruction; the user terminal generates logic process data for defining the collaborative robot arm to perform a target task in response to the user's dragging operation on the logic instruction in the human-computer interaction interface.

4. The system according to claim 3, characterized in that The user terminal generates teaching control instruction data in response to the user's parameter configuration of the predefined teaching template instruction.

5. The system according to claim 4, characterized in that The user terminal obtains the code data written by the user in response to the code writing operation of the user; The program data includes at least one of the target point data, task flow data, teaching control instruction data and the code data.

6. The system according to any one of claims 1 to 5, characterized in that: The user terminal is also used to store the program data according to a preset data structure format, obtain a program data file, and send the program data file to the compiler.

7. The system according to claim 6, characterized in that The compiler converts the program data into an instruction set in Python format according to preset compilation rules.

8. The system according to claim 7, characterized in that After executing the task corresponding to the collaborative robot arm control instruction, the collaborative robot arm generates position information, posture information, torque information and speed information, integrates the position information, posture information, torque information and speed information into feedback information, and feeds back the execution status of the task and the feedback information to the executor, wherein the first task feedback information includes the execution status of the task and the feedback information; After executing the task corresponding to the external device control instruction, the external device feeds back the execution status of the task and task feedback information to the executor, and the second task feedback information includes the execution status of the task and the task feedback information.

9. The system according to claim 8, characterized in that The executor is further used to adjust the control logic of the collaborative robot arm control instruction when the first task feedback information indicates that the task execution is abnormal, and return the adjusted instruction to the collaborative robot arm; When the second task feedback information indicates that the task execution is abnormal, the control logic of the external device control instruction is adjusted, and the adjusted instruction is returned to the external device.

10. A collaborative robot arm graphical programming compilation method, characterized in that: The method comprises: generating program data in response to a user's code writing operation on a human-computer interaction interface and / or a dragging operation on a graphic element; Based on preset compilation rules, the program data is converted into an instruction set in Python format, wherein the instruction set includes a robot arm control instruction, an external device control instruction, and a logic control instruction; Execute the logic control instruction, control the collaborative robot arm according to the robot arm control instruction, and control the external device according to the external device control instruction; When task feedback information from the collaborative robot arm and / or the external device is received, an execution result is generated according to the task feedback information, and the execution result is pushed to the user terminal.

Images