Task execution method and device

Abstract

The embodiment of the invention provides a task execution method and device, the task execution method and device are applied to equipment deployed with a visual control integrated system, the visual control integrated system is used for controlling a mechanical arm and visual equipment, and the mechanical arm and the visual equipment are deployed in the same working environment. Determining a task execution process indicated by the task starting instruction; and according to the connection relationship among the process nodes recorded in the task execution process, sequentially executing the control logic packaged by each process node, and controlling the mechanical arm and the visual equipment to execute the task, each process node comprises an action node packaged with action control logic of the mechanical arm and an action node packaged with action control logic of the visual equipment, and the connection relationship represents the execution sequence of the process nodes. According to the scheme, the task execution efficiency of the mechanical arm can be improved.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a task execution method and device. Background Art

[0002] With the continuous development of industrial automation and intelligent manufacturing technology, robotic arm control technology has been widely used. Its core lies in controlling the robotic arm to perform various operations on the task target, thereby completing the set task.

[0003] To cope with more diverse work scenarios and more complex tasks, robotic arms are generally required to work in conjunction with vision equipment to complete tasks. The vision equipment is responsible for providing visual functions for the robotic arm, enabling the robotic arm to perceive the environment in real time and successfully complete the task based on the perception results.

[0004] In related technologies, when controlling a robotic arm to perform a task, the robotic arm controller independently controls the robotic arm, while the visual control computer independently controls the visual equipment. This requires not only communication between the robotic arm controller and the robotic arm to control the robotic arm, and between the visual control computer and the camera to control the visual equipment, but also communication between the robotic arm controller and the visual control computer to achieve coordinated coordination between the robotic arm and the visual equipment. This results in a cumbersome task execution process and low task execution efficiency. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a task execution method and device to improve the task execution efficiency of a robotic arm. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present application provides a task execution method, which is applied to a device deployed with an integrated vision and control system, wherein the integrated vision and control system is used to control a robotic arm and a visual device, wherein the robotic arm and the visual device are deployed in the same working environment, and the method includes:

[0007] In response to a task start instruction, determining a task execution process indicated by the task start instruction;

[0008] According to the connection relationship between the process nodes recorded in the task execution process, the control logic encapsulated in each process node is executed in sequence to control the robotic arm and the visual device to perform the task, wherein the each process node includes: an action node encapsulating the motion control logic of the robotic arm and an action node encapsulating the motion control logic of the visual device, and the connection relationship represents: the execution order of the process nodes.

[0009] In a second aspect, an embodiment of the present application provides a task execution device, which is applied to a device deployed with an integrated vision and control system, wherein the integrated vision and control system is used to control a robotic arm and a visual device, wherein the robotic arm and the visual device are deployed in the same working environment, and the device includes:

[0010] a process determination module, configured to determine, in response to a task start instruction, a task execution process indicated by the task start instruction;

[0011] The task execution module is used to execute the control logic encapsulated in each process node in sequence according to the connection relationship between the process nodes recorded in the task execution process, and control the robotic arm and visual device to perform tasks, wherein the various process nodes include: an action node encapsulating the motion control logic of the robotic arm and an action node encapsulating the motion control logic of the visual device, and the connection relationship represents: the execution order of the process nodes.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0013] Memory for storing computer programs;

[0014] The processor is configured to implement the method described in the first aspect when executing the program stored in the memory.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0016] In a fifth aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect.

[0017] As can be seen from the above, in the solution provided by the embodiment of the present application, the process nodes in the task execution process include action nodes encapsulating the motion control logic of the robot arm and action nodes encapsulating the motion control logic of the visual device. After receiving the task start instruction, the electronic device can determine the task execution process indicated by the task start instruction, and execute the control logic encapsulated in each process node in sequence according to the connection relationship between the process nodes recorded in the task execution process, and control the robot arm and the visual device to perform the task. It can be seen that the electronic device deployed with the integrated visual control system can control the robot arm and the visual device in an orderly manner according to the task nodes and connection relationships recorded in the task execution process, thereby enabling the robot arm and the visual device to cooperate accurately to complete the task. Compared with the robot arm controller controlling the robot arm alone and the visual industrial computer controlling the visual device alone, the electronic device deployed with the integrated visual control system can realize the joint control of the robot arm and the visual device, eliminating the communication process between the robot arm controller and the visual industrial computer to achieve the coordinated cooperation of the robot arm and the visual device, simplifying the task execution process of the robot arm, and improving the efficiency of task execution.

[0018] In addition, in the related technology, when the robotic arm controller communicates with the visual industrial computer, the communication delay is high due to the influence of network environment fluctuations, etc.; while the electronic device in the solution provided in the embodiment of the present application can realize coordinated control of the robotic arm and the visual device, eliminating the communication process between the robotic arm controller and the visual industrial computer, thereby reducing the communication delay and improving the real-time performance when controlling the robotic arm and the visual device.

[0019] Of course, it is not necessary to achieve all the advantages described above at the same time when implementing any product or method of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0021] Figure 1 A schematic diagram of a robotic arm performing a task according to an embodiment of the present application;

[0022] Figure 2 A flowchart of a task execution method provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of the first process node and node relationship provided in an embodiment of the present application;

[0024] Figure 4 A flowchart of the first task execution process construction method provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of the second process node and node relationship provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of a composite action node provided in an embodiment of the present application;

[0027] Figure 7 A flowchart of the second task execution process construction method provided in an embodiment of the present application;

[0028] Figure 8 A schematic diagram of the structure of a task execution device provided in an embodiment of the present application;

[0029] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of this application.

[0031] First, the application scenarios of the solutions provided in the embodiments of the present application are introduced.

[0032] The application scenario of the solution provided in the embodiment of the present application is: a scenario in which a robotic arm locates a task target in a working environment with the help of the visual function provided by a visual device, and then performs a set action on the task target.

[0033] Among them, the above-mentioned visual equipment may include image acquisition equipment, light source equipment, etc. The visual equipment can be installed on the robotic arm or installed at a set position in the working environment. It is only necessary to ensure that the task target is within the field of view of the visual equipment; the above-mentioned task target can be workpieces, goods, packages, etc. in the production line; the above-mentioned set actions can be grasping, adsorption, assembly, measurement and other actions for the task target, which can be flexibly set according to actual task requirements. The above-mentioned set actions can be specifically implemented by the end effector in the robotic arm (such as grippers, suction cups, cutting tools, welding tools, etc.).

[0034] The following combination Figure 1 This section provides a more intuitive introduction to the above scenario.

[0035] like Figure 1As shown, an image acquisition device 102 is installed on the robot arm 101, and the image acquisition device 102 acquires images of the workpiece 103 in the production line. The acquired images are used to locate the workpiece 103; the robot arm controls the gripper 1011 to grab the workpiece 103 according to the positioning result, and then places the workpiece 103 to the set position in the working environment.

[0036] It should be noted that Figure 1 The illustrated scenarios are merely examples for ease of understanding and do not constitute limitations on the embodiments of this application. For example, the image acquisition device 102 can be mounted via a fixed bracket in the space above or below the robotic arm. Furthermore, after the gripper 1011 grasps the workpiece 103, it can perform a predetermined assembly operation on the workpiece 103 and then return the assembled workpiece to the production line. All of these are reasonable options.

[0037] Next, the execution entity of the solution provided in the embodiment of this application is introduced.

[0038] The implementation subject of the solution provided in the embodiments of the present application is any electronic device with data processing, storage, communication and other functions, which is equipped with an integrated visual control system and can be a background control device or a robotic arm itself. Specifically, when the electronic device is a background control device, the background control device pre-establishes a communication connection with both the robotic arm and the visual device; when the electronic device is a robotic arm, the robotic arm only needs to establish a communication connection with the visual device.

[0039] The above-mentioned integrated vision and control system is used to control the robotic arm and visual equipment, which are introduced below respectively.

[0040] 1. Control of the robotic arm:

[0041] The robotic arm connected to the integrated vision and control system sends control instructions to control the robotic arm to perform various actions indicated by the control instructions.

[0042] Specifically, the integrated vision and control system defines the first instructions for controlling the robotic arm to perform various actions.

[0043] From the perspective of instruction categories, the first instructions may include motion control instructions, tool control instructions, input / output (IO) control instructions, etc. IO control instructions are used to input digital or analog signals to the robotic arm, or to control the robotic arm to output digital or analog signals.

[0044] Motion control instructions may include: joint motion instructions, straight line instructions, circular motion instructions, curved motion instructions, etc.; tool control instructions may include gripper control instructions, suction cup control instructions, welding gun control instructions, etc.; IO control instructions include digital input (DI) instructions, digital output (DO) instructions, analog input (AI), analog output (AO) instructions, etc.

[0045] It should be noted that the above instruction types are only examples for ease of understanding. Those skilled in the art can set various specific instructions according to actual needs, and the embodiments of the present application are not limited to this.

[0046] In this way, the integrated vision and control system can control the connected robotic arm to perform various actions by sending various first instructions to the robotic arm, and can control the robotic arm to complete various complex actions through a combination of first instructions.

[0047] The form of the first instruction varies depending on the data interaction method adopted between the integrated vision and control system and the robotic arm.

[0048] In one case, if the integrated visual control system and the robotic arm use Data Distribution Service (DDS) technology for data interaction, such as FastDDS technology, OpenDDS technology, etc., then the first instruction can be in the form of a DDS instruction. When the integrated visual control system and the robotic arm interact with data based on DDS technology, the topic is the core identifier of data classification and routing, which is used to uniquely identify a type of data type to ensure accurate matching and transmission of data between the publishing end and the subscribing end. Specifically, the robotic arm, as the subscriber, can subscribe to the DDS message of the set topic. The integrated visual control system, as the publisher, uses the DDS instruction processing function to encapsulate the DDS instruction into a DDS message with a set topic and send it to the network. Through network forwarding, the robotic arm recognizes that the DDS message belongs to the topic it has subscribed to, and thus receives the above DDS message, that is, receives the DDS instruction.

[0049] In this case, the integrated vision and control system can leverage behavior tree technology, such as the BehaviorTree.CPP framework, to implement the system's DDS command processing functions as independently executable behavior tree action nodes. This allows these DDS command processing functions to execute their corresponding functions within the behavior tree framework. In other words, each DDS command processing function is encapsulated into an independent, executable unit that can be combined and executed according to the structure and rules of the behavior tree.

[0050] 2. Control of visual equipment:

[0051] The integrated vision and control system can send control instructions to the connected visual devices to control the visual devices to perform various actions indicated by the control instructions.

[0052] The above-mentioned visual equipment may include light source equipment and image acquisition equipment. The image acquisition equipment may include visible light cameras, infrared cameras, laser cameras, radars and other equipment that can obtain the target shape or appearance characteristics. The image acquisition equipment can also be called sampling equipment.

[0053] Among them, the integrated visual control system can be connected to multiple visual devices, and the multiple visual devices connected can be visual devices of the same type or visual devices of different types.

[0054] Specifically, the integrated vision and control system defines second instructions for controlling visual devices to perform various actions. From the perspective of instruction categories, the second instructions can include environmental control instructions, visual sampling instructions, visual analysis instructions, etc.

[0055] Among them, environmental control instructions may include: light source switch instructions, light source brightness adjustment instructions, focus distance adjustment instructions, etc.; visual sampling instructions may include: sampling device connection instructions, sampling parameter configuration instructions, sampling action execution instructions, sampling data acquisition instructions, etc.; visual analysis instructions may include image enhancement instructions, filtering instructions and other image processing instructions, and may also include circle finding instructions, template matching instructions and other target recognition instructions.

[0056] It should be noted that the above instruction types are only examples for ease of understanding. Those skilled in the art can set various specific instructions according to actual needs, and the embodiments of the present application are not limited to this.

[0057] In this way, the integrated vision and control system can control the visual devices to perform various actions by sending various second instructions to the connected visual devices, and can control the visual devices to complete various complex actions through the combination of second instructions.

[0058] The form of the first instruction varies depending on the interaction method adopted between the integrated vision and control system and the visual device.

[0059] In one scenario, if the integrated visual control system and the visual device use DDS technology for data exchange, the second instruction can be in the form of a DDS instruction. Similarly, in this case, the DDS instruction processing functions of the integrated visual control system can be implemented as independently executable behavior tree action nodes based on behavior tree technology, allowing these DDS instruction processing functions to perform their corresponding functions within the behavior tree framework.

[0060] The following is a detailed description of the task execution scheme provided in the embodiment of the present application with reference to the accompanying drawings.

[0061] See also Figure 2 , is a flow chart of a task execution method provided in an embodiment of the present application, the method includes the following steps S201-S202.

[0062] Step S201: In response to a task start instruction, determining a task execution process indicated by the task start instruction.

[0063] When receiving a task start instruction, the electronic device can determine that there is a need to execute the task, and thus can determine the task execution process indicated by the task start instruction from the constructed task execution process. The task execution process indicated by the task start instruction is also the task execution process to be executed.

[0064] Among them, the electronic device can respond to the user's execution process editing operation in the interactive interface, determine the process nodes included in the task execution process and the connection relationship between the process nodes based on the operation information of the execution process editing operation, and then construct the task execution process based on the determined process nodes and connection relationships. The specific construction method may include configuring node components in the interactive interface and entering programming statements in the interactive interface. Please refer to the detailed introduction in the subsequent embodiments, which will not be described in detail here. As can be seen from the above, the solution provided by the embodiment of the present application supports the construction of the execution process of the task through the interactive interface, which is more user-friendly, reduces the difficulty of constructing the execution process, and improves the efficiency of constructing the execution process.

[0065] The following describes a method for determining the task execution process indicated by the task start instruction.

[0066] In one case, if the task start instruction carries a process identifier of a task execution process, the electronic device can determine the task execution process corresponding to the process identifier from the constructed task execution processes as the task execution process to be executed.

[0067] In another case, if the task start instruction does not carry the process identifier of the task execution process, the electronic device may use the established default task execution process as the task execution process to be executed.

[0068] The task start instruction may be triggered by a user clicking on an interactive interface, which may be provided by the electronic device or by a client device that has established a communication connection with the electronic device. The client device may be a teaching pendant equipped with a client.

[0069] The interactive interface may display the process identifiers of each established task execution process and a task execution button. The process identifier may be the name of the task execution process, etc. In this case, the user can select the process identifier of the task execution process to be executed according to actual needs, and then click the task execution button, so that the electronic device can receive the task start instruction carrying the process identifier.

[0070] Of course, the above task execution instruction can also be directly input by the user into the electronic device.

[0071] In one embodiment of the present application, after the user selects the process identifier of the task execution process to be executed, a graphical task execution process can be displayed in the interactive interface, including each process node in the task execution process and the connection relationship between the process nodes, so that the user can quickly and intuitively understand the specific task execution logic described in the task execution process, and then determine whether the task execution process meets the actual task requirements.

[0072] Step S202: according to the connection relationship between the process nodes recorded in the task execution process, the control logic encapsulated in each process node is executed in sequence to control the robotic arm and the visual device to perform the task.

[0073] The above process nodes correspond to the behavior tree action nodes mentioned above that encapsulate the control logic, including the action nodes that encapsulate the motion control logic of the robotic arm and the action nodes that encapsulate the motion control logic of the visual device.

[0074] When a process node is executed by the electronic device, it sends control instructions to the robotic arm or vision device according to the control logic encapsulated in the process node, thereby controlling the robotic arm or vision device to perform the action. In this way, the electronic device sequentially executes each process node, controlling the robotic arm and vision device to coordinate and perform various actions, thereby completing the task.

[0075] The above connection relationship represents: the execution order of process nodes. Specifically, the connection relationship includes a relationship start point and a relationship end point, and the execution order of the process node corresponding to the relationship start point is before the execution order of the process node corresponding to the relationship end point.

[0076] Below through Figure 3 Provide an intuitive introduction to the above process nodes and connection relationships.

[0077] See first Figure 3, robot arm action A and robot arm action B are action nodes that represent the action performed by the robot arm, light source action A, camera action A, and visual action A are action nodes that represent the action performed by the visual device. The above action nodes are process nodes; the direction of the arrows between the process nodes represents the connection relationship between the process nodes. Specifically, the execution order of the process node connected to the starting point of the arrow is before the execution order of the process node connected to the end point of the arrow.

[0078] Among them, the user can configure the process nodes included in the execution process and the connection relationship between the process nodes by configuring node components in the interactive interface or entering programming statements in the interactive interface. Please refer to the introduction in the subsequent embodiments.

[0079] In one embodiment of the present application, the above process nodes may include not only action nodes but also process control nodes.

[0080] The above-mentioned action node encapsulates specific control logic for the robotic arm or visual device, which is used to control the robotic arm or visual device to perform actions when executed; the above-mentioned process control node encapsulates execution sequence control logic, which is used to control the execution sequence of the action nodes. For details, please refer to the subsequent introduction.

[0081] As can be seen from the above, in the solution provided by the embodiment of the present application, the process nodes in the task execution process include action nodes encapsulating the motion control logic of the robot arm and action nodes encapsulating the motion control logic of the visual device. After receiving the task start instruction, the electronic device can determine the task execution process indicated by the task start instruction, and execute the control logic encapsulated in each process node in sequence according to the connection relationship between the process nodes recorded in the task execution process, and control the robot arm and the visual device to perform the task. It can be seen that the electronic device deployed with the integrated visual control system can control the robot arm and the visual device in an orderly manner according to the task nodes and connection relationships recorded in the task execution process, thereby enabling the robot arm and the visual device to cooperate accurately to complete the task. Compared with the robot arm controller controlling the robot arm alone and the visual industrial computer controlling the visual device alone, the electronic device deployed with the integrated visual control system can realize the joint control of the robot arm and the visual device, eliminating the communication process between the robot arm controller and the visual industrial computer to achieve the coordinated cooperation of the robot arm and the visual device, simplifying the task execution process of the robot arm, and improving the efficiency of task execution.

[0082] In addition, in the related technology, when the robotic arm controller communicates with the visual industrial computer, the communication delay is high due to the influence of network environment fluctuations, etc.; while the electronic device in the solution provided in the embodiment of the present application can realize coordinated control of the robotic arm and the visual device, eliminating the communication process between the robotic arm controller and the visual industrial computer, thereby reducing the communication delay and improving the real-time performance when controlling the robotic arm and the visual device.

[0083] In one embodiment of the present application, some process nodes may be encapsulated with exception handling logic. In this case, the electronic device responds to an exception during the execution of the first control logic encapsulated by the first process node in the task execution process. If the first process node encapsulates exception handling logic, the execution of the first control logic is suspended and the first exception handling logic encapsulated by the first process node is executed; after the first exception handling logic is executed, the first control logic is re-executed.

[0084] That is, when the electronic device determines that an exception occurs in the control logic of a process node during execution, it can interrupt the task in a timely manner and run the exception handling logic. After the exception handling logic is completed, the control logic of the process node is re-executed for retry, thereby improving the robustness of the task execution process.

[0085] Specifically, in one embodiment of the present application, for a second process node encapsulating a second control logic for controlling a visual device to perform image acquisition, the second process node encapsulates the following exception handling logic:

[0086] An abnormal image in the target image is determined, and based on image quality information of the abnormal image, a position adjustment parameter of the visual device is determined, and then the movement of the visual device is controlled based on the position adjustment parameter. The target image is the image captured by the visual device controlled by the second control logic.

[0087] The abnormal images may include images with a clarity lower than a set threshold, or images in which the mission target is not detected, or images in which the mission target is detected but is incomplete, etc. Correspondingly, the image quality information may include image clarity, mission target completeness, etc.

[0088] Position adjustment parameters are used to adjust the position of the visual device so that the device can capture better images. The following example describes how to determine the position adjustment parameters of the visual device.

[0089] In one case, the position adjustment parameters can be determined based on parameters such as image clarity, depth of field, and the expected distance between the visual device and the task target.

[0090] The above-mentioned expected distance is a distance that enables the image acquisition device to capture a clear image.

[0091] In automated industrial manufacturing scenarios, industrial cameras are often used in conjunction with robotic arms to perform tasks. Industrial cameras generally lack autofocus capabilities. Therefore, if the distance between the target and the camera is too large or too small, the camera struggles to capture high-definition images. Therefore, focusing can be adjusted by adjusting the distance between the camera and the target, enabling the camera to capture clear images.

[0092] Specifically, since the actual distance between the industrial camera and the task target is closely related to the depth of field and clarity of the captured image, the actual distance corresponding to the above clarity and depth of field can be determined according to the set correspondence, and then the difference between the actual distance and the expected distance is calculated as the above position adjustment parameter.

[0093] In another case, the position adjustment parameters can be determined according to the missing portion of the task target in the image.

[0094] For example, if the missing portion of the task target in the image is on the right side, the position adjustment parameters for controlling the visual device to move to the right can be determined. Of course, the specific distance to the right can also be determined based on the proportion of the missing portion, which is also reasonable.

[0095] It should be noted that if the visual device is deployed on a robotic arm, the above-mentioned position adjustment parameters can be sent to the robotic arm so that the robotic arm drives the visual device to move; if the visual device is deployed on other movable mechanisms (such as a pan-tilt head), the above-mentioned position adjustment parameters can be sent to the movable mechanism so that the movable mechanism drives the visual device to move.

[0096] In this embodiment, based on the pre-packaged exception handling logic, when an image that meets the requirements is not captured after the second process node is executed, the position of the visual device can be adaptively adjusted so that the visual device after the adjustment can capture a better image, thereby improving the accuracy of visual recognition and thus improving the accuracy of task execution.

[0097] In one embodiment of the present application, the electronic device may, in response to receiving input / output (IO) interface configuration information, control the robotic arm to establish a one-way IO interface subscription relationship with the electronic device, so that the robotic arm synchronizes the IO interface configuration with the electronic device. The IO interface configuration is used to determine the action nodes that can be used to control the robotic arm.

[0098] The embodiment of the present application does not limit the method of establishing the above-mentioned unidirectional IO interface subscription relationship, which is introduced below by way of example.

[0099] For example, the steps of establishing the aforementioned one-way IO interface subscription relationship may include the robotic arm publishing IO configurations and the electronic device subscribing to IO configurations. Specifically, the electronic device, as the subscriber, can subscribe to messages on a set topic. The robotic arm, as the publisher, sends the IO interface information it supports binding to the network. Through network forwarding, the electronic device recognizes that the message belongs to the topic it has subscribed to, and thus receives the message, thus implementing IO interface subscription.

[0100] In one embodiment of the present application, the electronic device can, in response to receiving IO signal configuration information, control the robotic arm to establish a bidirectional IO signal subscription relationship with the electronic device, so that the IO signals between the robotic arm and the electronic device are synchronized. The above IO signals are used to determine the action execution status of the robotic arm during the execution of the task execution process.

[0101] The establishment method of the bidirectional IO signal subscription relationship is similar to the above-mentioned unidirectional IO interface subscription relationship. The only difference is that the bidirectional IO signal subscription relationship needs to be established between the robotic arm and the electronic device, which will not be repeated here.

[0102] In this way, the electronic device can promptly know the IO interfaces supported and bound by the robotic arm, and thus know the action nodes that can be used to control the robotic arm, so that a usable task execution process can be constructed subsequently; and the robotic arm and the electronic device can synchronize IO signals in a timely manner, so that the electronic device can promptly know the execution status of various IO operations performed by the robotic arm, that is, the task execution status.

[0103] The following describes how to construct the aforementioned task execution process.

[0104] Specifically, in one embodiment of the present application, a user can conveniently and quickly construct a task execution process by selecting node components and editing the relationships between process nodes on a visual interactive interface.

[0105] See also Figure 4 , which is a flow chart of the first task execution process construction method provided in an embodiment of the present application, the above method includes the following steps S401-S403.

[0106] Step S401: In response to a user's selection operation on a node component displayed in an interactive interface, a node corresponding to the selected node component is determined as a process node included in a task execution process.

[0107] The interactive interface displays node components corresponding to multiple types of nodes and their associated information. These node components are primarily categorized as action nodes and process control nodes. Action nodes include those that control the robot arm to perform physical operations and those that drive the visual device to complete inspection tasks, while process control nodes configure the logical execution sequence between action nodes. Each visual component in the interactive interface is linked to its corresponding functional node in real time, forming a complete process management system.

[0108] In this way, users can select node components from the displayed node components by dragging, clicking, etc. according to actual task requirements, thereby flexibly determining the process nodes required to be used in the execution process.

[0109] The aforementioned process control nodes are used to control the execution order of action nodes, and may specifically include sequence nodes, parallel nodes, conditional nodes, fallback nodes, branch nodes, and loop nodes. Specifically, when the process control subsystem executes a process control node, it determines the execution order of the action nodes based on the control logic encapsulated within the process control node (e.g., sequential execution, conditional branching, loop control, etc.). Different process control nodes affect the execution order of action nodes in different ways, as illustrated below with examples.

[0110] For example, a conditional node is used to determine the next node to be executed from multiple adjacent nodes in the backward direction based on the output of the forward node. Figure 5 , Robotic arm action A-Robot arm action C are action nodes that represent the action performed by the robot arm, light source action A, camera action A and visual action A are action nodes that represent the action performed by the visual device, and the black filled rectangle represents the conditional node in the process control node. Execution Figure 5 When executing the task shown in the process, first, according to the node relationship between the process nodes, the robot arm action A, light source action A, camera action A and visual device A are executed in sequence; then, the condition node is executed, and according to the control logic encapsulated by the condition node, it is determined whether the target is recognized after executing visual action A. If the target is recognized, the robot arm action B can be executed. The robot arm action B can be performing a setting operation for the template, etc. If the target is not recognized, the robot arm action C can be executed. The robot arm action C can be adjusting the robot arm posture, etc.

[0111] For example, a loop node is used to indicate the execution of a loop through several nodes. A loop node consists of a loop start node and a loop end node. Whenever the loop end node is reached, execution returns to the node immediately adjacent to the corresponding loop start node, thus achieving loop execution of the nodes between the loop start node and the loop end node.

[0112] In one embodiment of the present application, an action node may include a composite action node that integrates multiple process nodes and relationships between process nodes. The process nodes in the composite action node may include action nodes and process control nodes. A composite action node represents a pre-built execution logic. After the user selects the node component corresponding to the composite action node, the composite action node can be added to the task execution process. When the electronic device executes the composite action node, it executes the control logic encapsulated in each process node in sequence according to the connection relationship between the process nodes integrated in the composite action node. It can be seen that the composite action node integrates multiple process nodes and relationships between process nodes. By selecting the composite action node, the user can reduce the selection of a single action node and the relationship editing operations, thereby improving the efficiency of process construction.

[0113] Among them, the electronic device can store pre-defined composite nodes that encapsulate commonly used functions, or it can encapsulate the various process nodes selected by the user and the connection relationships between the constructed process nodes into composite nodes according to the user's actual needs for subsequent use by the user.

[0114] The process of generating a composite node by an electronic device is similar to the task execution process, that is, in response to the user's selection operation on the node component displayed in the interactive interface, the node corresponding to the selected node component is determined as the process node included in the composite action node; then, in response to the user's relationship editing operation on the process node corresponding to the selected node component in the interactive interface, based on the operation information of the relationship editing operation, the connection relationship between the process nodes targeted by the relationship editing operation is determined; finally, based on the process node corresponding to the selected node component and the connection relationship between the process nodes, the composite action node is generated.

[0115] In this way, the electronic device can encapsulate each process node and the connection relationship between the constructed process nodes into a composite node, which is convenient for the user to select and use later, thereby improving the efficiency of process construction.

[0116] In one case, the action node for controlling the robotic arm, the action node for controlling the visual device (camera, light source), and the relationship between the nodes can be integrated into a composite action node.

[0117] For example, a composite action node can include control of the light source, control of the camera, control of the visual inspection process, and control of the robotic arm execution process, and has control over the handling of abnormal action execution. Specifically, the integrated functional action node can realize the control of the light source, including whether the light source is started and the working mode of the light source (constant, always off, or lit when taking pictures); it can realize the control of the camera, including whether to use camera sampling, how long to wait for the light source to illuminate when the camera samples, and whether the camera needs to be in a specific position when sampling; it can realize the control of the visual inspection process, including whether to use visual recognition of the task target, whether to report an error or continue execution when the vision does not recognize the task target, etc.; it can realize the control of the robotic arm execution process, including whether the robotic arm moves the end effector to the target position recognized by the vision when the vision recognizes the task target, etc.

[0118] In this way, users can complete the construction of the core execution process of the task by selecting only one compound action node. Of course, users can select other action nodes, process control nodes, and compound action nodes according to actual task requirements to improve the task execution process.

[0119] The following is an intuitive introduction to a composite action node provided in an embodiment of the present application through a specific example.

[0120] See also Figure 6 , shows a schematic diagram of a composite action node that integrates the workpiece grasping process, where rectangles represent judgment nodes or action nodes, and diamonds represent branch nodes. The designation of the photo location, light source, and camera activation can all be set by the user when selecting the composite action node. Figure 6 The process represented by the composite action node shown is introduced.

[0121] First, determine whether the photo position is specified. If so, trigger the control instruction and control the robot arm to move to the photo position, otherwise directly execute the subsequent process; then, determine whether the light source is enabled. If it is enabled, control the light source to turn on, otherwise directly execute the subsequent process; then, determine whether the camera is started. If so, control the camera to obtain the stream. Otherwise, determine whether to continue the process. If so, return to the judgment of whether the photo position is specified. If not, end the process; after controlling the camera to obtain the stream, determine whether visual recognition is enabled. If enabled, control the camera to identify the workpiece. If not, determine whether to continue the process. If so, return to the judgment of whether the photo position is specified. If so, end the process; after controlling the camera to identify the workpiece, determine whether the workpiece is identified. If so, control the robot arm to grab the workpiece and end the process. If not, determine whether to continue the process. If so, return to the judgment of whether the photo position is specified. If not, end the process.

[0122] by Figure 6 For example, after the user selects the compound action node, the relationship between the nodes can be edited according to actual needs.

[0123] For example, if the working environment of the robotic arm is well-lit and there is no need to turn on the light source for fill light, the user can directly connect the action node that controls the movement of the robotic arm to the photo position to the node that controls the camera to obtain the flow, thus eliminating redundant judgment operations.

[0124] It should be noted that Figure 6 This is a schematic diagram of a composite action node provided only for ease of understanding. Those skilled in the art may set a specific composite action node according to actual needs, and the embodiments of the present application do not limit this.

[0125] Step S402: In response to a user's relationship editing operation on a process node in the interactive interface, based on operation information of the relationship editing operation, determining a connection relationship between the process nodes targeted by the relationship editing operation.

[0126] The above relationship editing operation may include a variety of specific operations, which are introduced below in different situations.

[0127] In one case, if the relationship editing operation is a line connection operation, a connection relationship between the process nodes connected by the line connection operation is generated according to the line connection direction of the line connection operation.

[0128] For example, if the direction of the connection between process node P1 and process node P2 is from P1 to P2, the connection relationship is: P1→P2.

[0129] In another case, if the relationship editing operation is an editing operation for editing the relationship between process nodes integrated in the composite action node, the relationship between process nodes integrated in the composite action node is updated based on the operation information of the editing operation.

[0130] As can be seen from the above introduction, a compound action node integrates multiple process nodes and the relationships between process nodes. After the user selects a compound action node, the interactive interface will display the multiple process nodes integrated in the compound action node and the relationships between process nodes. The user can trigger editing operations on the relationships between process nodes. The above editing operations can include deleting relationships and creating new relationships.

[0131] In this way, users can flexibly use a variety of editing operations on the interactive interface to generate connection relationships between process nodes, improving the flexibility and convenience when building execution processes.

[0132] Step S403: constructing a task execution process based on the determined process nodes and connection relationships.

[0133] As can be seen from the above, in this embodiment, users do not need to have programming knowledge. They only need to select node components and edit the relationship between process nodes on the visual interactive interface to conveniently and quickly build the execution process of the task, bringing users a more convenient and user-friendly experience.

[0134] In one embodiment of the present application, after the user constructs the task execution process in the above manner, a graphical task execution process can be displayed in the interactive interface, including each process node in the task execution process and the connection relationship between the process nodes, so that the user can quickly and intuitively understand the specific task execution logic described in the task execution process, and then determine whether the task execution process meets the actual task requirements.

[0135] Specifically, in one embodiment of the present application, the user can also construct a task execution process by inputting programming statements in the interactive interface, which will be described in detail below with reference to the accompanying drawings.

[0136] See also Figure 7 , which is a flow chart of the second task execution process construction method provided in an embodiment of the present application, the above method includes the following steps S701-S703.

[0137] Step S701: Parse the programming statement input by the user in the interactive interface, determine the action node and process control node represented by the programming statement, and use the determined action node as the process node included in the task execution process.

[0138] Specifically, according to preset parsing rules or correspondences, a first programming statement representing an action node and a second programming statement representing a process control node can be determined from the programming statements, and the action node represented by the first programming statement and the process control node represented by the second programming statement can be used as process nodes included in the task execution process.

[0139] For example, if the first programming statement is "RobotActionA," the pre-set parsing rules determine that the action node corresponding to RobotActionA is Robot Action A. This means the first programming statement is identified as representing an action node. The method for identifying the second programming statement representing a process control node is similar and will not be further illustrated here.

[0140] Step S702: Generate connection relationships between process nodes included in the task execution process according to the arrangement order of the first programming statement and the second programming statement.

[0141] The above-mentioned first programming statement and second programming statement represent action nodes and process control nodes respectively. Therefore, the arrangement order of the first programming statement and the second programming statement can reflect the execution order of each action node and process control node, that is, the connection relationship between each action node and process control node.

[0142] The following describes how to generate connection relationships based on the arrangement order.

[0143] Specifically, the connection relationship between the process nodes corresponding to the programming statements may be generated according to the top-to-bottom arrangement relationship of adjacent programming statements.

[0144] For example, programming statements C1 and C2 are adjacent, and C1 is arranged above C2. The action node corresponding to C1 is robot action N1, and the action node corresponding to C2 is light source action N2. Then the connection relationship between robot action N1 and light source action N2 is determined to be: robot action N1→light source action N2; for another example, programming statements C3-C6 are arranged in sequence, and the nodes corresponding to C3 and C6 are the loop start node N3 and the loop end node N6 in the process control node, respectively. The nodes corresponding to C4 and C5 are robot action N4 and robot action N5, respectively. Then the connection relationship between each process node is determined to be: loop start node N3→robot action N4→robot action N5→loop end node N6.

[0145] Step S703: Constructing a task execution process based on the determined process nodes and connection relationships.

[0146] pass Figure 4 The component selection method shown in the visual interface allows users to quickly and easily build task execution flows without requiring programming experience, meeting the needs of most practical scenarios. However, in a few complex scenarios, more sophisticated or advanced operations on the robotic arm and / or vision device are required, and the pre-packaged flow nodes may not be sufficient. In these cases, users can also construct task execution flows by entering programming statements in the interactive interface, achieving more fine-grained or advanced operations.

[0147] It can be seen that this system also supports the execution process of generating tasks based on the programming language input by the user, which makes it easier for users to build more complex execution processes based on more professional programming languages.

[0148] In one embodiment of the present application, after a user constructs a task execution process using the above-described method, a graphical task execution process can be displayed in the interactive interface, including the various process nodes in the task execution process and the connection relationships between the process nodes. Thus, after the user constructs the task execution process by inputting programming statements, the electronic device can display the converted graphical task execution process on the electronic device, allowing the user to quickly and intuitively understand the specific task execution logic described in the task execution process, and then determine whether the task execution process meets the actual task requirements.

[0149] Corresponding to the above-mentioned task execution scheme, an embodiment of the present application also provides a task execution device.

[0150] See also Figure 8 , is a schematic diagram of the structure of a task execution device provided in an embodiment of the present application, which is applied to an electronic device deployed with an integrated vision and control system, wherein the integrated vision and control system is used to control a robotic arm and a visual device, wherein the robotic arm and the visual device are deployed in the same working environment, and the device includes:

[0151] A process determination module 801 is configured to determine, in response to a task start instruction, a task execution process indicated by the task start instruction;

[0152] The task execution module 802 is used to execute the control logic encapsulated in each process node in sequence according to the connection relationship between the process nodes recorded in the task execution process, and control the robotic arm and visual device to perform tasks, wherein the each process node includes: an action node encapsulating the motion control logic of the robotic arm and an action node encapsulating the motion control logic of the visual device, and the connection relationship represents: the execution order of the process nodes.

[0153] As can be seen from the above, in the solution provided by the embodiment of the present application, the process nodes in the task execution process include action nodes encapsulating the motion control logic of the robot arm and action nodes encapsulating the motion control logic of the visual device. After receiving the task start instruction, the electronic device can determine the task execution process indicated by the task start instruction, and execute the control logic encapsulated in each process node in sequence according to the connection relationship between the process nodes recorded in the task execution process, and control the robot arm and the visual device to perform the task. It can be seen that the electronic device deployed with the integrated visual control system can control the robot arm and the visual device in an orderly manner according to the task nodes and connection relationships recorded in the task execution process, thereby enabling the robot arm and the visual device to cooperate accurately to complete the task. Compared with the robot arm controller controlling the robot arm alone and the visual industrial computer controlling the visual device alone, the electronic device deployed with the integrated visual control system can realize the joint control of the robot arm and the visual device, eliminating the communication process between the robot arm controller and the visual industrial computer to achieve the coordinated cooperation of the robot arm and the visual device, simplifying the task execution process of the robot arm, and improving the efficiency of task execution.

[0154] In addition, in the related technology, when the robotic arm controller communicates with the visual industrial computer, the communication delay is high due to the influence of network environment fluctuations, etc.; while the electronic device in the solution provided in the embodiment of the present application can realize coordinated control of the robotic arm and the visual device, eliminating the communication process between the robotic arm controller and the visual industrial computer, thereby reducing the communication delay and improving the real-time performance when controlling the robotic arm and the visual device.

[0155] In one embodiment of the present application, the task execution process is constructed based on the following modules:

[0156] a connection relationship determination module, configured to determine, in response to an execution process editing operation performed by a user in an interactive interface, process nodes included in a task execution process and connection relationships between process nodes based on operation information of the execution process editing operation;

[0157] The process construction module is used to construct the task execution process based on the determined process nodes and connection relationships.

[0158] As can be seen from the above, the solution provided in the embodiment of the present application supports constructing the execution process of the task through an interactive interface, which is more user-friendly, reduces the difficulty of constructing the execution process, and improves the efficiency of constructing the execution process.

[0159] In one embodiment of the present application, the execution process editing operation includes: a selection operation on a node component displayed in the interactive interface and a relationship editing operation on the node component.

[0160] The connection relationship determination module is specifically used to respond to the user's selection operation on the node component displayed in the interactive interface, determine the node corresponding to the selected node component as the process node included in the task execution process, wherein the nodes corresponding to the displayed node component include: action nodes and process control nodes, and the process control nodes are used to control the execution order of action nodes; in response to the user's relationship editing operation on the process node in the interactive interface, based on the operation information of the relationship editing operation, determine the connection relationship between the process nodes targeted by the relationship editing operation.

[0161] As can be seen from the above, in this embodiment, users do not need to have programming knowledge. They only need to select node components and edit the relationship between process nodes on the visual interactive interface to conveniently and quickly build the execution process of the task, bringing users a more convenient and user-friendly experience.

[0162] In one embodiment of the present application, the node corresponding to the displayed node component includes a composite action node, wherein the composite action node integrates multiple process nodes and relationships between process nodes.

[0163] It can be seen that the composite action node integrates multiple process nodes and the relationships between process nodes. By selecting the composite action node, users can reduce the selection of single action nodes and relationship editing operations, thereby improving the efficiency of process construction.

[0164] In one embodiment of the present application, the composite action node is generated in the following manner:

[0165] In response to a user's selection operation on a node component displayed in the interactive interface, the node corresponding to the selected node component is determined as a process node included in the composite action node; in response to a user's relationship editing operation on a process node corresponding to the selected node component in the interactive interface, based on operation information of the relationship editing operation, the connection relationship between the process nodes targeted by the relationship editing operation is determined; based on the connection relationship between the process node corresponding to the selected node component and the process nodes, the composite action node is generated.

[0166] In this way, the electronic device can encapsulate each process node and the connection relationship between the constructed process nodes into a composite node, which is convenient for the user to select and use later, thereby improving the efficiency of process construction.

[0167] In one embodiment of the present application, the connection relationship determination module is specifically used to generate a connection relationship between the process nodes connected by the connection operation according to the connection direction of the connection operation if the relationship editing operation is a connection operation; if the relationship editing operation is an editing operation of editing the relationship between the process nodes integrated in the composite action node, update the relationship between the process nodes integrated in the composite action node based on the operation information of the editing operation.

[0168] In this way, users can flexibly use a variety of editing operations on the interactive interface to generate connection relationships between process nodes, improving the flexibility and convenience when building execution processes.

[0169] In one embodiment of the present application, the execution process editing operation includes: inputting a programming statement in the interactive interface,

[0170] The connection relationship determination module is specifically used to parse the programming statements input by the user in the interactive interface, determine the action nodes and process control nodes represented by the programming statements, and use the determined action nodes as the process nodes included in the task execution process, wherein the process control nodes are used to control the execution order of the action nodes; according to the arrangement order of the first programming statement and the second programming statement, generate the connection relationship between the process nodes included in the task execution process, wherein the first programming statement is a programming statement representing the action node, and the second programming statement is a programming statement representing the process control node.

[0171] It can be seen that this system also supports the execution process of generating tasks based on the programming language input by the user, which makes it easier for users to build more complex execution processes based on more professional programming languages.

[0172] In one embodiment of the present application, the device further comprises:

[0173] an exception handling module, configured to, in response to an exception occurring during the execution of a first control logic encapsulated by a first process node in the task execution process, suspend the execution of the first control logic and execute the first exception handling logic encapsulated by the first process node if the first process node encapsulates exception handling logic;

[0174] A retry module is configured to re-execute the first control logic in response to completion of execution of the first exception handling logic.

[0175] That is, when the electronic device determines that an exception occurs in the control logic of a process node during execution, it can interrupt the task in a timely manner and run the exception handling logic. After the exception handling logic is completed, the control logic of the process node is re-executed for retry, thereby improving the robustness of the task execution process.

[0176] In one embodiment of the present application, the second process node included in the task execution process encapsulates the following exception handling logic, and the second process node is: a process node encapsulating the second control logic for controlling the visual device to perform image acquisition:

[0177] Determine an abnormal image in a target image, wherein the target image is: executing the second control logic to control the image captured by the visual device; determining a position adjustment parameter of the visual device based on image quality information of the abnormal image; and controlling the movement of the visual device based on the position adjustment parameter.

[0178] In this embodiment, based on the pre-packaged exception handling logic, when an image that meets the requirements is not captured after the second process node is executed, the position of the visual device can be adaptively adjusted so that the visual device after the adjustment can capture a better image, thereby improving the accuracy of visual recognition and thus improving the accuracy of task execution.

[0179] In one embodiment of the present application, the device further comprises:

[0180] an interface configuration module, configured to, in response to receiving input / output (IO) interface configuration information, control the robotic arm to establish a one-way IO interface subscription relationship with the electronic device, so that the robotic arm synchronizes the IO interface configuration with the electronic device, wherein the IO interface configuration is used to determine an action node that can be used to control the robotic arm;

[0181] A signal configuration module is used to control the robotic arm to establish a bidirectional IO signal subscription relationship with the electronic device in response to receiving IO signal configuration information, so that the IO signals between the robotic arm and the electronic device are synchronized with each other, wherein the IO signals are used to determine the action execution status of the robotic arm during the execution of the task execution process.

[0182] In this way, the electronic device can promptly know the IO interfaces supported and bound by the robotic arm, and thus know the action nodes that can be used to control the robotic arm, so that a usable task execution process can be constructed subsequently; and the robotic arm and the electronic device can synchronize IO signals in a timely manner, so that the electronic device can promptly know the execution status of various IO operations performed by the robotic arm, that is, the task execution status.

[0183] The present application also provides an electronic device, such as Figure 9 As shown, including:

[0184] Memory 901, used for storing computer programs;

[0185] The processor 902 is configured to execute the program stored in the memory 901 and implement the following steps:

[0186] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 902 , the communication interface, and the memory 901 communicate with each other via the communication bus.

[0187] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0188] The communication interface is used for communication between the above electronic device and other devices.

[0189] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0190] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0191] In another embodiment provided by the present application, a computer-readable storage medium is further provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned task execution method are implemented.

[0192] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the task execution methods in the above embodiments.

[0193] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state drive (SSD).

[0194] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0195] Each embodiment in this specification is described in a related manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the method, apparatus, electronic device, and storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For related portions, reference can be made to the descriptions of the method embodiments.

[0196] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. A task execution method, characterized in that: Applied to an electronic device deployed with an integrated vision and control system, the integrated vision and control system being used to control a robotic arm and a visual device, the robotic arm and the visual device being deployed in the same working environment, the method comprising: In response to a task start instruction, determining a task execution process indicated by the task start instruction; According to the connection relationship between the process nodes recorded in the task execution process, the control logic encapsulated in each process node is executed in sequence to control the robotic arm and the visual device to perform the task, wherein the each process node includes: an action node encapsulating the motion control logic of the robotic arm and an action node encapsulating the motion control logic of the visual device, and the connection relationship represents: the execution order of the process nodes.

2. The method according to claim 1, characterized in that The task execution process is constructed as follows: In response to an execution process editing operation performed by a user in an interactive interface, determining process nodes included in a task execution process and connection relationships between the process nodes based on operation information of the execution process editing operation; The task execution process is constructed based on the determined process nodes and connection relationships.

3. The method according to claim 2, characterized in that The execution process editing operation includes: a selection operation on a node component displayed in an interactive interface and a relationship editing operation on the node component. The response to the user's execution process editing operation in the interactive interface is to determine the process nodes included in the task execution process and the connection relationships between the process nodes based on the operation information of the execution process editing operation, including: In response to a user selecting a node component displayed in the interactive interface, determining a node corresponding to the selected node component as a process node included in the task execution process, wherein the node corresponding to the displayed node component includes: an action node and a process control node, wherein the process control node is used to control the execution order of the action node; In response to a user's relationship editing operation on process nodes in the interactive interface, a connection relationship between the process nodes targeted by the relationship editing operation is determined based on operation information of the relationship editing operation.

4. The method according to claim 3, characterized in that The nodes corresponding to the displayed node components include compound action nodes, where: The composite action node integrates multiple process nodes and the relationships between the process nodes.

5. The method according to claim 4, characterized in that The composite action node is generated in the following manner: In response to a user's selection operation on a node component displayed in the interactive interface, determining a node corresponding to the selected node component as a process node included in the composite action node; In response to a user performing a relationship editing operation on a process node corresponding to a selected node component in the interactive interface, determining a connection relationship between the process nodes targeted by the relationship editing operation based on operation information of the relationship editing operation; The composite action node is generated based on the process nodes corresponding to the selected node components and the connection relationship between the process nodes.

6. The method according to claim 4, characterized in that The determining, based on the operation information of the relationship editing operation, the connection relationship between the process nodes targeted by the relationship editing operation includes: If the relationship editing operation is a connection operation, generating a connection relationship between the process nodes connected by the connection operation according to the connection direction of the connection operation; If the relationship editing operation is an editing operation for editing the relationship between process nodes integrated in the composite action node, the relationship between process nodes integrated in the composite action node is updated based on the operation information of the editing operation.

7. The method according to claim 2, characterized in that The execution process editing operation includes: inputting a programming statement in the interactive interface; and determining the process nodes included in the task execution process and the connection relationship between the process nodes based on the operation information of the execution process editing operation includes: Parsing a programming statement input by a user on the interactive interface, determining an action node and a process control node represented by the programming statement, and using the determined action node as a process node included in the task execution process, wherein the process control node is used to control the execution order of the action nodes; According to the arrangement order of the first programming statement and the second programming statement, a connection relationship between the process nodes included in the task execution process is generated, wherein the first programming statement is a programming statement representing an action node, and the second programming statement is a programming statement representing a process control node.

8. The method according to claim 1, characterized in that The method further comprises: In response to an exception occurring during the execution of a first control logic encapsulated by a first process node in the task execution process, if the first process node encapsulates exception handling logic, suspending execution of the first control logic and executing the first exception handling logic encapsulated by the first process node; In response to the first exception handling logic being executed, the first control logic is re-executed.

9. The method according to claim 1, characterized in that The second process node included in the task execution process encapsulates the following exception handling logic, and the second process node is: a process node encapsulating a second control logic for controlling the visual device to perform image acquisition: Determining an abnormal image in a target image, wherein the target image is: an image acquired by the visual device controlled by executing the second control logic; determining a position adjustment parameter of the visual device based on image quality information of the abnormal image; Based on the position adjustment parameter, the movement of the visual device is controlled.

10. The method according to any one of claims 1 to 9, characterized in that The method further comprises: In response to receiving input / output (IO) interface configuration information, controlling the robotic arm to establish a one-way IO interface subscription relationship with the electronic device, so that the robotic arm synchronizes the IO interface configuration with the electronic device, wherein the IO interface configuration is used to determine an action node that can be used to control the robotic arm; In response to receiving the IO signal configuration information, the robotic arm is controlled to establish a bidirectional IO signal subscription relationship with the electronic device so that the IO signals between the robotic arm and the electronic device are synchronized with each other, wherein the IO signal is used to determine the action execution status of the robotic arm during the execution of the task execution process.

11. A task execution device, characterized in that: Applicable to electronic equipment deployed with an integrated vision and control system, wherein the integrated vision and control system is used to control a robotic arm and a visual device, wherein the robotic arm and the visual device are deployed in the same working environment, the device comprises: a process determination module, configured to determine, in response to a task start instruction, a task execution process indicated by the task start instruction; The task execution module is used to execute the control logic encapsulated in each process node in sequence according to the connection relationship between the process nodes recorded in the task execution process, and control the robotic arm and visual device to perform tasks, wherein the various process nodes include: an action node encapsulating the motion control logic of the robotic arm and an action node encapsulating the motion control logic of the visual device, and the connection relationship represents: the execution order of the process nodes.

12. The device according to claim 11, characterized in that The task execution process is built based on the following modules: a connection relationship determination module, configured to determine, in response to an execution process editing operation performed by a user in an interactive interface, process nodes included in a task execution process and connection relationships between process nodes based on operation information of the execution process editing operation; A process construction module, configured to construct the task execution process based on the determined process nodes and connection relationships; and / or The execution process editing operation includes: a selection operation for a node component displayed in an interactive interface and a relationship editing operation for the node component. The connection relationship determination module is specifically used to, in response to the user's selection operation for the node component displayed in the interactive interface, determine the node corresponding to the selected node component as a process node included in the task execution process, wherein the node corresponding to the displayed node component includes: an action node and a process control node, and the process control node is used to control the execution order of the action node; in response to the user's relationship editing operation for the process node in the interactive interface, determine the connection relationship between the process nodes targeted by the relationship editing operation based on the operation information of the relationship editing operation; and / or The nodes corresponding to the displayed node components include composite action nodes, wherein the composite action nodes integrate multiple process nodes and relationships between process nodes; and / or The composite action node is generated in the following manner: in response to a user's selection operation on a node component displayed in the interactive interface, determining a node corresponding to the selected node component as a process node included in the composite action node; in response to a user's relationship editing operation on a process node corresponding to the selected node component in the interactive interface, determining, based on operation information of the relationship editing operation, a connection relationship between the process nodes targeted by the relationship editing operation; and generating the composite action node based on the connection relationship between the process node corresponding to the selected node component and the process node; and / or The connection relationship determination module is specifically configured to, if the relationship editing operation is a connection operation, generate a connection relationship between the process nodes connected by the connection operation according to the connection direction of the connection operation; if the relationship editing operation is an editing operation of the relationship between the process nodes integrated in the composite action node, update the relationship between the process nodes integrated in the composite action node based on the operation information of the editing operation; and / or The execution process editing operation includes: an operation of inputting a programming statement in the interactive interface; the connection relationship determination module is specifically used to parse the programming statement input by the user in the interactive interface, determine the action node and process control node represented by the programming statement, and use the determined action node as the process node included in the task execution process, wherein the process control node is used to control the execution order of the action node; based on the arrangement order of the first programming statement and the second programming statement, generate a connection relationship between the process nodes included in the task execution process, wherein the first programming statement is a programming statement representing an action node, and the second programming statement is a programming statement representing a process control node; and / or The apparatus further includes: an exception handling module, configured to, in response to an exception occurring during the execution of a first control logic encapsulated by a first process node in the task execution process, suspend execution of the first control logic and execute the first exception handling logic encapsulated by the first process node if the first process node encapsulates exception handling logic; and a retry module, configured to, in response to completion of execution of the first exception handling logic, re-execute the first control logic; and / or The second process node included in the task execution process encapsulates the following exception handling logic, and the second process node is: a process node encapsulating a second control logic for controlling the visual device to perform image acquisition: determining an abnormal image in a target image, wherein the target image is an image acquired by the visual device under the control of executing the second control logic; determining a position adjustment parameter of the visual device based on image quality information of the abnormal image; and controlling the movement of the visual device based on the position adjustment parameter; and / or The device also includes: an interface configuration module, which is used to control the robotic arm to establish a unidirectional IO interface subscription relationship with the electronic device in response to receiving input and output IO interface configuration information, so that the robotic arm synchronizes the IO interface configuration to the electronic device, wherein the IO interface configuration is used to determine an action node that can be used to control the robotic arm; a signal configuration module, which is used to control the robotic arm to establish a bidirectional IO signal subscription relationship with the electronic device in response to receiving IO signal configuration information, so that the robotic arm and the electronic device synchronize IO signals with each other, wherein the IO signal is used to determine the action execution status of the robotic arm during the execution of the task execution process.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 10 when executing a program stored in a memory.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images