Mechanical arm control method and device

Abstract

The embodiment of the invention provides a mechanical arm control method and device, and relates to the technical field of robots, and the method applied to a server side comprises the steps that in response to a task starting instruction, a to-be-executed first visual processing scheme is loaded from multiple sets of visual processing schemes; an end effector of the mechanical arm is controlled to move to the position, recorded in a first execution configuration sub-scheme included in the first visual processing scheme, of triggering image collection, first visual equipment configured in a first equipment management sub-scheme included in the first visual processing scheme is controlled to conduct image collection, and a first image is obtained; processing the first image according to an image processing flow recorded in a first image processing sub-scheme included in the first visual processing scheme to obtain a target position of a task target in the first image; and controlling the end effector to move to the target position, and controlling the end effector to execute the action recorded in the first execution configuration sub-scheme. By applying the scheme provided by the embodiment of the invention, the task execution efficiency of the mechanical arm can be improved.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a method and device for controlling a robotic arm. Background Art

[0002] With the continuous development of industrial automation and intelligent manufacturing technology, robotic arm control technology has been widely used. Its core lies in achieving precise positioning and operation of the robotic arm on the mission target, so as to complete the task accurately and efficiently.

[0003] In order to cope with more diverse work scenarios and more complex work content, it is generally necessary for the robot arm and the camera to work together to complete the task. Among them, the camera is responsible for providing visual functions for the robot arm, so that the robot arm can perceive the environment in real time and then successfully complete the task based on the perception results. In related technologies, the robot arm is controlled by the robot arm controller, and the camera is controlled by the visual industrial computer. The interaction process between the various devices is as follows: the camera sends the collected image to the visual industrial computer; then, the visual industrial computer processes the image, calculates the posture information of the task target in the image based on the processing results, and sends the posture information to the robot arm controller; finally, the robot arm controller plans the movement path of the robot arm based on the posture information, and generates control instructions to control the robot arm to move according to the planned path.

[0004] As can be seen, the above process requires not only a communication connection between the robot controller and the robot arm, and a communication connection between the visual industrial computer and the camera, but also a communication connection between the robot controller and the visual industrial computer. As a result, in the case of network fluctuations, the communication latency between these various devices is high. In severe cases, the fluctuations may even cause the devices to disconnect, resulting in inefficient interaction between the devices and, consequently, low efficiency in the robot arm's task execution. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a method and device for controlling a robotic arm to improve the efficiency of the robotic arm in executing tasks. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present application provides a robotic arm control method, which is applied to a server, wherein the server is connected to a robotic arm and a visual device, wherein the robotic arm and the visual device are deployed in the same working environment, wherein the visual device is used to collect images of task targets, and wherein the server is registered with multiple sets of visual processing schemes, each set of visual processing schemes including: a device management sub-scheme, an image processing sub-scheme, and an execution configuration sub-scheme, wherein the method includes:

[0007] In response to a task start instruction, loading a first visual processing solution to be executed from the plurality of visual processing solutions;

[0008] controlling the end effector of the robotic arm to move to a position for triggering image acquisition recorded in a first execution configuration sub-scheme included in the first visual processing scheme, and controlling a first visual device configured in a first device management sub-scheme included in the first visual processing scheme to perform image acquisition to obtain a first image;

[0009] Processing the first image according to the image processing flow recorded in the first image processing sub-scheme included in the first visual processing scheme to obtain a target position of the task target in the first image;

[0010] The end effector of the robotic arm is controlled to move to the target position, and the end effector is controlled to execute the action recorded in the first execution configuration sub-scheme.

[0011] In a second aspect, an embodiment of the present application provides a method for controlling a robotic arm, which is applied to a client, and the method includes:

[0012] In response to a device editing operation for the visual processing solution to be configured in the first configuration interface, sending device editing information to the server, so that the server configures the device management sub-solution in the visual processing solution to be configured based on the received device editing information;

[0013] In response to an image processing flow editing operation for the visual processing solution to be configured in the second configuration interface, sending image processing flow editing information to the server, so that the server configures the image processing sub-solution in the visual processing solution to be configured based on the received image processing flow editing information;

[0014] In response to the execution process editing operation for the visual processing scheme to be configured in the third configuration interface, execution process editing information is sent to the server, so that the server configures the execution configuration sub-scheme in the visual processing scheme to be configured based on the received execution process editing information.

[0015] In a third aspect, an embodiment of the present application provides a robotic arm control device, which is applied to a server, wherein the server is connected to a robotic arm and a visual device, wherein the robotic arm and the visual device are deployed in the same working environment, wherein the visual device is used to collect images of task targets, and wherein the server is registered with multiple sets of visual processing schemes, each set of visual processing schemes including: a device management sub-scheme, an image processing sub-scheme, and an execution configuration sub-scheme, wherein the device includes:

[0016] a solution loading module, configured to load a first visual processing solution to be executed from the plurality of visual processing solutions in response to a task start instruction;

[0017] a first image acquisition module, configured to control the end effector of the robotic arm to move to a position triggering image acquisition recorded in the first execution configuration sub-scheme included in the first visual processing scheme, and control the first visual device configured in the first device management sub-scheme included in the first visual processing scheme to perform image acquisition to obtain a first image;

[0018] an image processing module, processing the first image according to the image processing flow recorded in the first image processing sub-scheme included in the first visual processing scheme, to obtain a target position of the task target in the first image;

[0019] An action execution module is used to control the end effector to move to the target position and control the end effector to execute the action recorded in the first execution configuration sub-scheme.

[0020] In a fourth aspect, an embodiment of the present application provides a robotic arm control device, applied to a client, the device comprising:

[0021] a first operation response module, configured to, in response to a device editing operation for the visual processing solution to be configured in the first configuration interface, send device editing information to the server, so that the server configures a device management sub-solution in the visual processing solution to be configured based on the received device editing information;

[0022] a second operation response module, configured to, in response to an image processing flow editing operation for the visual processing solution to be configured in the second configuration interface, send image processing flow editing information to the server, so that the server configures the image processing sub-solution in the visual processing solution to be configured based on the received image processing flow editing information;

[0023] The third operation response module is used to respond to the execution process editing operation for the visual processing scheme to be configured in the third configuration interface, and send execution process editing information to the server, so that the server configures the execution configuration sub-scheme in the visual processing scheme to be configured based on the received execution process editing information.

[0024] In a fifth aspect, an embodiment of the present application provides an electronic device, including:

[0025] Memory for storing computer programs;

[0026] The processor is configured to implement the method described in the first aspect or the second aspect when executing a program stored in the memory.

[0027] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect or the second aspect is implemented.

[0028] In a seventh aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect or the second aspect.

[0029] As can be seen from the above, in the solution provided by the embodiment of the present application, the server is connected to the robotic arm and the visual device at the same time, so that the robotic arm and the visual device can be collaboratively controlled according to the visual processing solution. Compared with the visual industrial computer controlling the visual device alone and the robotic arm controller controlling the robotic arm alone, the above method eliminates the communication interaction between the visual industrial computer and the robotic arm controller, increases communication efficiency while reducing communication delay and improving response speed, that is, improving the real-time performance of controlling the robotic arm, thereby improving the task execution efficiency of the robotic arm, and performing excellently in tasks requiring high precision and high real-time performance.

[0030] In addition, in related technologies, professional engineers are required to set up corresponding processes or input instructions on the robot arm controller and the visual industrial computer according to the specific process of the task to be executed, so that the robot arm and camera can cooperate smoothly to perform the task. In the solution provided by the embodiment of the present application, the server registers multiple sets of visual processing solutions. Therefore, in response to the task start instruction, the server can directly control the robot arm and visual equipment to perform the task according to the visual processing solution to be executed, without the need for professional personnel to set up the process, which reduces the difficulty of controlling the robot arm and improves the control efficiency.

[0031] Furthermore, in the solution provided in the embodiment of the present application, a server is used to control the connected robotic arms and visual devices, thereby realizing the deep integration of visual control and motion control. Compared with the visual industrial computer separately controlling the visual devices and the robotic arm controller separately controlling the robotic arms, the system architecture is simplified, which not only reduces the hardware cost, but also improves the stability of the system and reduces the maintenance cost of the system.

[0032] Of course, it is not necessary to achieve all the advantages described above at the same time when implementing any product or method of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0034] Figure 1a A schematic diagram of a robotic arm performing a task according to an embodiment of the present application;

[0035] Figure 1b A schematic diagram of the first visual processing solution provided in an embodiment of the present application;

[0036] Figure 1c A schematic diagram of the second visual processing solution provided in an embodiment of the present application;

[0037] Figure 2 A schematic flow chart of a first robotic arm control method provided in an embodiment of the present application;

[0038] Figure 3 A flowchart of a visual calibration method provided in an embodiment of the present application;

[0039] Figure 4 A schematic diagram of a calibration point provided in an embodiment of the present application;

[0040] Figure 5 A schematic diagram of a visual calibration process provided in an embodiment of the present application;

[0041] Figure 6 A schematic diagram of an operator execution process provided in an embodiment of the present application;

[0042] Figure 7 A schematic diagram of a serialization method provided in an embodiment of the present application;

[0043] Figure 8 A schematic diagram of a synchronization method provided in an embodiment of the present application;

[0044] Figure 9 A schematic diagram of a robotic arm control process provided in an embodiment of the present application;

[0045] Figure 10 A schematic flow chart of a second robotic arm control method provided in an embodiment of the present application;

[0046] Figure 11 A schematic structural diagram of a robotic arm system provided in an embodiment of the present application;

[0047] Figure 12a A schematic diagram of a robotic arm control scenario in related technology;

[0048] Figure 12b A schematic diagram of a robotic arm control scenario provided in an embodiment of the present application;

[0049] Figure 13 A schematic structural diagram of a first robotic arm control device provided in an embodiment of the present application;

[0050] Figure 14 A schematic structural diagram of a second robotic arm control device provided in an embodiment of the present application;

[0051] Figure 15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of this application.

[0053] First, the application scenarios of the solutions provided in the embodiments of the present application are introduced.

[0054] The application scenario of the solution provided in the embodiment of the present application is a scenario in which a robotic arm locates a task target in a working environment with the help of the visual function provided by a visual device, and then performs a set action on the task target.

[0055] Among them, the above-mentioned visual equipment may include image acquisition equipment, light source equipment, etc. The visual equipment can be installed on the robotic arm or installed at a set position in the working environment. It is only necessary to ensure that the task target is within the field of view of the visual equipment; the above-mentioned task target can be workpieces, goods, packages, etc. in the production line; the above-mentioned set actions can be grasping, adsorption, assembly, measurement and other actions for the task target, which can be flexibly set according to actual task requirements. The above-mentioned set actions can be specifically implemented by the end effector in the robotic arm (such as grippers, suction cups, cutting tools, welding tools, etc.).

[0056] The following combination Figure 1a This section provides a more intuitive introduction to the above scenario.

[0057] like Figure 1a As shown, an image acquisition device 102 is installed on the robot arm 101, and the image acquisition device 102 acquires images of the workpiece 103 in the production line. The acquired images are used to locate the workpiece 103; the robot arm controls the gripper 1011 to grab the workpiece 103 according to the positioning result, and then places the workpiece 103 to the set position in the working environment.

[0058] It should be noted that Figure 1aThe illustrated scenarios are merely examples for ease of understanding and do not constitute limitations on the embodiments of this application. For example, the image acquisition device 102 can be mounted via a fixed bracket in the space above or below the robotic arm. Furthermore, after the gripper 1011 grasps the workpiece 103, it can perform a predetermined assembly operation on the workpiece 103 and then return the assembled workpiece to the production line. All of these are reasonable options.

[0059] The following is an introduction to the execution entity of the solution provided in the embodiment of this application.

[0060] The execution subject of the solution provided in the embodiment of the present application is any electronic device with data processing, storage, communication and other functions. Specifically, the execution subject of the solution provided in the embodiment of the present application is the server deployed in the above electronic device, and the above server is software deployed in the electronic device. The above electronic device can be a control device. When the control device executes this solution, the server deployed therein can establish communication connections with both the robot arm and the visual device in advance. The above electronic device can also be the robot arm itself. When the robot arm executes this solution, the server deployed therein can establish communication connections with both the visual device in advance.

[0061] To facilitate understanding of the solutions provided in the embodiments of the present application, the visual processing solutions involved in the embodiments of the present application are first introduced below.

[0062] The visual processing solution, also known as the visual control solution or visual solution, is a solution used to control the coordination and cooperation between the robotic arm and the visual device to complete the task. In other words, the server can control the coordination and cooperation between the robotic arm and the visual device to complete the task according to the information recorded in the visual processing solution.

[0063] The following combination Figure 1b and Figure 1c Provides an intuitive introduction to vision processing solutions.

[0064] like Figure 1b As shown in the figure, the vision processing solution is registered in the form of a visual control solution list with the electronic device deployed on the server. When the electronic device is a robotic arm, the vision processing solution can be registered with the robotic arm's main project. The above-mentioned vision processing solutions include multiple vision processing solutions such as Vision Solution 1, Vision Solution 2, Vision Solution 3, and Vision Solution 4.

[0065] Each visual processing solution includes: device management sub-solution, image processing sub-solution and execution configuration sub-solution, such as Figure 1b Visual scheme 2 shown. Figure 1b The visual scheme 1 and visual schemes 3 to 4 shown also include the above three sub-schemes, which are not shown to make the drawings simpler.

[0066] The device management sub-solution, also known as the device solution, manages visual devices, including cameras and input / output (IO) light sources. The device management sub-solution configures the operating status, name, and IP address of each visual device connected to the server through an interface. These operating states include powering on, dormant, and shutting down the device.

[0067] The visual equipment configured in the above device management sub-solution can be set up at a set photo-taking point in the robot arm's working scene, or it can be deployed directly on the robot arm.

[0068] In one embodiment of the present application, the device management sub-scheme can also configure the image acquisition frame rate, exposure mode, exposure time, gain, and other parameters of the image acquisition device within the visual system. This allows personnel to configure these acquisition parameters within the device management sub-scheme based on the actual needs of the task execution environment, ensuring that the image acquisition device captures clear images during the task execution process.

[0069] The image processing sub-scheme can also be called the image processing scheme, which is responsible for managing the image processing process, which records the visual operators and the operator execution process of the visual operators, and is used to process the images collected by the visual device. The above-mentioned visual operators can also be called image processing operators.

[0070] The execution configuration sub-scheme can also be called the execution configuration scheme, which is responsible for managing the overall task execution process, including the photo-taking points of the task targets, the actions performed for each task target, the execution order of each action, etc. Among them, the above-mentioned photo-taking points can also be called the positions that trigger image acquisition. When the end effector of the robot arm moves to the photo-taking point, the camera can be controlled to capture the image. Specifically, after the end effector moves to the photo-taking point, it can be considered that the robot arm is ready to perform operations on the task target. At this time, the task target needs to be positioned more accurately. Therefore, the server can control the camera to capture images, so as to accurately locate the task target based on the image, so that the end effector can accurately move to the position of the task target.

[0071] Specifically, when running the configuration sub-scheme, the main operations are image processing and action flow control, which are introduced below.

[0072] Action flow control operations include point movement and device control. For example, controlling the end effector of the robotic arm to move to the photo point recorded in the execution configuration sub-plan and controlling the visual device to take a photo. Then, controlling the end effector to move to the target position of the task objective and controlling the end effector to perform the action recorded in the execution configuration sub-plan. When performing device control, it is necessary to call the information related to the visual device recorded in the device management sub-plan to determine the visual device involved in the task execution and the acquisition parameters of the visual device.

[0073] Image processing refers to calling the image processing sub-scheme to process the collected image to obtain the target position of the task target.

[0074] It can be seen that each visual processing solution is an executable solution. Staff can customize the visual processing solution to implement different functional applications based on the actual needs of various scenarios.

[0075] In one embodiment of the present application, Figure 1c As shown, the visual processing solution may also include: a visual calibration sub-solution.

[0076] The vision calibration sub-scheme is used to configure the calibration method between the robotic arm and the visual device. It also records the camera installation method, which includes camera installation above (installed above the robotic arm), camera installation below (installed below the robotic arm), and camera end installation (installed at the end of the robotic arm). Different installation methods affect the calculations during the vision calibration process. In addition, the vision calibration sub-scheme is also responsible for applying the calibration results.

[0077] The above calibration methods include contour point calibration and image calibration. Contour point calibration only requires the staff to teach the calibration center point and calibration corner point. See the following for details. Figure 3 The description in the illustrated embodiment will not be described in detail here; image calibration refers to calibration using a preset calibration plate.

[0078] In one scenario, the robotic arm and camera can be pre-calibrated for different tasks to be performed, and the calibration results obtained from this pre-calibration can be recorded in the visual calibration sub-scheme. This allows the calibration results to be directly applied when executing the visual processing scheme. Furthermore, when executing different visual processing schemes, image processing can be performed according to the mapping between different image coordinate systems and the robotic arm base coordinate system based on the different calibration results in the visual calibration sub-scheme. This allows for application in diverse and complex scenarios, making the application of the visual processing scheme more flexible.

[0079] The general process of the robotic arm control solution provided in the embodiment of the present application is then introduced.

[0080] The server first controls the movement of the robotic arm's end effector to the position that triggers image acquisition, also known as the photo point, according to the execution process recorded in the execution configuration sub-scheme of the visual processing scheme. Then, according to the device management sub-scheme, it determines the visual device used for image acquisition and controls the visual device to acquire the image. It then obtains the image of the working environment captured by the visual device and calls the visual operators configured in the image processing sub-scheme to preprocess and extract features from the acquired image. During this process, the server can use visual operators such as image enhancement and template matching to identify and analyze the task target in the image, obtain the image coordinates of the task target in the image, and provide key information for subsequent robotic arm motion planning.

[0081] After obtaining the image coordinates of the task target based on image recognition, the server can call the calibration results recorded in the visual calibration sub-scheme included in the visual processing solution, convert the above image coordinates into the target posture of the robot arm, and pass the target posture to the robot arm control module. The robot arm control module performs motion planning so that the end effector of the robot arm can accurately reach the position of the task target and perform corresponding actions for the task target.

[0082] The following is a detailed introduction to the robotic arm control solution provided in the embodiments of the present application.

[0083] See also Figure 2 , which is a flow chart of the first robotic arm control method provided in an embodiment of the present application. The above method is applied to the server and includes the following steps S201-S204.

[0084] Step S201: In response to a task start instruction, a first visual processing solution to be executed is loaded from a plurality of visual processing solutions.

[0085] First, the situation where the server confirms receipt of the task start instruction is introduced.

[0086] In one scenario, a worker can trigger a task initiation operation by, for example, clicking an icon on the client's interactive interface. In response to this operation, the client sends task initiation information to the server, at which point the server deems it has received the task initiation instruction. The client is deployed in a teach pendant, pre-connected to the server, and configured to send configuration information to the server in response to configuration operations triggered by the worker on the interactive interface. The server can then generate a visual solution based on the received configuration information (see subsequent embodiments for details). The client is also configured to send task initiation information, task termination information, and other information to the server in response to the worker's operation.

[0087] Alternatively, a worker can directly enter a task start instruction in the server console, at which point the server will deem it has received the task start instruction. When the server executes the visual processing solution by entering a task start instruction, the worker can use advanced instructions to indicate how the visual processing solution should be executed. For example, they can instruct the execution of the entire visual processing solution, or instruct the execution of a portion of the entire visual processing solution, or modify a process in the visual processing solution and execute the modified process.

[0088] The following introduces the first visual processing solution to be executed.

[0089] In one case, the first visual processing scheme to be executed may be a default visual processing scheme set among multiple sets of visual processing schemes.

[0090] In another case, the first visual processing solution may also be the visual processing solution corresponding to the solution identifier sent by the client. For example, a staff member may trigger a solution selection operation by clicking an icon on the client's interactive interface. In response to this operation, the client sends the solution identifier of the selected visual processing solution to the server, and the server then determines the first visual processing solution as the visual processing solution corresponding to the solution identifier.

[0091] Step S202: Control the end effector of the robot arm to move to the position for triggering image acquisition recorded in the first execution configuration sub-scheme included in the first visual processing scheme, control the first visual device configured in the first device management sub-scheme included in the first visual processing scheme to perform image acquisition, and obtain a first image.

[0092] As described above for the visual processing solution, the first execution configuration sub-solution included in the first visual processing solution records the location that triggers image acquisition, also known as the photo capture point mentioned above. The first device management sub-solution can also be configured with a visual device responsible for capturing images when the end effector moves to the image acquisition location.

[0093] The above-mentioned first visual device is: a visual device configured in the first device management sub-solution included in the first visual processing solution, which is used to collect images when the end effector moves to the photo-taking point.

[0094] After the end effector moves to the photo-taking point, the robotic arm can send feedback information to the server. Upon receiving the feedback information, the server can assume that the robotic arm is ready to execute subsequent processes, thereby controlling the first visual device to capture images and obtain a first image, so as to accurately locate the position of the task target based on the first image.

[0095] In one embodiment of the present application, a visual device includes: an image acquisition device and a light source device, wherein the image acquisition device is connected to an electronic device deployed on a server through a first interface, and the light source device is connected to the above-mentioned electronic device through a second interface. In this case, when the server controls the first visual device to acquire an image, it can first send a first control instruction to the first light source device in the first visual device through the second interface, so that the first light source device starts lighting; then, it sends a second control instruction to the first image acquisition device in the first visual device through the first interface, so that the first image acquisition device performs image acquisition; finally, in response to the first image acquisition device successfully acquiring the first image, it sends a third control instruction to the first light source device through the second interface, so that the first light source device turns off lighting.

[0096] The above-mentioned first interface can be a network port. In this case, the first control instruction is an image acquisition instruction sent through the network port; the above-mentioned second interface can be a network port. In this case, the second control instruction and the third control instruction are respectively the light source turn-on instruction and the light source turn-off instruction sent through the network port; the above-mentioned second interface can also be an IO interface. In this case, the second control instruction and the third control instruction are respectively the light source turn-on signal and the light source turn-off signal sent through the IO interface.

[0097] In this way, the server can first control the light source device to turn on the lighting, and then control the image acquisition device to capture the image, reducing the probability of low brightness in the captured image due to insufficient ambient brightness, and improving the quality of the captured image.

[0098] Step S203: Process the first image according to the image processing flow recorded in the first image processing sub-scheme included in the first visual processing scheme to obtain the target position of the task target in the first image.

[0099] The first image processing sub-scheme records an image processing process, and the image processing process can be recorded in various forms.

[0100] In one scenario, the image processing flow is recorded in the form of an image processing operation linked list. Specifically, the first image processing sub-scheme defines an image processing operation linked list, which includes multiple image operation nodes. The pointer directions between the image operation nodes represent the order in which the image is processed. Each image operation node corresponds to an image processing operator used to implement the image processing operation. These image processing operators can also be called vision operators.

[0101] In this case, the image processing operator corresponding to the image processing operation list defined in the first image processing sub-scheme can be determined and instantiated, and then each instantiated operator is called in sequence to process the operator input image according to the execution order of the image processing operator.

[0102] Among them, the operator input image processed by the first instantiated operator in the calling sequence is: the first image. As mentioned above, after the end effector moves to the photo-taking point, the robotic arm can send feedback information to the server. After receiving the above feedback information, the server can consider that the robotic arm is ready to execute the subsequent process, thereby controlling the first visual device to perform image acquisition and obtain the first image; the operator input image of the remaining instantiated operators except the first instantiated operator is: the image output after the image processing of the forward adjacent instantiated operator of the instantiated operator.

[0103] The operator input images of all instantiated operators except the first can also be called intermediate images. For example, the calling order of instantiated operators OP1, OP2, and OP3 is: OP1 → OP2 → OP3. OP1 is the first instantiated operator in the calling order, so the first image is the operator input image of OP1. OP1 is first called to process the first image, obtaining intermediate image I1, which is also the operator input image of OP2. OP2 is called to process intermediate image I1, obtaining intermediate image I2, which is also the operator input image of OP3. Finally, OP3 is called to process intermediate image I2 to obtain the final processing result.

[0104] The specific steps of operator instantiation are detailed in the subsequent embodiments and will not be described in detail here.

[0105] In another case, the image processing process is recorded in the form of programming statements.

[0106] In this case, the above-mentioned programming statements can be parsed to determine the first programming statement representing the image processing operator in the programming statements, and identify the image processing operator described in the determined first programming statement. At this time, since the first programming statement is used to describe the image processing operator, the execution order of the image processing operators corresponding to each first programming statement can be determined according to the arrangement order of each first programming statement; then, the image processing operator is instantiated, and each instantiated operator is called in sequence to process the operator input image according to the execution order of the image processing operator.

[0107] After processing the first image according to the image processing flow, the image coordinates of the task target in the first image can be determined; then, according to the set calibration results, the image coordinates of the task target in the first image can be converted into three-dimensional coordinates in the robot arm base coordinate system to obtain the target position of the task target in the real world.

[0108] Step S204: Control the end effector to move to the target position, and control the end effector to execute the action recorded in the first execution configuration sub-scheme.

[0109] As can be seen from the above, in the solution provided by the embodiment of the present application, the server is connected to the robotic arm and the visual device at the same time, so that the robotic arm and the visual device can be collaboratively controlled according to the visual processing solution. Compared with the visual industrial computer controlling the visual device alone and the robotic arm controller controlling the robotic arm alone, the above method eliminates the communication interaction between the visual industrial computer and the robotic arm controller, increases communication efficiency while reducing communication delay and improving response speed, that is, improving the real-time performance of controlling the robotic arm, thereby improving the task execution efficiency of the robotic arm, and performing excellently in tasks requiring high precision and high real-time performance.

[0110] In addition, in related technologies, professional engineers (robotic arm engineers and vision engineers) are required to set up corresponding processes or input instructions on the robotic arm controller side and the visual industrial computer side respectively according to the specific process of the task to be performed, so that the robotic arm and camera can cooperate smoothly to perform the task. In the solution provided by the embodiment of the present application, the server has multiple sets of visual processing solutions registered. Therefore, in response to the task start instruction, the server can directly control the robotic arm and visual equipment to perform the task according to the visual processing solution to be executed, without the need for professional engineers to set up processes, which reduces the difficulty of controlling the robotic arm and improves the control efficiency.

[0111] Furthermore, in the solution provided in the embodiment of the present application, a server is used to control the connected robotic arms and visual devices, thereby realizing the deep integration of visual control and motion control. Compared with the visual industrial computer separately controlling the visual devices and the robotic arm controller separately controlling the robotic arms, the system architecture is simplified, which not only reduces the hardware cost, but also improves the stability of the system and reduces the maintenance cost of the system.

[0112] In addition, in the related technology, when controlling the robotic arm, the visual industrial computer is required to send posture information to the robotic arm controller, and the robotic arm controller then controls the movement of the robotic arm based on the posture information. The interaction process is relatively complex and involves many system variables. In the solution provided in the embodiment of the present application, the server is connected to the robotic arm and the visual device, so that the robotic arm and the visual device can be conveniently controlled in a coordinated manner. After determining and identifying the target position in the image captured by the visual device, a system variable can be used to control the robotic arm to perform the action, thereby simplifying the interaction process.

[0113] In one embodiment of the present application, the server can save the images captured by the image acquisition device during the execution of the visual processing solution in response to an execution result save instruction triggered by a staff member. The images can include images in which target recognition is successful and no errors are reported (pass images) as well as images in which target recognition fails and an error is reported (ng images).

[0114] In this way, after browsing the above images, the staff can evaluate the task execution of the robotic arm based on the image content. For example, based on the image captured after the end effector performs the operation of placing an object, they can determine whether the placement position of the object is accurate; furthermore, they can intervene in time when the task execution fails or an error occurs.

[0115] Among them, the server can also send the various images captured by the image acquisition device during the execution of the above-mentioned visual processing solution to the client, so that the client can save the above-mentioned images and display the above-mentioned images in the interactive interface in response to the viewing operation of the staff.

[0116] exist Figure 2 Based on the illustrated embodiment, the visual processing solution may also include a visual calibration sub-solution. In this case, before the server controls the robotic arm and visual device to perform tasks according to the execution configuration sub-solution, it may also first perform visual calibration on the robotic arm and visual device based on the visual calibration sub-solution. In view of the above, the embodiments of the present application provide a visual calibration method.

[0117] See also Figure 3 , is a flow chart of a visual calibration method provided in an embodiment of the present application, the method comprising the following steps S301 to S303:

[0118] Step S301: Generate a calibration path based on the positions of calibration points recorded in the first visual calibration sub-scheme included in the first visual processing scheme.

[0119] The position of the above-mentioned calibration point is the spatial position of the index fixed point in the robot arm base coordinate system, which can be obtained through teaching by the staff.

[0120] Specifically, the calibration path may be generated in the following manner.

[0121] In one case, the visual calibration sub-scheme records: the first position of the calibration center point, the second position of the calibration corner points, the rotation angle of the rotation calibration points, and the calibration point distribution information. The calibration center point is: the calibration point located at the center of the calibration area covered by the calibration path to be generated, and the calibration corner points are: the calibration points located at the corners of the calibration area. If the positions of the calibration center point and calibration corner points are known, the positions of the remaining calibration points in the calibration path can be calculated according to the following steps A-B to obtain the complete calibration path:

[0122] Step A: Based on the first position, the second position and the calibration point distribution information and the number of calibration points recorded in the calibration point distribution information, determine the third position of the remaining translation calibration points (hereinafter referred to as the first calibration point for ease of expression) and the fourth position of the rotation calibration point.

[0123] Among them, the number of calibration points records the total number of translation calibration points, which can be 9. The translation calibration points include: calibration center point, calibration corner point and first calibration point; the above-mentioned calibration center point and calibration corner point are translation calibration points. Based on the total number of translation calibration points recorded in the number of calibration points, the number of remaining translation calibration points (first calibration points) required for calibration can be determined; the number of calibration points can also record the number of rotation calibration points, which can be 1.

[0124] Specifically, based on the distance between the first position and the second position, and in combination with the calibration point distribution shape in the calibration point distribution information, the positions of the first calibration point and the rotation calibration point can be determined. Figure 4 Taking the calibration point distribution shape as a square as an example, the methods of determining the position of the first calibration point and the rotation calibration point are introduced respectively by giving examples, wherein, Figure 4 The circles in the figure represent the calibration points, and calibration point 1 and calibration point 2 are the calibration center point and calibration corner point respectively.

[0125] For the first calibration point:

[0126] First, calculate the direction vector S1 from calibration point 1 to calibration point 2. Given that calibration point 1 is the calibration center point and the calibration point distribution is a square, S1 is half the diagonal of the square. Rotate S1 90° to obtain vector S2. Calculate the difference between vector S2 and the calibration center point to obtain the position of the second calibration corner point (calibration point 4). Similarly, rotate S1 180° and 270° to obtain the positions of the third and fourth calibration corner points (calibration point 6 and calibration point 8). Next, calculate the midpoint coordinates between each pair of calibration corner points to obtain the positions of the four calibration points (calibration point 3, calibration point 5, calibration point 7, and calibration point 9). At this point, the positions of all nine translation calibration points are determined. Figure 4 shows the positions of each translation calibration point, indicating that calibration points 1 to 9 are translation calibration points.

[0127] For rotated calibration points:

[0128] The calibration point distribution information may record the position of the rotation calibration point in the calibration point distribution shape. For example, if the position of the rotation calibration point in the calibration point distribution shape recorded in the calibration point distribution information is the calibration center point, the position of the calibration center point may be directly used as the position of the rotation calibration point, i.e. Figure 4 The calibration point 1 in is the rotation calibration point. It can be seen that calibration point 1 is both a translation calibration point and a rotation calibration point.

[0129] Step B: Generate a calibration path including translation calibration points and rotation calibration points according to the traversal order information recorded in the calibration point distribution information, and generate control information for controlling the rotation angle of the end effector at the rotation calibration point based on the rotation angle.

[0130] The calibration point distribution information records the distribution shape of the calibration points and can also record the traversal order of each calibration point in the calibration point distribution shape. Therefore, the direction information between each translation calibration point and rotation calibration point can be generated according to the traversal order to obtain the calibration path.

[0131] Continuing the example above where the calibration points are distributed in a square shape, the generated path is as follows Figure 4 As shown, Figure 4 In the figure, the arrows starting from the translation calibration point indicate the traversal order between calibration points, and the arrows starting from the rotation calibration point indicate the corresponding rotation angle of the rotation calibration point. For calibration point 1, it is both a translation calibration point and a rotation calibration point. Among the three arrows starting from calibration point 1, the arrow pointing to the upper right indicates the traversal order, and the two arrows pointing to the upper left and lower left indicate the two rotation angles corresponding to calibration point 1.

[0132] The control information is used to control the rotation angle of the end effector of the robot arm at the rotation calibration point. After the control information is generated, it can be added to the calibration path as the attribute information of the rotation point, or it can be stored separately.

[0133] In this embodiment, the electronic device can automatically generate a calibration path based on the calibration center point and calibration corner points recorded in the visual calibration sub-scheme. Based on the calibration path, the device controls the movement of the robotic arm and the image acquisition of the visual device to complete the visual calibration. This means that the operator only needs to teach the calibration center point and calibration corner points, and the electronic device automatically generates the calibration path and completes the calibration steps. The calibration process does not require a calibration plate, simplifying the visual calibration process and improving its efficiency.

[0134] In another case, the visual calibration sub-scheme records the positions and traversal order information of all calibration points required for calibration, so that a calibration path can be generated based on the above complete information.

[0135] Step S302: Control the movement of the end effector according to the calibration path. In response to the end effector moving to the position of the calibration point included in the calibration path, control the first visual device configured in the first device management sub-scheme to perform image acquisition to obtain a second image.

[0136] Among them, after the end effector moves to the position of the calibration point included in the calibration path, the robotic arm can send feedback information to the server. After receiving the above feedback information, the server can determine that the end effector moves to the calibration point included in the calibration path, so as to control the first visual device to perform image acquisition and obtain the second image.

[0137] Specifically, when the calibration points include translation calibration points and rotation calibration points, the server controls the first visual device to perform image acquisition in the following situations:

[0138] If it is determined that the end effector moves to the position of the translation calibration point included in the calibration path, the first visual device to be calibrated configured in the first device management sub-scheme is directly controlled to perform image acquisition; if it is determined that the end effector moves to the position of the rotation calibration point included in the calibration path, the end effector is controlled to rotate a first angle, and then the first visual device is controlled to perform image acquisition, and the above-mentioned first angle is the rotation angle indicated by the above-mentioned control information.

[0139] In this embodiment, the calibration path includes two types of calibration points: translation calibration points and rotation calibration points. For translation calibration points, after the server determines that the end effector has moved to the position of each translation calibration point, it controls the first visual device to capture images. For rotation calibration points, when the server determines that the end effector has moved to the position of the rotation calibration point, it first controls the end effector to rotate based on the control information, and then controls the visual device to capture images. It can be seen that the trajectory of the end effector along the calibration path includes both translation and rotation trajectories. In this way, the first visual device can capture a second image of the end effector in multiple positions, thereby improving the accuracy of visual calibration based on image content.

[0140] Step S303: After determining the calibration points included in the calibration path traversed by the end effector, the robot arm and the first vision device are visually calibrated based on the spatial position and image position of the feature point of the set marker in the acquired second image.

[0141] In this step, visual calibration is performed to obtain a conversion relationship between the robot arm base coordinate system and the image coordinate system of the image captured by the first visual device.

[0142] The aforementioned robot base coordinate system is a three-dimensional coordinate system established based on the actual position of the robot. For example, it can be a coordinate system fixed to the base of the robot, with a point on the base as the origin. The coordinate axis directions follow certain standard conventions, such as the X axis pointing horizontally to the right, the Y axis pointing horizontally forward, and the Z axis pointing vertically upward.

[0143] The following is an introduction to the above markers:

[0144] The above-mentioned marker can be a set object with obvious features and easy-to-extract feature points, and the marker moves as the end effector moves.

[0145] In one case, the marker can be an object fixed to or grasped by the end effector, such as a rectangular plate or a reflective ball. In this case, feature points of the object can be extracted, such as the center or corner of the rectangular plate as the feature point of the rectangular plate, or the center of the reflective ball as the feature point of the reflective ball.

[0146] Alternatively, if the end effector itself has distinct features, the marker can be the end effector itself. In this case, feature points of the end effector can be extracted. For example, if the end effector is a spray gun, feature points of a specific location on the gun (such as the nozzle) can be extracted.

[0147] In this embodiment, the first visual device can be fixedly mounted in the working environment, with the marker within the field of view of the first visual device. Thus, the second image captured by the first visual device includes the marker. Because the marker moves with the end effector, as the end effector moves along the calibration path to the first position of each calibration point, multiple second images can be captured that include the marker in different positions. The positions of the feature points of the marker extracted from these multiple second images will also be different.

[0148] The following describes a method for determining the image position and spatial position of the feature points of the marker in each calibration image.

[0149] Since the marker moves with the end effector, multiple calibration images containing different marker positions are captured as the end effector moves along the calibration path to the locations of the various calibration points. Using a pre-determined feature point extraction algorithm, feature points of the markers at different locations are extracted from each calibration image, and the image positions of the extracted feature points are determined. Furthermore, since the positions of the various calibration points in the manipulator's base coordinate system are known, the spatial positions of the marker's feature points in the calibration images corresponding to each calibration point in the manipulator's base coordinate system can be determined based on the relative positional relationships between the pre-measured feature points and the calibration points.

[0150] After obtaining the image position and spatial position of the feature points in each calibration image, visual calibration can be performed. Visual calibration is performed to obtain the conversion relationship between the robot base coordinate system and the image coordinate system of the image captured by the camera.

[0151] Specifically, the intrinsic parameter matrix of the camera can be obtained first. The above intrinsic parameter matrix is used to describe the geometric characteristics inside the camera, is related to the hardware parameters, can be obtained according to the factory parameters of the camera, and is used to project the points in the camera coordinate system to the image plane. Based on the intrinsic parameter matrix, the first transformation relationship between the image coordinate system and the camera coordinate system can be obtained; then, based on the spatial position and image position of each feature point, the extrinsic parameter matrix of the camera is solved using algorithms such as the Perspective-n-Point (PnP) algorithm. The above extrinsic parameter matrix is also the second transformation relationship between the camera coordinate system and the robotic arm base space coordinate system; after obtaining the above first transformation relationship and the second transformation relationship, the transformation relationship between the robotic arm base coordinate system and the image coordinate system can be obtained through the image coordinate system → camera coordinate system → robotic arm base coordinate system.

[0152] In one embodiment of the present application, when the calibration path includes both translation calibration points and rotation calibration points, the conversion relationship between the robot arm base coordinate system and the image coordinate system can be calculated based on the spatial position and image position of the feature points in the calibration image captured by the camera at each translation calibration point using the above method; then, the conversion relationship between the robot arm base coordinate system and the image coordinate system is iteratively optimized using the spatial position and image position of the feature points in the calibration image captured by the camera after the end effector rotates at the rotation calibration point.

[0153] In one embodiment of the present application, in addition to the above-mentioned calibration method, a combination of multiple calibration methods and installation methods is also supported. The staff can configure different calibration methods according to current actual needs and different camera installation methods, such as calibration plate calibration or the above-mentioned contour calibration, which is highly flexible.

[0154] In this embodiment, the visual processing solution also includes a visual calibration sub-solution. This allows the server to calibrate the robotic arm and visual device based on the visual calibration sub-solution before controlling them to perform a task according to the execution configuration sub-solution. This allows the server to obtain the latest calibration results before executing the task. This allows subsequent operations to be performed based on the latest calibration results, further improving the accuracy of visual processing and, consequently, the efficiency of the robotic arm's task execution.

[0155] In one embodiment of the present application, the server may further execute a visual calibration sub-scheme in the visual processing scheme in response to a staff member triggering a calibration instruction on the client.

[0156] Specifically, when the staff has a calibration requirement, they can select the visual calibration sub-scheme in the client's interactive interface and click the "one-click calibration" button, so that the server can calibrate the visual calibration sub-scheme selected by the staff according to the Figure 3The steps shown automatically generate a calibration path and perform visual calibration.

[0157] In one embodiment of the present application, the server controls the robotic arm and the image acquisition device to complete the above-mentioned calibration process. After obtaining the calibration relationship, the server can send the calibration relationship and the translation error and rotation error generated during the calibration process to the client deployed in the aforementioned teaching pendant. The calibration relationship can be sent in the form of a homogeneous transformation matrix (HomogeneousTransformationMatrix); the above-mentioned translation error and rotation error can be obtained by substituting the image position of each feature point into the calibration relationship and comparing the obtained result with the spatial position of each feature point.

[0158] In this way, the staff can browse the calibration relationship and the above-mentioned calibration error obtained from this calibration on the client, and evaluate the calibration results. If the calibration results are considered to be highly accurate, they can be applied directly; otherwise, the electronic device can be re-calibrated by changing the calibration path, changing the marker, changing the image processing algorithm, etc. to obtain more accurate calibration results.

[0159] In one embodiment of the present application, after obtaining the calibration results, if the accuracy of the calibration results is low, the server can obtain the recognition result evaluation information of each calibration image; then, based on the recognition result evaluation information, the calibration path and / or the visual processing parameters used when extracting feature points can be adjusted.

[0160] The above-mentioned recognition result evaluation information may include the completeness of the marker in the second image, the accuracy of the feature points extracted from the second image, etc., which are respectively introduced below with examples.

[0161] For example, if it is determined that the markers in the second image corresponding to certain calibration points are incomplete, this indicates that the end effector moved the marker beyond the field of view of the image acquisition device when moving to certain calibration points. In this case, the calibration path can be changed so that when the end effector moves the marker to each calibration point included in the new calibration path, the marker is within the field of view of the image acquisition device. This ensures that the image corresponding to each calibration point contains a complete marker, which can improve the accuracy of subsequent feature point extraction and, therefore, the accuracy of calibration.

[0162] For example, if it is determined that the position error of the feature points in the image recognition results corresponding to certain calibration points is large, that is, a large error occurs in the stage of extracting the feature points of the marker, in this case, the maximum number of matches required for extracting image feature points, the number of calipers (used to scan the area of interest of edge points), edge polarity and other parameters can be adjusted so that the feature points can be identified more accurately based on the adjusted parameters.

[0163] In one embodiment of the present application, the server can send a second image captured by the visual device when the end effector moves to each calibration point in the calibration path, as well as the recognition results of the feature points of the marker in the image, to the client. The client can display the second image and recognition results so that the staff can determine whether the calibration process is accurate based on the displayed second image and recognition results. In this way, if the staff believes that the above recognition results are inaccurate, they can promptly adjust the calibration path and / or adjust the visual processing parameters used when extracting feature points to obtain a more accurate calibration result.

[0164] Next, pass Figure 5 , gives an intuitive introduction to the overall process of visual calibration.

[0165] Step S501: Setting the calibration center point.

[0166] Step S502: setting calibration corner points.

[0167] Step S503: setting the rotation angle.

[0168] Step S504: instructing to generate a calibration path.

[0169] In the above steps, the staff configures the calibration center point, calibration corner point and rotation angle of the rotation calibration point in the robotic arm module on the server, and instructs the robotic arm module to generate a calibration path.

[0170] Step S505: The robotic arm module controls the end effector to move to the calibration point.

[0171] The robotic arm module in the server generates a calibration path based on the staff's configuration, and controls the end effector of the robotic arm to traverse each calibration point in the calibration path in sequence.

[0172] Step S506: The robotic arm module sends the position and posture to the vision module.

[0173] After the robot arm module determines that the end effector has moved to the calibration point, it sends the current posture of the robot arm to the vision module in the server.

[0174] Step S507: The vision module turns on the light source.

[0175] After receiving the current posture, the vision module controls the light source device to turn on the lighting.

[0176] S508: The vision module controls the vision device to collect images.

[0177] After the vision module determines that the lighting is turned on, it controls the camera to obtain the stream, that is, to collect images, and record the current posture and the position of the feature points of the markers in the image.

[0178] Step S509: The vision module turns off the light source.

[0179] After the visual module completes the recording, it turns off the light source to confirm that the calibration collection of a calibration point is completed.

[0180] Repeating the above steps S505 to S509 can complete the calibration collection of each calibration point in sequence.

[0181] Step S510: The vision module outputs the calibration matrix and error.

[0182] After determining to traverse each calibration point, the vision module can perform visual calibration based on the collected data, output the calibration matrix and calculate the calibration error, and synchronize the above calibration results and calibration error to the client.

[0183] Step S511: Feedback application results.

[0184] The staff can confirm whether to apply the calibration result based on the calibration result and calibration error displayed by the client. If applicable, the current calibration result can be configured into the visual calibration sub-scheme; if not, the server can be instructed to recalibrate.

[0185] The following is about the above Figure 2 The instantiation operator mentioned in step S203 of the embodiment shown is introduced. In step S203, the first image is processed according to the image processing flow recorded in the first image processing sub-scheme included in the first visual processing scheme, which can be implemented by following steps C to F:

[0186] Step C: Identify an image processing operator for implementing the image processing operation represented by each first image operation node.

[0187] The first image operation node is an image operation node in a first image processing operation linked list, and the first image processing operation linked list is: an image processing operation linked list defined in a first image processing sub-scheme included in the first visual processing scheme.

[0188] Specifically, the image processing operation corresponding to the first image operation node may be determined according to the set correspondence between the image operation node and the image processing operation, and then the image processing operator for implementing the above image processing operation may be determined.

[0189] It should be noted that an image operation node represents an image processing operation; an image processing operator is also used to implement an image processing operation.

[0190] Step D: Load the identified image processing operator from pre-packaged image processing operators, instantiate the loaded image processing operator, and obtain an instantiated operator.

[0191] Pre-packaged image processing operators may include image format conversion operators, image preprocessing operators, template matching operators, circle finding operators, and line detection operators. The above-mentioned image preprocessing operators may include image enhancement operators, image binarization operators, image morphological transformation operators, etc., which are not limited in the embodiments of the present application.

[0192] Specifically, an operator resource can be created and registered for each first image operation node in the first image processing operation linked list, and the operator function of the image processing operator corresponding to each first image operation node can be identified; then, steps such as creating an operator tool, creating resource memory, registering the operator, and binding a callback function are executed to instantiate the above-mentioned image processing operator to obtain an instantiated operator.

[0193] Step E: Determine the calling order of the instantiated operators according to the pointer directions between the image operation nodes in the image processing operation linked list.

[0194] For example, if the pointer direction between image operation node P1 and image operation node P2 is from P1 to P2 (ie, P1→P2), the calling order between the instantiation operators corresponding to P1 and P2 is: call P1 first, then call P2.

[0195] Step F: Call each instantiated operator in turn to process the operator input image in the calling order.

[0196] In this embodiment, the image processing sub-scheme defines a linked list of image processing operations. By parsing this linked list, the image processing operators and their execution order can be easily determined. By loading and instantiating the operators, the image processing flow can be generated. This eliminates the need to store all image processing operators required for the flow within the image processing sub-scheme, reducing the storage space required by the image processing sub-scheme.

[0197] In one embodiment of the present application, the image processing sub-scheme may be configured with: Region of Interest (ROI) information and / or ROI inheritance information of the image operation node. In this case, the above step F may be implemented using the following steps F1-F2:

[0198] Step F1: If the ROI inheritance information configured by the image operation node corresponding to the instantiation operator indicates an inherited ROI, the first ROI is used as the target ROI of the instantiation operator; if the ROI inheritance information configured by the image operation node corresponding to the instantiation operator indicates a non-inherited ROI, the second ROI is used as the target ROI of the instantiation operator.

[0199] Step F2: Call each instantiation operator in sequence according to the calling order, and process the target ROI corresponding to each instantiation operator in the acquired image.

[0200] The first ROI is: the ROI output by the image operation node corresponding to the forward adjacent instantiation operator of the instantiation operator, and the second ROI is: the ROI configured by the image operation node corresponding to the instantiation operator, or the ROI obtained by correcting the configured ROI.

[0201] That is, for the instantiation operator corresponding to the image operation node set to inherit ROI, the ROI output by its forward adjacent instantiation operator can be used as the target ROI; for the instantiation operator corresponding to the image operation node set not to inherit ROI, the ROI configured by itself or the ROI obtained by modifying the configured ROI can be used as the target ROI.

[0202] It can be seen that there is one way to determine the target ROI for image operation nodes that have inherited ROIs set, and two ways to determine the target ROI for image operation nodes that do not have inherited ROIs set. In other words, there are three parallel methods in total, which are introduced below:

[0203] Method 1: Set ROI directly:

[0204] Directly use the ROI configured by the image operation node as the target ROI of the instantiated operator.

[0205] Method 2, inherit ROI:

[0206] The ROI output by the image operation node's forward adjacent image operation node is used as the target ROI of the instantiation operator.

[0207] Method 3: Correct ROI:

[0208] Specifically, when the image processing sub-scheme is configured with ROI correction information of the image operation node, when determining the target ROI of the instantiated operator, if the ROI correction information configured by the image operation node corresponding to the instantiated operator represents the corrected ROI, then the second ROI configured by the instantiated operator itself can be corrected based on the first feature point to obtain the target ROI of the instantiated operator.

[0209] The first feature point is a feature point of an object of interest output by an image operation node corresponding to a forward adjacent instantiation operator of the instantiation operator.

[0210] That is, for the instantiation operator, the second ROI set by itself can be adaptively modified according to the recognition result of the target of interest obtained by the previous instantiation operator to improve the accuracy of the ROI.

[0211] For example, instantiation operator OP1 is a positioning operator, and OP1's forward instantiation operator OP2 is also a positioning operator. The ROI of OP1 is the ROI set by the operator, and its accuracy is limited due to the dynamic motion of the robotic arm. After OP2 outputs the first feature point of the object of interest, the position of the first feature point reflects the latest position of the object of interest in the image. Therefore, the position of the ROI set by the operator for OP1 can be corrected based on the first feature point, resulting in a more accurate ROI.

[0212] Specifically, the position of the second ROI can be adjusted based on the relative positional relationship between the first feature points and the second ROI, using the set correction rules. For example, if the first feature points are mostly located above the second ROI, the correction rules can be used to move the second ROI upward to obtain the target ROI.

[0213] See also Figure 6 For the image operation node, after loading the current node image resource, the current image operation node can determine the ROI inheritance status of its own node, that is, determine whether to input the ROI. If not, the function is executed directly. If yes, it indicates that the ROI is inherited, then the ROI is input, that is, the ROI of the previous image processing operation node is loaded into the current image operation node, and the function is executed, that is, the image processing operation corresponding to the current image operation node is executed; then determine whether the function is executed. If yes, the process ends; if not, the current image operation node outputs the image to the next node, that is, the processed image is input to the next image operation node; then, according to the ROI inheritance status of the next image operation node, determine whether to output the ROI. If the ROI inheritance status of the next image operation node indicates that the ROI is inherited, the ROI is output to the next node, otherwise the ROI is not output, and the process returns to the step of loading the current node image resource.

[0214] It can be seen that in this embodiment, each image operation node can process the ROI in the image in a targeted manner, thereby reducing the computing resources consumed by image processing; furthermore, by setting the ROI and ROI inheritance status information for each image operation node, the ROI that each image operation node needs to pay attention to can be flexibly set, thereby improving the flexibility of image processing.

[0215] In one embodiment of the present application, the image processing sub-scheme in the visual processing scheme registered by the server is stored in the form of serialized data, which can reduce the storage space required to store the image processing sub-scheme and save storage resources.

[0216] In one embodiment of the present application, in order to enable the image processing sub-scheme to support scheme replication (including copying to other visual processing schemes or the contour calibration scheme of the present visual processing scheme), and to enable the server to efficiently import and export the configuration parameters of the visual operators corresponding to the image operation nodes included in the current image processing sub-scheme (for the sake of ease of description, the visual operators corresponding to the image operation nodes included in the image configuration sub-scheme are referred to as: visual operators included in the image configuration sub-scheme), the embodiment of the present application designs operator function nodes, and the above-mentioned operator function nodes are introduced in detail below.

[0217] An operator function node corresponds one-to-one with a visual operator and can be understood as a function module. By calling an operator function node, the server can obtain the configuration parameters of the visual operator and set the configuration parameters to the visual operator. The following describes when the server calls the operator function node to obtain and set these configuration parameters.

[0218] Regarding the acquisition of configuration parameters, after configuring the visual operator in the image configuration sub-scheme, the server can call the operator function node at any time to obtain the configuration parameters of the visual operator.

[0219] Regarding the setting of configuration parameters, the server can call the operator function node to set the above configuration parameters before the image processing sub-scheme is executed. For example, the server can call the operator function node to set the configuration parameters of any visual processing operator included in the image processing sub-scheme when the image processing sub-scheme is not in the execution state; or, the server can also call the operator function node to set the configuration parameters of the visual processing operator in the image processing sub-scheme that has not been instantiated and is not in the running state when the image processing sub-scheme is in the execution state.

[0220] Operator function nodes define feature interfaces, including one for obtaining and setting the configuration parameters of visual operators. When other software modules running on the electronic device or external devices invoke these feature interfaces, the operator function node can retrieve or set the configuration parameters of the visual operators based on the specific functionality of the invoked feature interface.

[0221] In addition, the operator function node also has serialization and deserialization functions. Among them, the serialization function means that the operator function node serializes the configuration parameters of the visual operator to obtain serialized data, so that other software modules running in the electronic device or external software development kits (SDKs), teaching devices, etc. can batch obtain the above serialized data. The data format of the above serialized data can be Json format, and the serialized data can also be called Json configuration files; the deserialization function means that the operator function node can respond to the call of the server and deserialize the input serialized data to parse out the configuration data of the visual operator, such as parsing the input Json configuration file into the configuration data of the visual operator.

[0222] Based on the above functions provided by the operator function node, the embodiment of the present application can realize quick import and export of configuration parameters for a single visual operator, as well as quick import and export of configuration parameters for all visual operators included in the entire image processing sub-scheme.

[0223] The following combination Figure 7 , provides an intuitive description of the process of importing and exporting the configuration parameters of a single vision operator and the process of importing and exporting all the configuration parameters included in the image processing sub-scheme.

[0224] Depend on Figure 7 As can be seen, the electronic device defines visual operators and runs various operator function nodes 1-N, each of which is used to manage a visual operator. Specifically, the operator function node manages the visual operator by: parsing the serialized data of the visual operator into configuration data through the deserialization function, and converting the configuration data of the visual operator into serialized data through the serialization function. The arrows between the server, the operator function node, and the visual operator represent calls, data input, or data output.

[0225] First, the process of importing and exporting the configuration parameters of a single visual operator is introduced.

[0226] See also Figure 7 , assuming that the serialized data of the visual operator V1 is C1, and the operator function node used to manage V1 is operator function node 2.

[0227] From the perspective of importing configuration parameters:

[0228] After receiving C1, the server parses C1 and determines that the visual operator corresponding to C1 is V1. It also determines that the operator function node that manages V1 is Operator Function Node 2. The server then calls Operator Function Node 2 and inputs C1 into Operator Function Node 2. Operator Function Node 2 then performs a deserialization operation on C1, obtains the configuration parameters of V1, and applies them to V1. This completes the import of configuration parameters for a single visual operator.

[0229] From the perspective of exporting configuration parameters:

[0230] The server calls the operator function node 2 corresponding to V1. In response to the server's call, the operator function node 2 serializes the configuration parameters of V1 and outputs the serialized data C1, which the server can then obtain. This completes the export of the configuration parameters for a single visual operator.

[0231] The methods for importing and exporting configuration parameters for the remaining operator function nodes are similar to those for operator function node 2 and are not described here.

[0232] It can be seen that by implementing the serialization and deserialization of a single visual operator configuration parameter, the server can conveniently obtain and set the configuration parameters of the visual operator, and realize the quick import and export of the visual operator configuration parameters.

[0233] Then the process of importing and exporting all configuration parameters included in the image processing sub-scheme is introduced.

[0234] Continue to see Figure 7 , assuming that all visual operators included in the image processing sub-scheme X are V1-VN, the serialized data of V1-VN are C1-CN respectively, and the operator function nodes used to manage V1-VN are operator function nodes 1 to N respectively,

[0235] From the perspective of importing configuration parameters:

[0236] The server can identify all visual operators V1-VN included in image processing sub-solution X and determine the operator function nodes that manage V1-VN as operator function nodes 1-N. Consequently, the server can call operator function nodes 1-N and send C1-CN to them respectively. Operator function nodes 1-N deserialize the input data to obtain the configuration parameters of V1-VN, that is, the configuration parameters of all visual operators included in image processing sub-solution X, and apply the configuration parameters to visual operators V1-VN. This completes the import of all configuration parameters included in the image processing sub-solution.

[0237] From the perspective of exporting configuration parameters:

[0238] The server calls the operator function nodes 1 through N corresponding to V1 through V2, respectively. In response to the server's instructions, operator function nodes 1 through N serialize the configuration parameters of their managed visual operators and output the serialized data C1 through CN for V1 through VN. The server then obtains the serialized data C1 through CN, which represents the serialized data for all visual operators included in image processing sub-solution X. This completes the export of all configuration parameters for the image processing sub-solution.

[0239] It can be seen that by realizing the serialization and deserialization of the configuration parameters of the visual operators included in the entire image sub-scheme, the server can conveniently obtain and configure the configuration parameters of the entire image sub-scheme, thereby realizing the quick import, export, backup, copy and other functions of the image sub-scheme.

[0240] As described above, the server can be deployed in a control device or in a robotic arm. In one embodiment of the present application, when the server is deployed in a control device, if the device configuration sub-scheme includes input and output (IO) interface configuration information, the server can control the robotic arm to establish a one-way IO interface subscription relationship with the server, so that the robotic arm synchronizes the IO interface configuration with the server.

[0241] The embodiment of the present application does not limit the method of establishing the above-mentioned unidirectional IO interface subscription relationship, which is introduced below by way of example.

[0242] like Figure 8 As shown, the steps of establishing a one-way IO interface subscription relationship may include: the robot arm publishing the IO configuration and the server subscribing to the IO configuration. Specifically, the electronic device, as the subscriber, can subscribe to messages on a set topic. The robot arm, as the publisher, sends the IO interface information it supports binding to the network. Through network forwarding, the electronic device recognizes that the message belongs to the topic it has subscribed to and receives the message, thus implementing IO interface subscription.

[0243] If the device configuration sub-scheme includes IO signal configuration information, the server can control the robotic arm to establish a bidirectional IO signal subscription relationship with the server, so that the IO signals between the robotic arm and the server are synchronized with each other.

[0244] The establishment of a bidirectional IO signal subscription relationship is similar to the above-mentioned unidirectional IO interface subscription relationship, such as Figure 8 As shown, the steps of establishing a bidirectional IO signal subscription relationship may include: the server publishes the IO level, the robotic arm subscribes to the IO level, the robotic arm publishes the IO level, and the server subscribes to the IO level.

[0245] Regarding the synchronization of the IO interface, after the robot arm is successfully initialized, it will actively push all currently configured IO information to the server. After the server subscribes to the configuration information, it will synchronize all current IO configurations and provide feedback to the client on the IO operations currently supported by the robot arm. These IO operations include actions that the end effector can perform, etc., so that the client can display these IO operations on the interactive interface, allowing staff to configure visual solutions based on these IO operations, such as selecting the end effector action in the process configuration sub-solution from the actions supported by the robot arm. Each time the robot arm adds, deletes, or modifies an IO, the IO configuration will be published, and the server will receive the IO configuration information in real time to complete the IO configuration synchronization.

[0246] At the same time, for the synchronization of IO levels, both the server and the robotic arm subscribe to and publish the relevant IO levels. In this way, the server can actively publish IO level information for IO level modification. After the robotic arm receives the IO level information, it completes the output level change. In addition, the robotic arm can also actively push the modified IO information when the level is modified, so that the server can synchronize the IO level.

[0247] In this way, the server can promptly learn about the IO interfaces supported by the robotic arm, and achieve IO interface synchronization between the robotic arm and the visual module in the server, so that a usable visual processing solution can be built based on the supported IO operations in the future; and the robotic arm and the server can synchronize IO signals in a timely manner, so that the server can promptly learn about the execution status of various IO operations performed by the robotic arm, thereby improving the accuracy of controlling the robotic arm.

[0248] In one embodiment of the present application, a new visual processing solution can also be created through the following steps.

[0249] Step G: In response to receiving device editing information for the visual processing scheme to be configured sent by the client, configure the device management sub-scheme in the visual processing scheme to be configured based on the device editing information; in response to receiving image processing flow editing information for the visual processing scheme to be configured sent by the client, configure the image processing sub-scheme in the visual processing scheme to be configured based on the image processing flow editing information; in response to receiving execution flow editing information for the visual processing scheme to be configured sent by the client, configure the execution configuration sub-scheme in the visual processing scheme to be configured based on the execution flow editing information; in response to receiving visual calibration editing information for the visual processing scheme to be configured sent by the client, configure the visual calibration sub-scheme in the visual processing scheme to be configured based on the visual calibration editing information.

[0250] When configuring the image processing sub-scheme:

[0251] If the image processing configuration method indicated by the first editing information is new scheme creation, a first image processing operation linked list is generated based on the image operation nodes and the execution order information of the image operation nodes included in the first editing information, and an image processing sub-scheme including the first image processing operation linked list is obtained as the image processing sub-scheme in the visual processing scheme to be configured, wherein the first editing information is: the image processing flow editing information for the visual processing scheme to be configured sent by the client.

[0252] If the image processing configuration method indicated by the first editing information is scheme copying, the second image processing operation linked list of the target image processing sub-scheme indicated by the first editing information is directly copied to obtain the image processing sub-scheme including the second image processing operation linked list as the image processing sub-scheme in the visual processing scheme to be configured.

[0253] In one embodiment of the present application, the server may further edit the configured image processing sub-scheme in response to receiving scheme editing information for an existing image processing sub-scheme sent by the client.

[0254] For example, if the above-mentioned scheme editing information is an addition or deletion operation of an image operation node, the image operation node indicated by the above-mentioned operation in the configured image processing sub-scheme can be added or deleted; if the above-mentioned scheme editing information is an addition or deletion operation of a connection relationship between image operation nodes, the connection relationship between two image operation nodes indicated by the above-mentioned operation in the configured image processing sub-scheme can be added or deleted.

[0255] When adding an image operation node, the image operation node can be inserted into the image processing sub-scheme header, tail, or a designated location based on the location information included in the scheme editing information. The location information may include the identifier of the preceding and / or following adjacent image operation nodes of the image operation node to be inserted.

[0256] As can be seen from the above, in this embodiment, the image processing sub-scheme is customized by the staff through the client, which is flexible and portable, allowing the staff to quickly and easily complete the construction of the image processing sub-scheme when dealing with different scenarios and tasks. Specifically, when configuring the image processing sub-scheme, new schemes can be created to meet the staff's personalized needs, or existing schemes can be copied, which reduces the difficulty of configuring the image processing sub-scheme and improves the configuration efficiency. It can be seen that through multiple scheme configuration methods, the flexibility of creating image processing sub-schemes is improved.

[0257] In one embodiment of the present application, the staff can customize the number of execution steps for the image processing sub-scheme. In this way, when executing the image processing sub-scheme, the image processing can be stopped after the image operation node with the number of steps set by the staff is executed, which makes it easier for the staff to debug each image operation node in the image processing sub-scheme.

[0258] When configuring the device management sub-scheme:

[0259] If the second editing information includes an IO interface subscription instruction, IO interface configuration information is generated for controlling the robotic arm to establish a one-way IO interface subscription relationship with the server, and the IO interface configuration information is recorded in the device management sub-scheme in the visual processing scheme to be configured.

[0260] If the second editing information includes an IO signal subscription instruction, IO signal configuration information is generated for controlling the robotic arm to establish a bidirectional IO signal subscription relationship with the server, and the IO signal configuration information is recorded in the device management sub-scheme in the visual processing scheme to be configured.

[0261] The above-mentioned IO interface configuration information and IO signal configuration information are respectively used to control the establishment of the above-mentioned unidirectional IO interface subscription relationship and bidirectional IO signal subscription relationship, which may specifically include IO interface information, subscription topic information, etc., which is not limited in this embodiment of the present application.

[0262] In this way, IO interface configuration information and IO signal configuration information can be generated in response to the staff's instructions, and the above information can be recorded in the device management sub-plan. Therefore, when the device management sub-plan is executed subsequently, a unidirectional IO interface subscription relationship and a bidirectional IO signal subscription relationship can be established based on the configuration information, which facilitates the synchronization of IO interfaces and IO signals between the server and the robotic arm, thereby improving the accuracy of control.

[0263] In one embodiment of the present application, when configuring a visual calibration sub-scheme, a calibration sub-scheme generated in other visual processing schemes can be directly copied.

[0264] Specifically, if the calibration configuration mode indicated by the visual calibration editing information is scheme copying, the visual calibration sub-scheme in the visual processing scheme indicated by the visual calibration editing information can be directly copied as the visual calibration sub-scheme in the visual processing scheme to be configured.

[0265] Step H: If it is determined based on the configuration result that the visual processing solution to be configured is completed, register the visual processing solution obtained after the configuration is completed.

[0266] In this embodiment, the server can configure and register a new visual processing solution in response to the configuration information sent by the client, so that staff can simply and conveniently register customized visual processing solutions on the server according to the actual needs of various application scenarios, thereby improving the flexibility and applicability of the solution.

[0267] The following is an example of an overall robotic arm control process provided in an embodiment of the present application.

[0268] For example, workers can configure photo points, intermediate points, and IO trigger actions in the execution configuration sub-scheme through the client. The IO trigger action can directly trigger the IO variables of the robotic arm, so that the robotic arm outputs IO control signals to external tools (such as end effectors) to control the external tools to perform corresponding actions; photo points and intermediate points can be customized, and each point can be bound to the image processing sub-scheme.

[0269] The following combination Figure 9 Provide an intuitive explanation of the overall control process of the robotic arm.

[0270] See also Figure 9 First, the server controls the end effector to move to the photo point according to the execution configuration sub-scheme in the visual processing scheme. After confirming that the end effector has moved to the photo point, the server can load the image processing sub-scheme bound to the photo point. During the execution of the image processing sub-scheme, the scheme is first loaded, that is, the device management sub-scheme is loaded. According to the device information recorded in the device management sub-scheme, the visual device is controlled to turn on the light source, take the flow, and turn off the light source in sequence. At this point, the server obtains the image captured by the visual device when the end effector moves to the photo point. It can execute the image processing process and obtain point information based on the processing results. The above point information is: the pose information of the task target.

[0271] Then, the server can send the target posture information to the robotic arm, so that the robotic arm controls the end effector to move to the middle point based on the above target posture information, and after determining that the end effector moves to the above middle point, controls the end effector to perform IO action.

[0272] Finally, the server determines whether the execution configuration is completed. If so, it determines that the execution process recorded in the execution configuration sub-scheme has been completed and ends the control process of this task. If not, it determines that the execution process recorded in the above execution configuration sub-scheme has not been completed and returns to the step of controlling the end effector to move to the photo point.

[0273] In this way, staff can build and apply specific execution processes for various tasks by customizing intermediate points, photo points and IO trigger actions, simplifying the collaborative process of the robotic arm and vision module. At the same time, the integration of image processing and motion control optimizes data transmission and collaborative communication delays, which can better meet the real-time requirements of the task.

[0274] Corresponding to the above-mentioned method for controlling a robotic arm applied to a server, the present invention also provides a method for controlling a robotic arm applied to a client.

[0275] See also Figure 10 , which is a flow chart of a second robotic arm control method provided in an embodiment of the present application, wherein the method is applied to a client and includes the following steps S1001 to S1003:

[0276] Step S1001: In response to the device editing operation for the visual processing solution to be configured in the first configuration interface, send device editing information to the server, so that the server configures the device management sub-solution in the visual processing solution to be configured based on the received device editing information.

[0277] Step S1002: In response to the image processing process editing operation for the visual processing scheme to be configured in the second configuration interface, the image processing process editing information is sent to the server, so that the server configures the image processing sub-scheme in the visual processing scheme to be configured based on the received image processing process editing information.

[0278] Step S1003: In response to the execution process editing operation for the visual processing scheme to be configured in the third configuration interface, the execution process editing information is sent to the server, so that the server configures the execution configuration sub-scheme in the visual processing scheme to be configured based on the received execution process editing information.

[0279] The above-mentioned device editing operation, image processing process editing operation and execution process editing operation can be triggered by the staff on the corresponding configuration interface.

[0280] It should be noted that staff can flexibly configure visual processing solutions on the client side and select the visual processing solutions to be executed. Each visual processing solution is completely independent and decoupled, and can be created, deleted, copied, backed up, restored, and other operations.

[0281] It can be seen that in the solution provided by the embodiment of the present application, the staff can perform visual control configuration and robotic arm motion control configuration in a visual manner on a client, thereby obtaining a customized visual processing solution. There is no need for professional engineers to perform corresponding process settings or command input on the robotic arm controller side and the visual industrial computer side according to the specific process of the task to be performed, so that the staff can flexibly and simply deal with complex production environments, thereby improving the configuration efficiency of the visual processing solution.

[0282] In addition, based on the integrated client, staff can flexibly and conveniently adjust the task execution strategy in different tasks and environments, making the solution more adaptable and able to quickly respond to environmental changes and complex scenario requirements.

[0283] Furthermore, in the solution provided in the embodiment of the present application, after the configuration of the visual processing solution is completed based on the client, the server can collaboratively control the robotic arm and the visual device according to the visual processing solution. Compared with the visual industrial computer controlling the visual device alone and the robotic arm controller controlling the robotic arm alone, the above method eliminates the communication interaction between the visual industrial computer and the robotic arm controller, increases the communication efficiency while reducing the communication delay, improves the real-time performance of controlling the robotic arm, and thus improves the task execution efficiency of the robotic arm.

[0284] In one embodiment of the present application, the client can also send visual calibration editing information to the server in response to the visual calibration configuration operation for the visual processing scheme to be configured in the fourth configuration interface, so that the server configures the visual calibration sub-scheme in the visual processing scheme to be configured based on the received visual calibration editing information.

[0285] Thus, the visual processing solution also includes a visual calibration sub-solution. Therefore, before the server controls the robotic arm and visual device to perform a task according to the execution configuration sub-solution, it can first calibrate the robotic arm and visual device based on the visual calibration sub-solution to obtain the latest calibration results before executing the task. Therefore, subsequent operations based on the latest calibration results can further improve the accuracy of visual processing and thus enhance the efficiency of the robotic arm's task execution.

[0286] The following is an introduction to a robotic arm control system provided in an embodiment of the present application.

[0287] See also Figure 11, a schematic diagram of a robotic arm control system provided in an embodiment of the present application, comprising: a robotic arm equipped with an integrated visual control system (corresponding to the aforementioned server), a teach pendant equipped with a client, a camera device, a light source device, and external tools. The teach pendant and camera device can be connected to the robotic arm via a network port, the external tool can be connected to the robotic arm via an IO interface, and the light source device can be connected to the robotic arm via a network port and / or an IO interface.

[0288] The integrated vision and control system includes a robotic arm master control system and a visual operator execution component, which communicate via shared memory. The master control system controls the robotic arm's movements, while the visual operator execution component calls visual operators to perform visual processing on images.

[0289] As can be seen, the robotic arm control system provided in the embodiments of the present application utilizes a modular design and an efficient communication protocol, making the system more scalable and maintainable, improving the overall performance and efficiency of the system and facilitating subsequent functional expansion and system optimization. Furthermore, the aforementioned integrated design and modular architecture reduce the need for independent hardware devices and lower the system's hardware costs. The overall system is more stable and flexible than traditional systems, reducing maintenance costs and troubleshooting difficulties.

[0290] Next, combine Figure 12a and Figure 12b , a more intuitive introduction to the advantages of the solution provided in the embodiments of the present application compared to the related technologies.

[0291] See first Figure 12a , which is a schematic diagram of a robotic arm control scenario in related technology.

[0292] It can be seen that in the relevant technology, the control process of the robotic arm requires the participation of robotic arm engineers and vision engineers. On the one hand, the robotic arm engineer needs to send the control instructions of the robotic arm to the robotic arm controller through the teach pendant, so that the robotic arm controller controls the robotic arm to perform corresponding actions according to the control instructions; on the other hand, the vision engineer needs to send the control instructions of the visual device to the vision controller through the vision client and the light source client. The vision controller synchronizes the control instructions to the light source controller, and then the vision controller and the light source controller control the camera and light source to perform corresponding actions according to the control instructions. Furthermore, information synchronization is required between the vision controller and the robotic arm controller to ensure the coordination of the robotic arm and the vision device.

[0293] See also Figure 12b , is a schematic diagram of a robotic arm control scenario provided in an embodiment of the present application.

[0294] As can be seen, in the solution provided by the embodiment of this application, the visual processing flow is abstracted into the configuration of four plug-in processes. The entire configuration only requires the participation of a vision engineer, and the above-mentioned vision engineer does not need to have professional robotic arm programming or vision programming knowledge. Specifically, the vision engineer can directly configure the visual processing solution in the integrated vision and control controller through the configuration interface provided by the teach pendant. After the configuration is completed, the integrated vision and control controller can control the robotic arm, camera, and light source to cooperate in accordance with the visual processing solution to complete the task.

[0295] By comparison, it can be seen that compared with related technologies, the solution provided by the embodiment of the present application integrates the visual processing module directly into the robotic arm, eliminating the independent visual controller in traditional systems, achieving a deep integration of vision and motion control, significantly reducing data transmission latency, and improving the real-time performance and efficiency of the system. Moreover, only one staff member is required to complete the control of the entire task process, which greatly simplifies the configuration process of the visual processing solution and reduces the complexity of system debugging and operation.

[0296] In summary, the solution provided in the embodiment of the present application realizes the integration of visual processing and robotic arm control, overcoming the problems of delay, lack of flexibility, high hardware dependence, etc. caused by the separation of visual processing and motion control in related technologies; and, the innovative modular design makes this solution perform better in complex tasks and changing environments, with obvious technical advantages, improving the real-time, adaptability, and scalability of the system, and reducing costs and maintenance difficulties.

[0297] Corresponding to the above-mentioned robotic arm control method applied to the server, an embodiment of the present application also provides a robotic arm control device applied to the server.

[0298] See also Figure 13 , which is a structural diagram of a first type of robotic arm control device provided in an embodiment of the present application. The device is applied to a server, which is connected to a robotic arm and a visual device. The robotic arm and the visual device are deployed in the same working environment. The visual device is used to collect images of task targets. The server is registered with multiple visual processing solutions, each of which includes: a device management sub-solution, an image processing sub-solution, and an execution configuration sub-solution. The device includes:

[0299] A solution loading module 1301 is configured to load a first visual processing solution to be executed from the plurality of visual processing solutions in response to a task start instruction;

[0300] A first image acquisition module 1302 is configured to control the end effector of the robotic arm to move to a position triggering image acquisition recorded in the first execution configuration sub-scheme included in the first visual processing scheme, and control the first visual device configured in the first device management sub-scheme included in the first visual processing scheme to perform image acquisition to obtain a first image;

[0301] An image processing module 1303 processes the first image according to the image processing flow recorded in the first image processing sub-scheme included in the first visual processing scheme to obtain a target position of the task target in the first image;

[0302] The action execution module 1304 is used to control the end effector to move to the target position and control the end effector to execute the action recorded in the first execution configuration sub-scheme.

[0303] As can be seen from the above, in the solution provided by the embodiment of the present application, the server is connected to the robotic arm and the visual device at the same time, so that the robotic arm and the visual device can be collaboratively controlled according to the visual processing solution. Compared with the visual industrial computer controlling the visual device alone and the robotic arm controller controlling the robotic arm alone, the above method eliminates the communication interaction between the visual industrial computer and the robotic arm controller, increases communication efficiency while reducing communication delay and improving response speed, that is, improving the real-time performance of controlling the robotic arm, thereby improving the task execution efficiency of the robotic arm, and performing excellently in tasks requiring high precision and high real-time performance.

[0304] In addition, in related technologies, professional engineers are required to set up corresponding processes or input instructions on the robot arm controller and the visual industrial computer according to the specific process of the task to be executed, so that the robot arm and camera can cooperate smoothly to perform the task. In the solution provided by the embodiment of the present application, the server registers multiple sets of visual processing solutions. Therefore, in response to the task start instruction, the server can directly control the robot arm and visual equipment to perform the task according to the visual processing solution to be executed, without the need for professional engineers to set up processes, which reduces the difficulty of controlling the robot arm and improves control efficiency.

[0305] Furthermore, in the solution provided in the embodiment of the present application, a server is used to control the connected robotic arms and visual devices, thereby realizing the deep integration of visual control and motion control. Compared with the visual industrial computer separately controlling the visual devices and the robotic arm controller separately controlling the robotic arms, the system architecture is simplified, which not only reduces the hardware cost, but also improves the stability of the system and reduces the maintenance cost of the system.

[0306] In one embodiment of the present application, the visual processing solution further includes: a visual calibration sub-solution, and the device further includes:

[0307] a calibration path generation module, configured to generate a calibration path based on the positions of the calibration points recorded in the first visual calibration sub-scheme included in the first visual processing scheme before the first image acquisition module 1302 is triggered;

[0308] a second image acquisition module, configured to control the movement of the end effector according to the calibration path, and in response to the end effector moving to the position of the calibration point included in the calibration path, control the first visual device configured in the first device management sub-scheme to perform image acquisition to obtain a second image;

[0309] A visual calibration module is used to determine that after the end effector traverses each calibration point included in the calibration path, the robotic arm and the first visual device are visually calibrated based on the spatial position and image position of the feature point of the set marker in the captured second image, wherein the spatial position is determined based on the position of the calibration point included in the calibration path.

[0310] In this embodiment, the visual processing solution also includes a visual calibration sub-solution. Thus, before the server controls the robotic arm and visual device to perform a task according to the execution configuration sub-solution, it can first perform visual calibration on the robotic arm and visual device based on the visual calibration sub-solution to obtain the latest calibration results before executing the task. Therefore, subsequent operations based on the latest calibration results can further improve the accuracy of visual processing, thereby increasing the efficiency of task execution for the robotic arm.

[0311] In one embodiment of the present application, the visual calibration sub-scheme records: a first position of a calibration center point, a second position of a calibration corner point, a rotation angle of a rotation calibration point, and calibration point distribution information. The calibration center point is a calibration point located at the center of a calibration area covered by a calibration path to be generated. The calibration corner point is a calibration point located at a corner of the calibration area. The calibration path generation module includes:

[0312] a calibration point determination submodule, configured to determine, based on the first position, the second position, and the calibration point distribution information and the number of calibration points recorded in the calibration point distribution information, a third position of the first calibration point other than the calibration center point and the calibration corner point, and a fourth position of the rotational calibration point for visual calibration;

[0313] The calibration path generation submodule is used to generate a calibration path including translation calibration points and rotation calibration points according to the traversal order information recorded in the calibration point distribution information, and generate control information for controlling the rotation angle of the end effector at the rotation calibration point based on the rotation angle, wherein the translation calibration point includes: the calibration center point, the calibration corner point and the first calibration point.

[0314] In this embodiment, the electronic device can automatically generate a calibration path based on the calibration center point and calibration corner points recorded in the visual representation sub-scheme. Based on the calibration path, it controls the movement of the robotic arm and the image acquisition of the visual device to complete the visual calibration. As can be seen, for the staff, they only need to teach the calibration center point and calibration corner points, and the electronic device can automatically generate the calibration path and complete the calibration steps. The calibration process does not require a calibration plate, which simplifies the visual calibration process and improves the efficiency of visual calibration.

[0315] In one embodiment of the present application, the second image acquisition module is specifically used to control the movement of the end effector according to the calibration path. If it is determined that the end effector moves to the position of the translation calibration point included in the calibration path, the first visual device configured in the first device management sub-scheme is controlled to perform image acquisition; if it is determined that the end effector moves to the position of the rotation calibration point included in the calibration path, the first visual device is controlled to perform image acquisition based on controlling the end effector to rotate a first angle, wherein the first angle is: the rotation angle indicated by the control information of the rotation calibration point.

[0316] In this embodiment, the calibration path includes two types of calibration points: translation calibration points and rotation calibration points. For translation calibration points, after the server determines that the end effector has moved to the position of each translation calibration point, it controls the first visual device to capture images. For rotation calibration points, when the server determines that the end effector has moved to the position of the rotation calibration point, it first controls the end effector to rotate based on the control information, and then controls the visual device to capture images. It can be seen that the trajectory of the end effector along the calibration path includes both translation and rotation trajectories. In this way, the first visual device can capture a second image of the end effector in multiple positions, thereby improving the accuracy of visual calibration based on image content.

[0317] In one embodiment of the present application, the image processing sub-scheme defines an image processing operation linked list, wherein the image processing operation linked list includes multiple image operation nodes, wherein the pointer directions between the image operation nodes represent the order in which the images are processed, and each image operation node represents an image processing operation. The image processing module 1303 includes:

[0318] an operator identification submodule, configured to identify an image processing operator for implementing the image processing operation represented by each first image operation node, wherein the first image operation node is an image operation node included in a first image processing operation linked list, and the first image processing operation linked list is an image processing operation linked list defined in a first image processing sub-scheme included in the first visual processing scheme;

[0319] An operator instantiation submodule is used to load the identified image processing operator from pre-packaged image processing operators, instantiate the loaded image processing operator, and obtain an instantiated operator;

[0320] An order determination submodule, configured to determine the calling order of the instantiated operators according to the pointer directions between the image operation nodes in the image processing operation linked list;

[0321] The image processing submodule is used to call each instantiated operator in sequence to process the operator input image in the calling order, wherein the operator input image of the first instantiated operator in the calling order is: the first image, and the operator input images of the remaining instantiated operators except the first instantiated operator are: the image output after the forward adjacent instantiated operator of the instantiated operator performs image processing, to obtain the target position of the task target in the first image.

[0322] In this embodiment, the image processing sub-scheme defines a linked list of image processing operations. By parsing this linked list, the image processing operators and their execution order can be easily determined. By loading and instantiating the operators, the image processing flow can be generated. This eliminates the need to store all image processing operators required for the flow within the image processing sub-scheme, reducing the storage space required by the image processing sub-scheme.

[0323] In one embodiment of the present application, the image processing sub-scheme is configured with: region of interest ROI information and / or ROI inheritance information of the image operation node,

[0324] The image processing submodule is specifically configured to determine the target ROI of the instantiated operator as a first ROI if the ROI inheritance information configured by the image operation node corresponding to the instantiated operator represents an inherited ROI, wherein the first ROI is: the ROI output by the image operation node corresponding to the forward adjacent instantiated operator of the instantiated operator; if the ROI inheritance information configured by the image operation node corresponding to the instantiated operator represents a non-inherited ROI, then determine the target ROI of the instantiated operator as a second ROI, wherein the second ROI is: the ROI configured by the image operation node corresponding to the instantiated operator; and call each instantiated operator in sequence according to the calling order to process the target ROI corresponding to each instantiated operator in the captured image.

[0325] It can be seen that in this embodiment, each image operation node can process the ROI in the image in a targeted manner, thereby reducing the computing resources consumed by image processing; furthermore, by setting the ROI and ROI inheritance status information for each image operation node, the ROI that each image operation node needs to pay attention to can be flexibly set, thereby improving the flexibility of image processing.

[0326] In one embodiment of the present application, the server is deployed in a control device, and the apparatus further includes:

[0327] The IO interface subscription module is used to control the robotic arm to establish a unidirectional IO interface subscription relationship with the server if the device configuration sub-scheme includes input and output IO interface configuration information, so that the robotic arm synchronizes the IO interface configuration to the server; the IO signal subscription module is used to control the robotic arm to establish a bidirectional IO signal subscription relationship with the server if the device configuration sub-scheme includes IO signal configuration information, so that the robotic arm and the server synchronize IO signals with each other.

[0328] In this way, the server can promptly learn about the IO interfaces supported by the robotic arm, and achieve IO interface synchronization between the robotic arm and the visual module in the server, so that a usable visual processing solution can be built based on the supported IO operations in the future; and the robotic arm and the server can synchronize IO signals in a timely manner, so that the server can promptly learn about the execution status of various IO operations performed by the robotic arm, thereby improving the accuracy of controlling the robotic arm.

[0329] In one embodiment of the present application, the image processing sub-scheme in the visual processing scheme registered by the server is stored in the form of serialized data.

[0330] In this way, the storage space required for storing the image processing sub-scheme can be reduced, saving consumption of storage resources.

[0331] In one embodiment of the present application, the device further comprises:

[0332] a first configuration module, configured to, in response to receiving device editing information for a visual processing solution to be configured sent by a client, configure a device management sub-solution in the visual processing solution to be configured based on the device editing information;

[0333] a second configuration module, configured to, in response to receiving image processing flow editing information for the visual processing solution to be configured sent by the client, configure an image processing sub-solution in the visual processing solution to be configured based on the image processing flow editing information;

[0334] a third configuration module, configured to, in response to receiving the execution process editing information for the visual processing solution to be configured sent by the client, configure the execution configuration sub-solution in the visual processing solution to be configured based on the execution process editing information;

[0335] a fourth configuration module, configured to, in response to receiving the visual calibration editing information for the visual processing solution to be configured sent by the client, configure the visual calibration sub-scheme in the visual processing solution to be configured based on the visual calibration editing information;

[0336] The scheme registration module is used to register the visual processing scheme obtained after the configuration is completed if it is determined based on the configuration result that the visual processing scheme to be configured is completed.

[0337] In this embodiment, the server can configure and register a new visual processing solution in response to the configuration information sent by the client, so that staff can simply and conveniently register customized visual processing solutions on the server according to the actual needs of various application scenarios, thereby improving the flexibility and applicability of the solution.

[0338] In one embodiment of the present application, the second configuration module is specifically used to generate a first image processing operation linked list based on the image operation nodes included in the first editing information and the execution order information of the image operation nodes if the image processing configuration method indicated by the first editing information is scheme creation, and obtain an image processing sub-scheme including the first image processing operation linked list as the image processing sub-scheme in the visual processing scheme to be configured, wherein the first editing information is: the image processing flow editing information for the visual processing scheme to be configured sent by the client; if the image processing configuration method indicated by the first editing information is scheme copying, then copy the second image processing operation linked list of the target image processing sub-scheme indicated by the first editing information, and obtain an image processing sub-scheme including the second image processing operation linked list as the image processing sub-scheme in the visual processing scheme to be configured.

[0339] As can be seen from the above, in this embodiment, the image processing sub-scheme is customized by the staff through the client. It is not preset or fixed, and has flexibility and portability, allowing the staff to quickly and easily complete the construction of the image processing sub-scheme when dealing with different scenarios and tasks. Specifically, when configuring the image processing sub-scheme, new schemes can be created to meet the staff's personalized needs, and existing schemes can be copied, which reduces the difficulty of configuring the image processing sub-scheme and improves the configuration efficiency. It can be seen that through multiple scheme configuration methods, the flexibility in creating image processing sub-schemes is improved.

[0340] In one embodiment of the present application, the first configuration module is specifically used to generate IO interface configuration information for controlling the robotic arm to establish a unidirectional IO interface subscription relationship with the server if the second editing information includes an IO interface subscription instruction, and record the IO interface configuration information in the device management sub-scheme in the visual processing scheme to be configured, wherein the second editing information is: the device editing information for the visual processing scheme to be configured sent by the client; if the second editing information includes an IO signal subscription instruction, generate IO signal configuration information for controlling the robotic arm to establish a bidirectional IO signal subscription relationship with the server, and record the IO signal configuration information in the device management sub-scheme in the visual processing scheme to be configured.

[0341] In this way, IO interface configuration information and IO signal configuration information can be generated in response to the staff's instructions, and the above information can be recorded in the device management sub-plan. Therefore, when the device management sub-plan is executed subsequently, a unidirectional IO interface subscription relationship and a bidirectional IO signal subscription relationship can be established based on the configuration information, which facilitates the synchronization of IO interfaces and IO signals between the server and the robotic arm, thereby improving the accuracy of control.

[0342] In one embodiment of the present application, the visual device connected to the server includes: an image acquisition device and a light source device. The image acquisition device is connected to the electronic device where the server is deployed through a first interface, and the light source device is connected to the electronic device through a second interface.

[0343] The first image acquisition module 1302 is specifically used to control the end effector of the robotic arm to move to the position for triggering image acquisition recorded in the first execution configuration sub-scheme included in the first visual processing scheme, and send a first control instruction to the first light source device in the first visual device through the second interface to enable the first light source device to start lighting; send a second control instruction to the first image acquisition device in the first visual device through the first interface to enable the first image acquisition device to perform image acquisition; in response to the first image acquisition device successfully acquiring the first image, send a third control instruction to the first light source device through the second interface to enable the first light source device to turn off lighting.

[0344] In this way, the server can first control the light source device to turn on the lighting, and then control the image acquisition device to capture the image, reducing the probability of low brightness in the captured image due to insufficient ambient brightness, and improving the quality of the captured image.

[0345] Corresponding to the above-mentioned robotic arm control method applied to the client, an embodiment of the present application also provides a robotic arm control device applied to the client.

[0346] See also Figure 14 , is a structural diagram of a second robotic arm control device provided in an embodiment of the present application, applied to a client, the device comprising:

[0347] A first operation response module 1401 is configured to send device editing information to a server in response to a device editing operation for a visual processing solution to be configured in a first configuration interface, so that the server configures a device management sub-solution in the visual processing solution to be configured based on the received device editing information;

[0348] A second operation response module 1402 is configured to send image processing process editing information to the server in response to an image processing process editing operation for the visual processing solution to be configured in the second configuration interface, so that the server configures the image processing sub-solution in the visual processing solution to be configured based on the received image processing process editing information;

[0349] The third operation response module 1403 is used to respond to the execution process editing operation for the visual processing scheme to be configured in the third configuration interface, and send execution process editing information to the server, so that the server configures the execution configuration sub-scheme in the visual processing scheme to be configured based on the received execution process editing information.

[0350] It can be seen that in the solution provided by the embodiment of the present application, the staff can perform visual control configuration and robotic arm motion control configuration in a visual manner on a client, thereby obtaining a customized visual processing solution. There is no need for professional engineers to perform corresponding process settings or command input on the robotic arm controller side and the visual industrial computer side according to the specific process of the task to be performed, so that the staff can flexibly and simply deal with complex production environments, thereby improving the configuration efficiency of the visual processing solution.

[0351] In addition, based on the integrated customer segment, staff can flexibly and conveniently adjust the task execution strategy in different tasks and environments, making the solution more adaptable and able to quickly respond to environmental changes and complex scenario requirements.

[0352] Furthermore, in the solution provided in the embodiment of the present application, after the configuration of the visual processing solution is completed based on the client, the server can collaboratively control the robotic arm and the visual device according to the visual processing solution. Compared with the visual industrial computer controlling the visual device alone and the robotic arm controller controlling the robotic arm alone, the above method eliminates the communication interaction between the visual industrial computer and the robotic arm controller, increases the communication efficiency while reducing the communication delay, improves the real-time performance of controlling the robotic arm, and thus improves the task execution efficiency of the robotic arm.

[0353] In one embodiment of the present application, the device further comprises:

[0354] A fourth operation response module is used to send visual calibration editing information to the server in response to the visual calibration configuration operation for the visual processing scheme to be configured in the fourth configuration interface, so that the server configures the visual calibration sub-scheme in the visual processing scheme to be configured based on the received visual calibration editing information.

[0355] Thus, the visual processing solution also includes a visual calibration sub-solution. Therefore, before the server controls the robotic arm and visual device to perform a task according to the execution configuration sub-solution, it can first calibrate the robotic arm and visual device based on the visual calibration sub-solution to obtain the latest calibration results before executing the task. Therefore, subsequent operations based on the latest calibration results can further improve the accuracy of visual processing and thus enhance the efficiency of the robotic arm's task execution.

[0356] Corresponding to the above-mentioned robotic arm control method, an embodiment of the present application also provides an electronic device, a computer-readable storage medium, and a computer program.

[0357] The present application embodiment provides an electronic device, such as Figure 15 As shown, including:

[0358] Memory 1501, used for storing computer programs;

[0359] The processor 1502 is configured to implement the aforementioned robotic arm control method when executing the program stored in the memory 1501 .

[0360] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 1502, the communication interface, and the memory 1501 communicate with each other via the communication bus.

[0361] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0362] The communication interface is used for communication between the above electronic device and other devices.

[0363] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0364] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0365] In another embodiment provided in the present application, a computer-readable storage medium is further provided, in which a computer program is stored. When the computer program is executed by a processor, the above-mentioned robot arm control method is implemented.

[0366] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the above-mentioned robotic arm control method.

[0367] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital staff line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state drive (SSD).

[0368] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0369] Each embodiment in this specification is described in a related manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences from other embodiments. In particular, the device, electronic device, and storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For related portions, reference can be made to the descriptions of the method embodiments.

[0370] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. A method for controlling a robotic arm, characterized in that: Applied to a server, the server is connected to a robotic arm and a visual device, the robotic arm and the visual device are deployed in the same working environment, the visual device is used to collect images of the task target, and the server is registered with multiple sets of visual processing solutions, each set of visual processing solutions includes: a device management sub-solution, an image processing sub-solution, and an execution configuration sub-solution. The method includes: In response to a task start instruction, loading a first visual processing solution to be executed from the plurality of visual processing solutions; controlling the end effector of the robotic arm to move to a position for triggering image acquisition recorded in a first execution configuration sub-scheme included in the first visual processing scheme, and controlling a first visual device configured in a first device management sub-scheme included in the first visual processing scheme to perform image acquisition to obtain a first image; Processing the first image according to the image processing flow recorded in the first image processing sub-scheme included in the first visual processing scheme to obtain a target position of the task target in the first image; The end effector is controlled to move to the target position, and the end effector is controlled to execute the action recorded in the first execution configuration sub-scheme according to the task target.

2. The method according to claim 1, characterized in that The visual processing solution further includes: a visual calibration sub-solution, which, before controlling the end effector of the robotic arm to move to the position for triggering image acquisition recorded in the first execution configuration sub-solution included in the first visual processing solution, further includes: Generate a calibration path based on the positions of the calibration points recorded in the first visual calibration sub-scheme included in the first visual processing scheme; controlling the end effector to move according to the calibration path, and in response to the end effector moving to a position of a calibration point included in the calibration path, controlling the first visual device configured in the first device management sub-scheme to perform image acquisition to obtain a second image; After determining that the end effector traverses the calibration points included in the calibration path, the robotic arm and the first visual device are visually calibrated based on the spatial position and image position of the feature point of the set marker in the captured second image, wherein the spatial position is determined based on the position of the calibration point included in the calibration path.

3. The method according to claim 2, characterized in that The visual calibration sub-scheme records: a first position of a calibration center point, a second position of a calibration corner point, a rotation angle of a rotation calibration point, and calibration point distribution information, wherein the calibration center point is a calibration point located at the center of a calibration area covered by a calibration path to be generated, and the calibration corner point is a calibration point located at a corner point of the calibration area. The calibration path is generated based on the positions of the calibration points recorded in the first visual calibration sub-scheme included in the first visual processing scheme, including: Determining, based on the first position, the second position, and the calibration point distribution information and the number of calibration points recorded in the calibration point distribution information, a third position of the first calibration point other than the calibration center point and the calibration corner point and a fourth position of the rotational calibration point for visual calibration; According to the traversal order information recorded in the calibration point distribution information, a calibration path including the translation calibration point and the rotation calibration point is generated, and based on the rotation angle, control information for controlling the rotation angle of the end effector at the rotation calibration point is generated, wherein the translation calibration point includes: the calibration center point, the calibration corner point and the first calibration point.

4. The method according to claim 3, characterized in that In response to the end effector moving to the position of the calibration point included in the calibration path, controlling the first visual device configured in the first device management sub-scheme to perform image acquisition, including: If it is determined that the end effector moves to the position of the translation calibration point included in the calibration path, controlling the first visual device configured in the first device management sub-scheme to perform image acquisition; If it is determined that the end effector moves to the position of the rotation calibration point included in the calibration path, the end effector is controlled to rotate a first angle, and the first visual device is controlled to perform image acquisition, wherein the first angle is: the rotation angle indicated by the control information of the rotation calibration point.

5. The method according to claim 1, wherein The image processing sub-scheme defines: an image processing operation linked list, the image processing operation linked list includes multiple image operation nodes, the pointer direction between each image operation node represents the order of image processing, each image operation node represents an image processing operation, and the first image is processed according to the image processing flow recorded in the first image processing sub-scheme included in the first visual processing scheme, including: Identifying an image processing operator for implementing the image processing operation represented by each first image operation node, wherein the first image operation node is an image operation node included in a first image processing operation linked list, and the first image processing operation linked list is an image processing operation linked list defined in a first image processing sub-scheme included in the first visual processing scheme; Loading the identified image processing operator from pre-packaged image processing operators, instantiating the loaded image processing operator to obtain an instantiated operator; Determining the calling order of the instantiation operators according to the pointer directions between the image operation nodes in the image processing operation linked list; In accordance with the calling order, each instantiated operator is called in turn to process the operator input image, wherein the operator input image of the first instantiated operator in the calling order is: the first image, and the operator input images of the remaining instantiated operators except the first instantiated operator are: the image output after the image processing is performed by the forward adjacent instantiated operator of the instantiated operator.

6. The method according to claim 5, characterized in that The image processing sub-scheme is configured with: a region of interest ROI and / or ROI inheritance information of an image operation node, and sequentially calling each instantiated operator to process the operator input image in the calling order, including: If the ROI inheritance information configured in the image operation node corresponding to the instantiation operator represents an inherited ROI, then the first ROI is used as the target ROI of the instantiation operator, wherein the first ROI is: the ROI output by the image operation node corresponding to the forward adjacent instantiation operator of the instantiation operator; If the ROI inheritance information configured by the image operation node corresponding to the instantiation operator indicates that the ROI is not inherited, a second ROI is used as the target ROI of the instantiation operator, wherein the second ROI is: the ROI configured by the image operation node corresponding to the instantiation operator, or an ROI obtained by modifying the configured ROI; In accordance with the calling order, each instantiation operator is called in turn to process the target ROI corresponding to each instantiation operator in the operator input image.

7. The method according to claim 1, characterized in that The server is deployed in a control device, and the method further includes: If the device configuration sub-scheme includes input and output IO interface configuration information, the robot arm is controlled to establish a unidirectional IO interface subscription relationship with the server, so that the robot arm synchronizes the IO interface configuration to the server; if the device configuration sub-scheme includes IO signal configuration information, the robot arm is controlled to establish a bidirectional IO signal subscription relationship with the server, so that the robot arm and the server synchronize IO signals with each other; and / or The image processing sub-scheme in the visual processing scheme registered by the server is stored in the form of serialized data.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: In response to receiving device editing information for the visual processing solution to be configured sent by the client, configuring a device management sub-solution in the visual processing solution to be configured based on the device editing information; In response to receiving the image processing flow editing information for the visual processing solution to be configured sent by the client, configuring the image processing sub-solution in the visual processing solution to be configured based on the image processing flow editing information; In response to receiving the execution process editing information for the visual processing solution to be configured sent by the client, configuring the execution configuration sub-solution in the visual processing solution to be configured based on the execution process editing information; In response to receiving the visual calibration editing information for the visual processing solution to be configured sent by the client, configuring a visual calibration sub-scheme in the visual processing solution to be configured based on the visual calibration editing information; If it is determined based on the configuration result that the visual processing solution to be configured has been configured, the visual processing solution obtained after the configuration is completed is registered.

9. The method according to claim 8, characterized in that In response to receiving the image processing flow editing information for the visual processing solution to be configured sent by the client, configuring the image processing sub-solution in the visual processing solution to be configured based on the image processing flow editing information, including: If the image processing configuration mode indicated by the first editing information is scheme creation, then based on the image operation nodes included in the first editing information and the execution order information of the image operation nodes, a first image processing operation linked list is generated, and an image processing sub-scheme including the first image processing operation linked list is obtained as the image processing sub-scheme in the visual processing scheme to be configured, wherein the first editing information is: the image processing flow editing information for the visual processing scheme to be configured sent by the client; if the image processing configuration mode indicated by the first editing information is scheme copying, then the second image processing operation linked list of the target image processing sub-scheme indicated by the first editing information is copied, and an image processing sub-scheme including the second image processing operation linked list is obtained as the image processing sub-scheme in the visual processing scheme to be configured; and / or The step of configuring, in response to receiving device editing information for the visual processing solution to be configured sent by the client, a device management sub-solution in the visual processing solution to be configured based on the device editing information includes: If the second editing information includes an IO interface subscription instruction, IO interface configuration information is generated for controlling the robot arm to establish a unidirectional IO interface subscription relationship with the server, and the IO interface configuration information is recorded in the device management sub-scheme in the visual processing scheme to be configured, wherein the second editing information is: the device editing information sent by the client for the visual processing scheme to be configured; if the second editing information includes an IO signal subscription instruction, IO signal configuration information is generated for controlling the robot arm to establish a bidirectional IO signal subscription relationship with the server, and the IO signal configuration information is recorded in the device management sub-scheme in the visual processing scheme to be configured.

10. The method according to any one of claims 1 to 7, characterized in that The visual device connected to the server includes: an image acquisition device and a light source device, the image acquisition device is connected to the electronic device where the server is deployed through a first interface, and the light source device is connected to the electronic device through a second interface, and the first visual device configured in the first device management sub-scheme included in the first visual processing scheme is controlled to perform image acquisition to obtain a first image, including: Sending a first control instruction to a first light source device in a first visual device through the second interface, so that the first light source device starts lighting, wherein the first visual device is: a visual device configured in the first device management sub-scheme included in the first visual processing scheme; Sending a second control instruction to a first image acquisition device in the first visual device through the first interface, so that the first image acquisition device performs image acquisition; In response to the first image acquisition device successfully capturing the first image, a third control instruction is sent to the first light source device through the second interface, so that the first light source device turns off illumination.

11. A method for controlling a robotic arm, characterized in that: Applied to a client, the method includes: In response to a device editing operation for the visual processing solution to be configured in the first configuration interface, sending device editing information to the server, so that the server configures the device management sub-solution in the visual processing solution to be configured based on the received device editing information; In response to an image processing flow editing operation for the visual processing solution to be configured in the second configuration interface, sending image processing flow editing information to the server, so that the server configures the image processing sub-solution in the visual processing solution to be configured based on the received image processing flow editing information; In response to the execution process editing operation for the visual processing scheme to be configured in the third configuration interface, execution process editing information is sent to the server, so that the server configures the execution configuration sub-scheme in the visual processing scheme to be configured based on the received execution process editing information.

12. The method according to claim 11, characterized in that The method further comprises: In response to the visual calibration configuration operation for the visual processing scheme to be configured in the fourth configuration interface, visual calibration editing information is sent to the server, so that the server configures the visual calibration sub-scheme in the visual processing scheme to be configured based on the received visual calibration editing information.

13. A robotic arm control device, characterized in that: Applied to a server, the server is connected to a robotic arm and a visual device, the robotic arm and the visual device are deployed in the same working environment, the visual device is used to collect images of the task target, and the server is registered with multiple sets of visual processing solutions, each of which includes: a device management sub-solution, an image processing sub-solution, and an execution configuration sub-solution. The device includes: a solution loading module, configured to load a first visual processing solution to be executed from the plurality of visual processing solutions in response to a task start instruction; a first image acquisition module, configured to control the end effector of the robotic arm to move to a position triggering image acquisition recorded in the first execution configuration sub-scheme included in the first visual processing scheme, and control the first visual device configured in the first device management sub-scheme included in the first visual processing scheme to perform image acquisition to obtain a first image; an image processing module, processing the first image according to the image processing flow recorded in the first image processing sub-scheme included in the first visual processing scheme, to obtain a target position of the task target in the first image; An action execution module is used to control the end effector to move to the target position and control the end effector to execute the action recorded in the first execution configuration sub-scheme.

14. The device according to claim 13, characterized in that The visual processing scheme further includes: a visual calibration sub-scheme, and the apparatus further includes: a calibration path generation module for generating a calibration path based on the positions of calibration points recorded in the first visual calibration sub-scheme included in the first visual processing scheme before the first image acquisition module is triggered; a second image acquisition module for controlling the movement of the end effector according to the calibration path, and in response to the end effector moving to the position of the calibration point included in the calibration path, controlling the first visual device configured in the first device management sub-scheme to perform image acquisition to obtain a second image; a visual calibration module for determining that after the end effector traverses each calibration point included in the calibration path, the end effector performs visual calibration on the robotic arm and the first visual device based on the spatial position and image position of the feature point of the set marker in the acquired second image, wherein the spatial position is determined based on the position of the calibration point included in the calibration path; and / or The visual calibration sub-scheme records: a first position of a calibration center point, a second position of a calibration corner point, a rotation angle of a rotation calibration point, and calibration point distribution information. The calibration center point is a calibration point located at the center of a calibration area covered by a calibration path to be generated. The calibration corner point is a calibration point located at a corner of the calibration area. The calibration path generation module includes a calibration point determination sub-module for determining, based on the first position, the second position, and the calibration point distribution information and the number of calibration points recorded in the calibration point distribution information, a third position of the first calibration point other than the calibration center point and the calibration corner point, and a fourth position of the rotation calibration point for performing visual calibration; a calibration path generation submodule, for generating a calibration path including translation calibration points and the rotation calibration points according to the traversal order information recorded in the calibration point distribution information, and generating control information for controlling the rotation angle of the end effector at the rotation calibration point based on the rotation angle, wherein the translation calibration point includes: the calibration center point, the calibration corner point, and the first calibration point; and / or The second image acquisition module is specifically configured to control the movement of the end effector according to the calibration path, and if it is determined that the end effector moves to the position of the translation calibration point included in the calibration path, control the first visual device configured in the first device management sub-scheme to perform image acquisition; if it is determined that the end effector moves to the position of the rotation calibration point included in the calibration path, control the end effector to rotate a first angle, and control the first visual device to perform image acquisition, wherein the first angle is: the rotation angle indicated by the control information of the rotation calibration point; and / or The image processing sub-scheme is defined as follows: an image processing operation chain list, wherein the image processing operation chain list includes a plurality of image operation nodes, the pointer direction between each image operation node represents the order of image processing, and each image operation node represents an image processing operation. The image processing module includes: an operator identification sub-module for identifying the image processing operator used to implement the image processing operation represented by each first image operation node, wherein the first image operation node is: the image operation node included in the first image processing operation chain list, and the first image processing operation chain list is: the image processing operation chain list defined in the first image processing sub-scheme included in the first visual processing scheme; an operator instantiation sub-module for The identified image processing operator is loaded into the encapsulated image processing operator, and the loaded image processing operator is instantiated to obtain an instantiated operator; a sequence determination submodule is used to determine the calling sequence of the instantiated operator according to the pointer direction between each image operation node in the image processing operation linked list; the image processing submodule is used to sequentially call each instantiated operator to process the operator input image according to the calling sequence, wherein the operator input image of the first instantiated operator in the calling sequence is: the first image, and the operator input images of the remaining instantiated operators except the first instantiated operator are: the image output after the image processing of the forward adjacent instantiated operator of the instantiated operator, and the target position of the task target in the first image is obtained; and / or The image processing sub-scheme is configured with: region of interest (ROI) information and / or ROI inheritance information of the image operation node; the image processing sub-module is specifically configured to, if the ROI inheritance information configured by the image operation node corresponding to the instantiated operator indicates an inherited ROI, use a first ROI as the target ROI of the instantiated operator, wherein the first ROI is: the ROI output by the image operation node corresponding to the forward adjacent instantiated operator of the instantiated operator; if the ROI inheritance information configured by the image operation node corresponding to the instantiated operator indicates a non-inherited ROI, use a second ROI as the target ROI of the instantiated operator, wherein the second ROI is: the ROI configured by the image operation node corresponding to the instantiated operator, or an ROI obtained by correcting the configured ROI; and call each instantiated operator in sequence according to the calling order to process the target ROI corresponding to each instantiated operator in the captured image; and / or The server is deployed in a control device, and the apparatus further includes: an IO interface subscription module, configured to control the robotic arm to establish a unidirectional IO interface subscription relationship with the server if the device configuration sub-scheme includes input and output IO interface configuration information, so that the robotic arm synchronizes the IO interface configuration to the server; an IO signal subscription module, configured to control the robotic arm to establish a bidirectional IO signal subscription relationship with the server if the device configuration sub-scheme includes IO signal configuration information, so that the robotic arm and the server synchronize IO signals with each other; and / or the image processing sub-scheme in the visual processing scheme registered by the server is stored in the form of serialized data; and / or The apparatus further includes: a first configuration module for, in response to receiving device editing information for the visual processing scheme to be configured sent by a client, configuring the device management sub-scheme in the visual processing scheme to be configured based on the device editing information; a second configuration module for, in response to receiving image processing flow editing information for the visual processing scheme to be configured sent by the client, configuring the image processing sub-scheme in the visual processing scheme to be configured based on the image processing flow editing information; a third configuration module for, in response to receiving execution flow editing information for the visual processing scheme to be configured sent by the client, configuring the execution configuration sub-scheme in the visual processing scheme to be configured based on the execution flow editing information; a fourth configuration module for, in response to receiving visual calibration editing information for the visual processing scheme to be configured sent by the client, configuring the visual calibration sub-scheme in the visual processing scheme to be configured based on the visual calibration editing information; and a scheme registration module for, if it is determined based on the configuration result that the visual processing scheme to be configured is completed, registering the visual processing scheme obtained after the configuration is completed; and / or The second configuration module is specifically used to generate a first image processing operation chain list based on the image operation nodes included in the first editing information and the execution order information of the image operation nodes if the image processing configuration mode indicated by the first editing information is scheme creation, and obtain an image processing sub-scheme including the first image processing operation chain list as the image processing sub-scheme in the visual processing scheme to be configured, wherein the first editing information is: the image processing flow editing information for the visual processing scheme to be configured sent by the client; if the image processing configuration mode indicated by the first editing information is scheme copying, then copy the second image processing operation chain list of the target image processing sub-scheme indicated by the first editing information to obtain an image processing sub-scheme including the second image processing operation chain list, as The image processing sub-scheme in the visual processing scheme to be configured; and / or the first configuration module, specifically used to generate IO interface configuration information for controlling the robotic arm to establish a unidirectional IO interface subscription relationship with the server if the second editing information includes an IO interface subscription instruction, and record the IO interface configuration information in the device management sub-scheme in the visual processing scheme to be configured, wherein the second editing information is: the device editing information for the visual processing scheme to be configured sent by the client; if the second editing information includes an IO signal subscription instruction, generate IO signal configuration information for controlling the robotic arm to establish a bidirectional IO signal subscription relationship with the server, and record the IO signal configuration information in the device management sub-scheme in the visual processing scheme to be configured; and / or The visual device connected to the server includes: an image acquisition device and a light source device. The image acquisition device is connected to the electronic device deployed on the server through a first interface, and the light source device is connected to the electronic device through a second interface. The first image acquisition module is specifically used to control the end effector of the robotic arm to move to the trigger image acquisition position recorded in the first execution configuration sub-scheme included in the first visual processing scheme, and send a first control instruction to the first light source device in the first visual device through the second interface, so that the first light source device starts lighting, wherein the first visual device is: a visual device configured in the first device management sub-scheme included in the first visual processing scheme; send a second control instruction to the first image acquisition device in the first visual device through the first interface, so that the first image acquisition device performs image acquisition; in response to the first image acquisition device successfully acquiring the first image, send a third control instruction to the first light source device through the second interface, so that the first light source device turns off lighting.

15. A robotic arm control device, characterized in that: Applied to a client, the device includes: a first operation response module, configured to, in response to a device editing operation for the visual processing solution to be configured in the first configuration interface, send device editing information to the server, so that the server configures a device management sub-solution in the visual processing solution to be configured based on the received device editing information; a second operation response module, configured to, in response to an image processing flow editing operation for the visual processing solution to be configured in the second configuration interface, send image processing flow editing information to the server, so that the server configures the image processing sub-solution in the visual processing solution to be configured based on the received image processing flow editing information; The third operation response module is used to respond to the execution process editing operation for the visual processing scheme to be configured in the third configuration interface, and send execution process editing information to the server, so that the server configures the execution configuration sub-scheme in the visual processing scheme to be configured based on the received execution process editing information.

16. The device according to claim 15, characterized in that The device further comprises: A fourth operation response module is used to send visual calibration editing information to the server in response to the visual calibration configuration operation for the visual processing scheme to be configured in the fourth configuration interface, so that the server configures the visual calibration sub-scheme in the visual processing scheme to be configured based on the received visual calibration editing information.

17. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the method described in any one of claims 1-10 or 11-12 when executing a program stored in a memory.

18. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 or 11 to 12 is implemented.

Images