A robot process package construction method and system

Abstract

The present invention provides a method and system for constructing a robot process package, the method comprising obtaining process image information, including multiple static pictures or videos reflecting the actual process; analyzing the process image information, obtaining the robot component data and the process data of the process from the process image information; selecting matching components from the equipment database, and the motion properties of the components can meet the requirements of the process data of the process; constructing a visual model according to the selected robot components, setting the process of the robot model, and configuring the state and action flow of the robot model in the process according to the acquired process data of the process, to form a robot process package. The present invention can realize the robot process package design more quickly with the intelligent assistance of the system, improve the design efficiency, and the constructed process package is easy to reuse and modify, providing a process package design scheme and system with a high degree of automation.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a robot process package construction method and system. Background Art

[0002] In the prior art, the control of robots usually uses programming, which requires users to use command line code or graphical programming to write the steps that the robot terminal needs to execute one by one. This programming method requires users to master a lot of programming skills and robot knowledge in advance, which is not friendly to new users and difficult to get started. The writing process needs to be oriented towards every detail, lacking top-level design thinking for the overall process. The programming method is usually not intuitive, lacks intelligent assistance, and the design efficiency is low. In addition, the written code has weak inheritability and portability. If the underlying equipment changes, the code also needs to be modified and corrected. In short, it is difficult to realize the complete process of a process on a robot, and there is a lack of process package design solutions and systems with strong automation. Summary of the invention

[0003] In view of this, the present invention provides a robot process package construction method and system, which at least partially solve the above-mentioned problems existing in the prior art.

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] A method for constructing a robot process package, characterized in that it comprises the following steps:

[0006] S100: Acquire process scene information, and acquire process scene information according to an actual application scenario. The process scene information includes process image information, and the process image information includes a plurality of static pictures or videos reflecting the actual process;

[0007] S200: analyzing the process image information, and obtaining process data information from the process image information, including component data of the robot and process data of the process;

[0008] S300: Selecting a component of the robot, and selecting a component matching the component data of the robot from the equipment database according to the acquired process data information, and the motion attribute of the component can meet the process data requirements of the process;

[0009] S400: constructing a robot model, constructing a visualization model according to the selected robot components, and combining them into a visualization model of the robot for exchanging data of each component of the robot model with the user;

[0010] S500: Setting the process of the robot model, configuring the state and action flow of the robot model in the process according to the acquired process data of the process, and forming a robot process package.

[0011] In S100 of the above solution, the obtaining of process scenario information according to the actual application scenario includes:

[0012] S110: The user operates the robot to execute a process in an actual application scenario, records process image information when the robot executes the process through a camera, and transmits the process image information to an interactive terminal; or,

[0013] S120: The user directly transmits the pre-made process image information to the interactive terminal;

[0014] The obtaining of process scenario information according to the actual application scenario also includes:

[0015] S130: The user inputs process description information into the interactive terminal, where the process description information is used to provide a reference for the step of obtaining process data information.

[0016] In S200 of the above solution, the acquiring of process data information from process image information includes:

[0017] S210: Recognize a robot image from the process image information, and further determine component data of the robot in the process image information through image recognition.

[0018] In S210 of the above solution, determining the component data of the robot in the process image information by image recognition includes:

[0019] S211: extracting image features of the robot and each component in the process image information;

[0020] S212: Acquire image features of various robots and various components from a device database;

[0021] S213: comparing the image features extracted from the process image information with the image features obtained from the equipment database, and determining the model of the robot and the models of the components constituting the robot in the process image information according to the similarity;

[0022] S214: Using the model of the robot and the models of the components constituting the robot as component data of the robot;

[0023] S215: when it is difficult to determine the model of the robot and the models of the components constituting the robot, extracting structural dimension features of the robot and the components in the process image information;

[0024] S216: The structural dimension characteristics of the robot and its components are used as component data of the robot.

[0025] In S200 of the above solution, the acquiring of process data information from process image information includes:

[0026] S220: Identify the starting point, end point, running track, speed, function and state change of each action in the process from the process image information as process data of the process.

[0027] In S300 of the above solution, the components of the selected robot specifically include:

[0028] S301: When the component data of the robot includes the model of the robot and the models of the components constituting the robot, directly selecting the corresponding robot component in the device database according to the model;

[0029] S302: When the component data of the robot does not include a model but includes a structural dimension feature, selecting a matching component of the robot in the equipment database according to the structural dimension feature;

[0030] S303: Filter robot components that can meet the requirements of the process according to the process data of the process.

[0031] In S400 of the above solution, the step of constructing a visual model based on the selected robot components specifically includes:

[0032] S401: acquiring a visualization model of a corresponding component from a device database according to the selected component of the robot, or generating a visualization model of the component according to a preset visualization rule according to the selected component of the robot;

[0033] S402: Combining the visualization models of the components to obtain a visualization model of the robot; displaying the visualization model of the robot on an interactive terminal, providing data of the components of the robot model, and allowing the user to modify or set parameters of the components.

[0034] In S500 of the above solution, configuring the state and action flow of the robot model in the process includes, on the interactive interface of the interactive terminal, the user configures the process of controlling the robot terminal to change state or perform action according to the process data of the process.

[0035] In S500 of the above solution, configuring the state and action flow of the robot model in the process includes automatically generating a control program for controlling the robot terminal to switch states or perform actions in the form of graphic programming or code programming according to process data of the process, wherein the control program can be executed on the robot terminal to control the robot terminal to complete the process;

[0036] The control program and the component data of the robot model are packaged as a process package.

[0037] The above scheme also includes S600: saving the constructed process package into the equipment database as a new process package or an upgrade of an existing process package, and saving the theme of the actual application scenario, the key actions of the process or the key components of the robot model as a label of the process package for easy retrieval when the process package is used again in the future.

[0038] In the above scheme, the device database stores detailed data of various components of the robot, including the model of each component and one or more of the following parameters: structural size, electrical interface, motion range, motion speed, maximum load, applicable function, and label; the device database also stores image information or image features of each component.

[0039] A robot process package construction system, characterized in that it includes a processor and a memory, wherein the memory stores computer executable instructions, and when the processor executes the instructions, the robot process package construction system implements the method described in any of the above schemes.

[0040] By implementing the solution described in the present invention, the design of the robot control program is changed to start from the overall task purpose, and the system intelligently analyzes the scene image information to obtain the hardware and steps required to complete the process, and adaptively selects suitable candidate components from the database. New users can quickly implement the robot program design for executing a process with the intelligent assistance of the system, which improves the efficiency of the design, and the constructed process package can be reused many times. Even if the equipment changes, it is easy to modify and regenerate the process package, providing a process package design solution and system with a high degree of automation. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 The present invention provides a flowchart of the method for constructing a robot process package. DETAILED DESCRIPTION

[0043] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0044] It should be noted that the following embodiments and features in the embodiments may be combined with each other in the absence of conflict; and, based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making any creative work are within the scope of protection of the present disclosure.

[0045] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0046] The embodiment of the first aspect of the present invention provides a method for constructing a robot process package, characterized in that it includes the following steps:

[0047] S100: Acquire process scene information. Acquire process scene information according to an actual application scenario. The process scene information includes process image information. The process image information includes a plurality of static pictures or videos reflecting the actual process.

[0048] In this embodiment, the first problem to be solved is how to describe the process requirements. The requirements of the actual process scene do not require the user to use a specific programming language to describe, but can be obtained in the most intuitive way from the video of the actual process scene, and the specific requirements can be parsed through image analysis. For example, the user can manually operate the robot terminal on site to perform a set of actions to move goods, and use the camera of the robot terminal to record, and record various information of the handling process as the input information source for building the process package. Considering that the recording video has high requirements for the quality of equipment shooting, and has high requirements for video storage, transmission, and processing, as an optional alternative, multiple static pictures can also be used instead of videos. As long as the pictures can reflect the key information, the process requirements can also be parsed from them.

[0049] In addition to recording process videos on site, other methods can be used to obtain relevant videos. For example, some videos may have been recorded in advance, so they can be directly transferred to the system through the interactive terminal instead of being recorded on site. Others may be videos of other robots performing a certain process obtained through the network. In this case, the source of the video does not need to be limited, as long as it can be transferred to the system for analysis.

[0050] Although it is advocated to use images as the input of process requirements, it does not exclude the possibility that users can provide descriptive text or codes as an aid through the interactive terminal. For example, if the user already knows the purpose of the process, he can input the type of process, such as moving goods or screw assembly, etc. Such information will inevitably help the subsequent image analysis.

[0051] The above technical solution can help to obtain the demand information of actual application scenarios conveniently and quickly. The direct input of pictures or videos is very intuitive and reduces the pressure on users in describing their needs.

[0052] S200: Analyze the process image information, and obtain process data information from the process image information, including component data of the robot and process data of the process.

[0053] The obtaining of process data information from process image information includes:

[0054] S210: Recognize a robot image from the process image information, and further determine component data of the robot in the process image information through image recognition.

[0055] It is not difficult to identify a robot from a picture or video by means of object detection, but it is necessary not only to identify the robot but also to further identify the components of the robot. A relatively simple implementation method is to pre-attach a model number or a one-dimensional barcode or a two-dimensional barcode or other information graphics that can identify its identity to the robot in the video. In this way, as long as such image features can be extracted from the video or image, it is easy to parse the robot model and component model.

[0056] But more often, there may be no such information on the equipment in the process image, or the video source cannot be mastered, and it is impossible to produce a video with the above information again. Then it is necessary to analyze the image features of the robot or component in the image. A favorable solution is to store images of various types of equipment in the equipment database, such as five types of robotic arms, three types of grippers, etc. By extracting and comparing the image features of these components one by one with the image features of the components in the handling scene, it can be determined that the A3 type robotic arm and C2 type gripper are used in the handling scene image. In order to reduce the image storage pressure of the equipment database and the processing pressure of feature extraction, the image features of each component can be extracted in advance, and then only the image features can be stored in the equipment database to improve efficiency. Image features can use various features in the field of image processing, such as shape, color, texture, spatial features, frequency domain features, etc., which are not limited here, as long as they are conducive to image recognition. However, in comparison, storing the original images of each component can be suitable for more processing scenarios and has higher flexibility. These two forms can be used in combination. If it is recognized that the scene image includes images of certain components, the models of these components can be recorded for subsequent use. If there are multiple similar component images in the recognition result, multiple candidate components can be recommended for the user to choose from according to their similarity from high to low.

[0057] At the same time, it is possible that a device with a similar image cannot be found in the device database. It may be that the image of the actual application scene is not clear enough, or it may be that the appearance of the device is indeed different. In this case, it is necessary to obtain the structural dimension characteristics of the robot and each component from the process image information, and then query the matching components based on the structural dimension information. In order to more accurately identify the size of each component, a favorable implementation method includes setting a reference object such as a ruler in the scene being photographed. It is easy to obtain the size of other objects by comparing them with the reference object. In addition, a ruler can be pre-marked on the camera, and the size of the component can be easily obtained through geometric dimension relationships.

[0058] S220: Identify the starting point, end point, running track, speed, function and state change of each action in the process from the process image information as process data of the process.

[0059] For example, for a handling process, it is necessary to identify the starting and ending coordinates of each action of each component, the moving trajectory and speed, whether the action is clamping or loosening, etc., as process data of the process. These process data can be combined to realize the function of transporting goods from point A to point B. For example, it is necessary to identify that the robot arm moves from the initial point O to point A, the gripper opens, descends, clamps, and rises, the robot arm moves from point A to point B, the gripper descends, opens, rises, and clamps, and the robot arm moves from point B to point O. In particular, it is necessary to identify the three-dimensional coordinate values ​​of each point.

[0060] By analyzing process image information to obtain process data information, the system's intelligent auxiliary analysis function can be fully utilized. Image analysis can help users identify components, process actions and states, and reduce the difficulty of decomposing components and actions when users are not familiar with the system's equipment, components, etc.

[0061] S300: Selecting a component of the robot, and selecting a component that matches the component data of the robot from an equipment database according to the acquired process data information, and the motion properties of the component can meet the process data requirements of the process.

[0062] When the robot component data includes the robot model and the models of the components that make up the robot, the corresponding robot component is directly selected in the equipment database according to the model; when the robot component data does not include the model but includes structural size characteristics, the corresponding robot component is selected in the equipment database according to the structural size characteristics. For example, the identified component data includes that the robot arm is a four-axis robot arm, and the lengths of the two arms are 220mm and 170mm respectively. By querying the equipment database, the A3 type robot arm can meet the requirements.

[0063] In addition, it is also necessary to screen robot parts that can meet the process requirements based on the process data of the process. For example, if the motion range of the robot during handling is within a circular range of 350mm in diameter, then the A3 robot arm can be determined to meet the requirements based on its motion range. When there are multiple parts or combinations of parts that can meet the requirements, multiple options can be proposed for users to choose from.

[0064] The above solution can help users select suitable components to compose a robot, reducing the user's confusion about how to choose from a large number of components. The selection here has also taken into account the process data requirements of the process, avoiding the problem that users cannot meet their needs in actual scenarios when selecting inappropriate components.

[0065] S400: construct a robot model, construct a visualization model according to the selected robot components, and combine them into a visualization model of the robot for exchanging data of each component of the robot model with the user.

[0066] According to the selected robot components, the visualization model of the corresponding components is obtained from the device database, or the visualization model of the components is generated according to the preset visualization rules according to the selected robot components; the visualization models of the components are combined to obtain the visualization model of the robot. The visualization model here can be some 3D model diagrams, such as 3D model diagrams in STP, DWG and other formats. However, the visualization model is not necessarily a 3D component graphic in a certain format, but can also be a 2D graphic, or even a simplified icon-like graphic, as long as the user can intuitively identify the existence of these components in the interactive interface. These model files can be stored in the device database and can be queried and obtained by the component model.

[0067] The main function of the visualization model here is to present it on the interactive interface, so that users can understand what parts are there, provide the data of each part of the robot model, and facilitate users to select and view the data of each part, and allow users to change or set parameters for each part. If the user thinks that some parts or their parameters are unreasonable, or wants to modify them, they can click on these parts in the interactive interface to operate them very conveniently, without having to search repeatedly in the command line code.

[0068] S500: Setting the process of the robot model, configuring the state and action flow of the robot model in the process according to the acquired process data of the process, and forming a robot process package.

[0069] Configuring the state and action flow of the robot model during the process includes providing a graphical user interface on an interactive terminal, on which the user can configure step by step the process of controlling the robot terminal to change state or perform actions according to the process data of the process. Based on these configurations, the robot terminal can be controlled to execute corresponding instructions to realize the functions of these process processes.

[0070] Another optional solution is to automatically generate a control program for controlling the robot terminal to change state or perform actions according to the process data of the process through graphic programming or code programming, and the control program can be executed on the robot terminal to control the robot terminal to complete the process;

[0071] The control program and the component data of the robot model are packaged as a process package.

[0072] For example, the process of handling needs to realize these actions and state changes: the robot is powered on, reset to zero, the robot arm is translated from the initial point O to point A, the gripper opens, descends, clamps, and rises, the robot arm is translated from point A to point B, the gripper descends, opens, rises, and clamps, the robot arm is translated from point B to point O, reset to zero, and the robot is powered off. These specific actions are performed based on the A3 type robot arm and C2 type gripper identified by image analysis and recognition. Based on this information, the control program that controls the A3 type robot arm and C2 type gripper to complete the handling process is automatically generated. This control program is packaged together with the component data of these two types of components as the process package of this handling process. After the user builds the process package of this handling process on the interactive terminal, the interactive terminal is connected to the robot terminal. This robot terminal is composed of the A3 type robot arm and the C2 type gripper, or at least compatible with these two models. Then the control program in this process package can be loaded onto the robot terminal and control the robot terminal to complete the handling process.

[0073] The process package generated here hardly requires users to master sufficient programming skills. Most functions are completed automatically. Users only need to adjust some parameters according to personalized needs, which reduces the labor intensity of users and greatly improves the speed and efficiency of generating process packages. The program is mainly generated by the system, which also reduces the possibility of bugs and improves code security.

[0074] S600: Save the constructed process package to the equipment database as a new process package or an upgrade of an existing process package, and save the theme of the actual application scenario, the key actions of the process, or the key components of the robot model as the label of the process package, so that it can be searched when the process package is used again in the future. The stored process package can also be read again from the equipment database and displayed on the interactive terminal, so that the user can adjust and confirm the parameters again. After convenient adjustment and modification, the updated process package can be automatically generated again, and can be quickly loaded into the new robot terminal and executed.

[0075] This process package can be labeled with keywords such as "handling process" or "A3C2" that are easy to remember and reflect its characteristics, so that it is easy to find or use this process package again in the future. At the same time, the naming of the process package can also be regarded as a label. Components in the equipment database, such as A3 robot arms or C2 grippers, can also be labeled "handling process" as a component, so that when building a new handling process package in the future, it is easier to find components that have been used in similar processes.

[0076] In the above embodiment, the equipment database stores detailed data of various components of the robot, including the model of each component and one or more of the following parameters: structural dimensions, electrical interface, range of motion, motion speed, maximum load, applicable function, and label; the equipment database also stores image information or image features of each component. The equipment database can be set on a network server or on the same terminal device as the interactive terminal. The step of analyzing process image information can be processed on the interactive terminal using a local processor or on a cloud server in the cloud.

[0077] An embodiment of the second aspect of the present invention provides a robotic process package construction system, characterized in that it includes a processor and a memory, wherein the memory stores computer executable instructions, and when the processor executes the instructions, the robotic process package construction system implements the method described in any of the above schemes.

[0078] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for constructing a robot process package, characterized in that: The following steps are involved: S100: Acquire process scene information, and acquire process scene information according to an actual application scenario. The process scene information includes process image information, and the process image information includes a plurality of static pictures or videos reflecting the actual process; S200: analyzing the process image information, and obtaining process data information from the process image information, including component data of the robot and process data of the process; S3 00: Select the robot component, and select the component matching the robot component data from the equipment database according to the acquired process data information, and the motion attribute of the component can meet the process data requirements of the process; S400: constructing a robot model, constructing a visualization model according to the selected robot components, and combining them into a visualization model of the robot for exchanging data of each component of the robot model with the user; S500: Setting a process of the robot model, configuring the state and action flow of the robot model in the process according to the acquired process data of the process, and forming a robot process package; In S200, the acquiring of process data information from process image information includes: S210: recognizing a robot image from the process image information, and further determining component data of the robot in the process image information through image recognition; In S210, determining the component data of the robot in the process image information through image recognition includes: S211: extracting image features of the robot and each component in the process image information; S212: Acquire image features of various robots and various components from a device database; S213: comparing the image features extracted from the process image information with the image features obtained from the equipment database, and determining the model of the robot and the models of the components constituting the robot in the process image information according to the similarity; S214: Using the model of the robot and the models of the components constituting the robot as component data of the robot; S215: when it is difficult to determine the model of the robot and the models of the components constituting the robot, extracting structural dimension features of the robot and the components in the process image information; S216: The structural dimension characteristics of the robot and its components are used as component data of the robot.

2. The method for constructing a robot process package according to claim 1, characterized in that: In S100, the process scenario information is obtained according to the actual application scenario, including: S110: The user operates the robot to execute a process in an actual application scenario, records process image information when the robot executes the process through a camera, and transmits the process image information to an interactive terminal; or, S120: The user directly transmits the pre-made process image information to the interactive terminal; The obtaining of process scenario information according to the actual application scenario also includes: S130: The user inputs process description information into the interactive terminal, where the process description information is used to provide a reference for the step of obtaining process data information.

3. The method for constructing a robot process package according to claim 1, characterized in that: In S200, the acquiring of process data information from process image information includes: S220: Identify the starting point, end point, running track, speed, function and state change of each action in the process from the process image information as process data of the process.

4. The method for constructing a robot process package according to claim 1, characterized in that: In S300, the components of the selected robot specifically include: S301: When the component data of the robot includes the model of the robot and the models of the components constituting the robot, directly selecting the corresponding robot component in the device database according to the model; S302: When the component data of the robot does not include a model but includes a structural dimension feature, selecting a matching component of the robot in the equipment database according to the structural dimension feature; S303: Filter robot components that can meet the requirements of the process according to the process data of the process.

5. The method for constructing a robot process package according to claim 1, characterized in that: In S400, the step of constructing a visual model based on the selected robot components specifically includes: S401: acquiring a visualization model of a corresponding component from a device database according to the selected component of the robot, or generating a visualization model of the component according to a preset visualization rule according to the selected component of the robot; S402: Combining the visual models of the components to obtain a visual model of the robot; The interactive terminal displays a visual model of the robot, provides data of various components of the robot model, and allows the user to change or set parameters of various components.

6. The method for constructing a robot process package according to claim 1, characterized in that: In S500, configuring the state and action flow of the robot model in the process includes, on the interactive interface of the interactive terminal, the user configuring the process of controlling the robot terminal to change state or perform action according to the process data of the process.

7. The method for constructing a robot process package according to claim 1, characterized in that: In S500, configuring the state and action flow of the robot model in the process includes automatically generating a control program for controlling the robot terminal to change state or perform action in the form of graphic programming or code programming according to process data of the process, wherein the control program can be executed on the robot terminal to control the robot terminal to complete the process; The control program and the component data of the robot model are packaged as a process package.

8. The method for constructing a robot process package according to claim 1, wherein: Also includes the steps: S600: The constructed process package is saved in the equipment database as a new process package or an upgrade of an existing process package, and the theme of the actual application scenario, the key actions of the process or the key components of the robot model are saved as tags of the process package so that the process package can be searched when it is used again in the future.

9. The method for constructing a robot process package according to claim 1, wherein: The device database stores detailed data of various components of the robot, including the model of each component and one or more of the following parameters: structural dimensions, electrical interface, motion range, motion speed, maximum load, applicable functions, and labels; the device database also stores image information or image features of each component.

10. A robot process package construction system, characterized in that: The system comprises a processor and a memory, wherein the memory stores computer executable instructions. When the processor executes the instructions, the robot process package construction system implements the method according to any one of claims 1 to 8.

Images