Robot intelligent operation system integrated development framework

Abstract

The invention discloses an integrated development framework of a robot intelligent operating system, and belongs to the technical field of robots, the framework comprises a resource management layer used for providing bottom hardware equipment contained in the robot intelligent operating system; the driving layer is used for providing an application programming interface required for driving bottom hardware equipment; the communication bridge node layer is used for providing a management service corresponding to the underlying hardware equipment according to an application programming interface corresponding to the underlying hardware equipment; the interface layer is used for packaging a management service corresponding to the underlying hardware equipment and generating a standardized application programming interface corresponding to the underlying hardware equipment; the scheduling layer is used for generating a task scheduling and resource management program corresponding to the underlying hardware equipment according to the standardized application programming interface corresponding to the underlying hardware equipment; and the user interface layer is used for providing a user interaction interface supporting equipment management and program editing service. A hierarchical design principle is adopted, and the adding and upgrading process of hardware equipment can be simplified.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to an integrated development framework for a robotics intelligent operating system. Background Art

[0002] With the rapid development of industrial automation, robotics technology is increasingly being used in a variety of mission scenarios, which requires that robot operating systems should be more flexible and intelligent, and the efficiency of integrated development of robot operating systems has become a key technical challenge. However, in the existing robot operating system integrated development framework, there are often problems such as complex configuration processes, long time consumption, and limited scalability, which restricts the rapid iteration and technological innovation of robot operating systems. Summary of the invention

[0003] The present invention provides a robot intelligent operating system integrated development framework to solve one or more technical problems existing in the prior art and at least provide a beneficial choice or create conditions.

[0004] The embodiment of the present invention provides a robot intelligent operating system integrated development framework, including:

[0005] The resource management layer is used to provide the underlying hardware devices involved in the robot intelligent operating system;

[0006] The driver layer is used to provide an application programming interface required to drive the underlying hardware device;

[0007] The communication bridge node layer is used to provide management services corresponding to the underlying hardware devices according to the application programming interface corresponding to the underlying hardware devices;

[0008] The interface layer is used to encapsulate the management service corresponding to the underlying hardware device and generate a standardized application programming interface corresponding to the underlying hardware device;

[0009] The scheduling layer is used to generate a task scheduling program and a resource management program corresponding to the underlying hardware device according to the standardized application programming interface corresponding to the underlying hardware device;

[0010] The user interface layer is used to provide a user interaction interface that supports device management services and program editing services.

[0011] Furthermore, the underlying hardware devices include a robot device, a camera device, a microphone device and a tool device, and a MoveIt configuration package and a unified robot description format file are defined for the robot device.

[0012] Furthermore, the driver layer is also used to execute control commands on the underlying hardware devices, read the status information of the underlying hardware devices and transmit it to the communication bridge node layer.

[0013] Furthermore, the driving layer includes:

[0014] A robot driving module, used to control the motion of the robot device and provide real-time feedback of the motion information of the robot device;

[0015] A camera driver module, used to control the camera device and perform tasks such as image capture, format conversion and data transmission;

[0016] A microphone driver module, used to interface with the microphone device and provide an advanced Linux audio architecture and command word recognition interface;

[0017] The tool driving module is used to interface with the tool equipment and perform control based on the input and output interface of the robot control system.

[0018] Furthermore, a MoveIt tool, a robot function expansion tool and a robot driving tool are set in the robot driving module. The MoveIt tool is used to provide motion planning function for the robot device, the robot function expansion tool is used to support application requirements of the robot device, and the robot driving tool is used to connect to the robot control system to control the robot device.

[0019] Furthermore, the communication bridge node layer includes:

[0020] A camera communication bridge, used to construct an abstract class including the main functions of the camera device according to the basic performance of the camera device;

[0021] A microphone communication bridge, used to capture audio information provided by the microphone device by calling an application programming interface provided by the advanced Linux audio architecture, and then extract instruction information from the audio information by using the command word recognition interface;

[0022] A robot motion communication bridge, used to provide motion and path planning services corresponding to the robot device;

[0023] The tool communication bridge is used to construct an abstract class that includes the main functions of the tool device according to the working characteristics of the tool device in different application scenarios.

[0024] Furthermore, in the interface layer, the standardized application programming interface is generated by Python language, provides different parameter configuration options and supports a callback mechanism.

[0025] Furthermore, a device management module and an application management module are provided on the user interaction interface, the device management module is used to drive and control the underlying hardware device and visualize its status information, and the application management module is used to provide script editing and application building functions.

[0026] Furthermore, the application management module includes an algorithm management unit, a project creation unit and a project file management unit. The algorithm management unit is used to provide different program templates and function interfaces required to control the underlying hardware devices. The project creation unit is used to provide convenient functions for opening projects, creating projects and graphical programming. The project file management unit is used to manage program code files and configuration files written for the project.

[0027] Furthermore, a tutorial learning module is provided on the user interaction interface, and the tutorial learning module is used to provide experimental tutorials for implementing different types of robot applications.

[0028] The present invention has at least the following beneficial effects: by adopting the layered design principle, the driver layer of the hardware device and the scheduling layer of the user's actual operation are effectively isolated, which can efficiently realize the rapid adaptation of hardware device resources, making the tasks of adding and upgrading hardware devices easier, improving the maintainability and extensibility of the robot intelligent operating system, and will not affect the stability of the original robot intelligent operating system. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0030] Figure 1 It is a schematic diagram of the structure of a robot intelligent operating system integrated development framework provided by an embodiment of the present invention;

[0031] Figure 2 It is a schematic diagram of a user interaction interface provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0033] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0034] Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the present invention are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0035] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present invention. However, those skilled in the art will appreciate that the technical solution of the present invention can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present invention.

[0036] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0037] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined, so the actual execution order can be changed according to actual conditions.

[0038] First, several nouns involved in the present invention are analyzed:

[0039] The Robot Intelligent Operating System Integrated Development Framework aims to provide an integrated working environment for robot software developers to write, debug and deploy robot-related software applications through an integrated hardware and software environment for controlling robots and other devices.

[0040] The MoveIt configuration package is an important tool for configuring robot motion planning. It is generated through the MoveIt SetupAssistant tool, which provides a friendly user interface that allows users to step through the configuration process of loading robot models, setting planning groups, configuring kinematic solvers, setting collision detection, etc., and then automatically generates a MoveIt configuration package containing all configuration files. The MoveIt configuration package can be used in the ROS (Robot Operating System) workspace later.

[0041] The Universal Robot Description Format (URDF) file is an XML file format used to describe the robot hardware configuration, allowing users to define the robot's structure, joints, links, sensors, and other physical properties.

[0042] Application Programming Interface (API) is a specification that defines how different software applications interact with each other, allowing communication and data exchange between different software applications.

[0043] Advanced Linux Sound Architecture (ALSA) is a module in the Linux kernel for processing audio input and output. It provides a unified driver module for supporting various audio devices such as sound cards, microphones, speakers, etc.

[0044] With the rapid development of industrial automation, robotics technology is increasingly being used in a variety of mission scenarios, such as manufacturing automation, home services, medical assistance, disaster response, and other mission scenarios. This requires the robot operating system to be more flexible and intelligent, and the integrated development efficiency of the robot operating system has become a key technical challenge. In addition, the integration of artificial intelligence technology requires more advanced data processing and adaptive capabilities in the robot operating system, which puts higher requirements on the robot operating system integrated development environment. However, in the existing robot operating system integrated development framework, there are often problems such as complex configuration process, long time consumption, limited scalability, and steep learning curve. These problems limit the rapid iteration and technological innovation of the robot operating system.

[0045] In view of this, an embodiment of the present invention provides an integrated development framework for a robot intelligent operating system. By adopting the layered design principle, the driver layer of the hardware device and the scheduling layer of the user's actual operation are effectively isolated, which can efficiently realize the rapid adaptation of hardware device resources, making the tasks of adding and upgrading hardware devices easier, improving the maintainability and extensibility of the robot intelligent operating system, and will not affect the stability of the original robot intelligent operating system.

[0046] Please refer to Figure 1 , Figure 1 : This is a schematic diagram of the structure of a robot intelligent operating system integrated development framework provided by an embodiment of the present invention. The framework specifically includes a resource management layer, a driver layer, a communication bridge node layer, an interface layer, a scheduling layer and a user interface layer, wherein:

[0047] The resource management layer is used to provide the underlying hardware devices included in the robot intelligent operating system; the driver layer is used to provide the application programming interface required to drive the underlying hardware devices, and is used to execute control commands for the underlying hardware devices, read the status information of the underlying hardware devices and pass it to the communication bridge node layer; the communication bridge node layer is used to provide management services corresponding to the underlying hardware devices according to the application programming interface corresponding to the underlying hardware devices; the interface layer is used to encapsulate the management services corresponding to the underlying hardware devices and generate a standardized application programming interface corresponding to the underlying hardware devices; the scheduling layer is used to generate a resource management program and a task scheduling program corresponding to the underlying hardware devices according to the standardized application programming interface corresponding to the underlying hardware devices; the user interface layer is used to provide a user interaction interface that supports program editing services and device management services.

[0048] In some embodiments, the underlying hardware devices provided in the resource management layer include robot devices, microphone devices, camera devices and tool devices. The tool devices can be understood as end tools configured to enable the robot device to implement certain specific application functions, such as gripper tools, suction cup tools, etc. At the same time, a unified robot description format file and MoveIt configuration package are defined in advance for the robot device.

[0049] In some embodiments, the driver layer mainly realizes the abstraction of the underlying hardware device, which can be understood as consisting of drivers related to the underlying hardware device, specifically including a robot driver module, a camera driver module, a microphone driver module and a tool driver module, wherein:

[0050] The robot driving module is used to perform precise motion control on the robot device and to provide real-time feedback on the motion information of the robot device to support complex automated operations; the camera driving module is used to control the camera device through the SDK (Software Development Kit) file provided by the manufacturer of the camera device and to perform tasks such as image capture, format conversion and data transmission to ensure efficient and high-quality image processing effects; the microphone driving module is used to interface with the microphone device and provide a command word recognition interface and an advanced Linux audio architecture, the advanced Linux audio architecture is used to realize the capture, buffering and transmission functions of audio signals, and the command word recognition interface is used to assist in identifying audio signals and obtaining the instruction information contained therein; the tool driving module is used to interface with the tool device and to perform control based on the input and output interface of the robot control system, and the robot control system preferably adopts the Wasu III robot control system, referred to as the HSC3 control system.

[0051] In addition, a MoveIt tool, a robot driving tool, and a robot function expansion (ROS-I, Robot Operating System Industrial) tool are set in the robot driving module. The MoveIt tool is used to provide a motion planning function for the robot device, the robot function expansion tool is used to support specific application requirements of the robot device, and the robot driving tool is used to connect to the robot control system to control the robot device.

[0052] In some embodiments, the communication bridge node layer specifically includes a camera communication bridge, a microphone communication bridge, a robot motion communication bridge, and a tool communication bridge, wherein:

[0053] The camera communication bridge is used to construct an abstract class that includes the main functions of the camera device according to the basic performance of the camera device. In the camera communication bridge, adapting different types of camera devices only needs to follow the rules of the abstract class. Exemplarily, the camera device includes a 2D camera device and a 3D camera device. The camera communication bridge generates a hardware call service corresponding to the 2D camera device by dynamically loading the SDK file provided by the manufacturer of the 2D camera device. Similarly, the camera communication bridge generates a hardware call service corresponding to the 3D camera device by dynamically loading the SDK file provided by the manufacturer of the 3D camera device. The hardware call service only records the ID (Identity document, identity identifier or unique identifier) ​​and other working parameters of the camera device, such as resolution, exposure time, focal length, aperture size, etc.

[0054] The microphone communication bridge is used to capture the audio information fed back by the microphone device by calling the application programming interface provided by the advanced Linux audio architecture, and then use the command word recognition interface to extract the instruction information from the audio information and publish it through topic communication for use by the interface layer.

[0055] The robot motion communication bridge is used to provide the interface layer with motion and path planning services corresponding to the robot device through ROS service communication.

[0056] The tool communication bridge is used to construct an abstract class that includes the main functions of the tool device according to the working characteristics of the tool device in different application scenarios. In the tool communication bridge, adapting different types of tool devices only needs to follow the rules of the abstract class.

[0057] In some embodiments, the interface layer mainly implements the information flow and command transmission from the communication bridge node layer to the scheduling layer, and by setting a special standardized application programming interface, developers can implement advanced business logic and control strategies without directly contacting the underlying complex logic. Among them, the standardized application programming interface is generated by Python language, which is easy to call and integrate, while maintaining the readability and maintainability of the code, and can provide different parameter configuration options, such as providing some simple data types such as integers, strings, Boolean values, etc., or providing complex data structures such as objects, arrays, etc., and supports a callback mechanism to notify the caller after completing certain specific operations, thereby adapting to complex robot application requirements.

[0058] It should be noted that in actual application, the first parameter of the standardized application programming interface needs to be set as the device ID determined by the communication bridge node corresponding to the underlying hardware device, and the subsequent parameters are generally parameters related to the driver corresponding to the underlying hardware device; and, the standardized application programming interfaces set for different underlying hardware devices of the same type (such as camera devices with different properties) differ only in the inconsistency of the device ID.

[0059] For example, when it is necessary to control the camera device to take a picture, the format of the corresponding standardized application programming interface can be set to IcrpCamera.take_photo("camera1"); when it is necessary to control the robot device to perform a specific movement, the format of the corresponding standardized application programming interface can be set to IcrpRobot.moveto_point_jr("arm",J1,J2,J3,J4,J5,J6,vec,acc); when it is necessary to control the tool device to perform a grabbing action, the format of the corresponding standardized application programming interface can be set to IcrpGripper.open_gripper("gripper0").

[0060] In some embodiments, the user interface layer uses VSCode plug-in technology to generate a user interaction interface that integrates at least a device management module, an application management module, and a tutorial learning module. Figure 2 As shown, the extension and plug-in mechanism is used to support different robot function customization and expansion requirements, including:

[0061] In the device management module, the RoslibJS tool is introduced to directly call the device management service provided by the communication bridge node layer to drive and control the underlying hardware devices and visualize their corresponding status information. The relevant visualization content can be the axis data, enable signal (power on / off), reset signal, coordinate control signal, etc. of the robot device, as well as the operation status of other types of underlying hardware devices; the application management module mainly provides script editing and application building functions that allow users to operate; the tutorial learning module mainly provides experimental tutorials for implementing different types of robot applications for users to learn.

[0062] More specifically, the application management module includes an algorithm management unit, a project creation unit and a project file management unit, see Figure 2 As shown, where:

[0063] The algorithm management unit is used to provide different program templates and function interfaces required to control different types of underlying hardware devices, so that users can realize the application requirements of the robot device; the project creation unit is used to provide convenient functions for opening projects, creating projects and graphical programming. The graphical programming refers to determining all the underlying hardware devices related to meeting the application requirements of the robot device and generating corresponding graphic blocks, and then building corresponding application logic between all graphic blocks to form a reasonable application scenario; the project file management unit is used to manage the relevant configuration files and program code files written for the project.

[0064] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0065] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0066] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0067] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0068] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0069] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0070] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0071] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. A robot intelligent operating system integrated development framework, characterized in that: include: The resource management layer is used to provide the underlying hardware devices involved in the robot intelligent operating system; The driver layer is used to provide an application programming interface required to drive the underlying hardware device; The communication bridge node layer is used to provide management services corresponding to the underlying hardware devices according to the application programming interface corresponding to the underlying hardware devices; The interface layer is used to encapsulate the management service corresponding to the underlying hardware device and generate a standardized application programming interface corresponding to the underlying hardware device; The scheduling layer is used to generate a task scheduling program and a resource management program corresponding to the underlying hardware device according to the standardized application programming interface corresponding to the underlying hardware device; The user interface layer is used to provide a user interaction interface that supports device management services and program editing services.

2. The robot intelligent operating system integrated development framework according to claim 1, characterized in that: The underlying hardware devices include a robot device, a camera device, a microphone device and a tool device, and a MoveIt configuration package and a unified robot description format file are defined for the robot device.

3. The robot intelligent operating system integrated development framework according to claim 2, characterized in that: The driver layer is also used to execute control commands on the underlying hardware devices, read the status information of the underlying hardware devices and transmit it to the communication bridge node layer.

4. The robot intelligent operating system integrated development framework according to claim 3, characterized in that: The driving layer includes: A robot driving module, used to control the motion of the robot device and provide real-time feedback of the motion information of the robot device; A camera driver module, used to control the camera device and perform tasks such as image capture, format conversion and data transmission; A microphone driver module, used to interface with the microphone device and provide an advanced Linux audio architecture and command word recognition interface; The tool driving module is used to interface with the tool equipment and perform control based on the input and output interface of the robot control system.

5. The robot intelligent operating system integrated development framework according to claim 4, characterized in that: A MoveIt tool, a robot function expansion tool and a robot driving tool are set in the robot driving module. The MoveIt tool is used to provide a motion planning function for the robot device, the robot function expansion tool is used to support the application requirements of the robot device, and the robot driving tool is used to connect to the robot control system to control the robot device.

6. The robot intelligent operating system integrated development framework according to claim 4, characterized in that: The communication bridge node layer includes: A camera communication bridge, used to construct an abstract class including the main functions of the camera device according to the basic performance of the camera device; A microphone communication bridge, used to capture audio information provided by the microphone device by calling an application programming interface provided by the advanced Linux audio architecture, and then extract instruction information from the audio information by using the command word recognition interface; A robot motion communication bridge, used to provide motion and path planning services corresponding to the robot device; The tool communication bridge is used to construct an abstract class that includes the main functions of the tool device according to the working characteristics of the tool device in different application scenarios.

7. The robot intelligent operating system integrated development framework according to claim 1, characterized in that: In the interface layer, the standardized application programming interface is generated by Python language, provides different parameter configuration options and supports callback mechanism.

8. The robot intelligent operating system integrated development framework according to claim 1, characterized in that: A device management module and an application management module are provided on the user interaction interface. The device management module is used to drive and control the underlying hardware device and visualize its status information, and the application management module is used to provide script editing and application program building functions.

9. The robot intelligent operating system integrated development framework according to claim 8, characterized in that: The application management module includes an algorithm management unit, a project creation unit and a project file management unit. The algorithm management unit is used to provide different program templates and function interfaces required to control the underlying hardware devices. The project creation unit is used to provide convenient functions for opening projects, creating projects and graphical programming. The project file management unit is used to manage program code files and configuration files written for the project.

10. The robot intelligent operating system integrated development framework according to claim 8, characterized in that: A tutorial learning module is also provided on the user interaction interface, and the tutorial learning module is used to provide experimental tutorials for implementing different types of robot applications.

Images