Method and system for implementing functions for an application using middleware

Abstract

A method and system for implementing a function for an application using middleware is provided. The method can include the steps of initializing an application node in the middleware corresponding to the application, the initializing including binding the function to the application node and associating the application node with an encryptor; connecting the middleware to a remote device using the encryptor through a transport layer; publishing a message to invoke the function from the application node to the encryptor through the middleware; and communicating with the remote device using the encryptor to invoke the function according to the published message. The method can implement a more secure, stable and real-time message exchange system.

Description

Technical Field

[0001] This invention relates to methods and systems for implementing functions for applications using middleware, and more specifically, to methods and systems for facilitating message exchange in applications by using middleware, establishing corresponding application nodes, and publishing messages to an encryptor through nodes in the middleware, wherein the encryptor uses a transport layer to call functions and share information for the functions. Background Technology

[0002] Middleware, or message exchange systems, is an abstraction layer situated between the operating system and distributed software applications. Some goals of middleware design are managing the heterogeneity of hardware and network interfaces, improving the quality of software applications, and simplifying software architecture to reduce development costs. Because the inputs and outputs of such a system are fixed, developers only need to build the algorithms for their applications without modifying the entire system, thus increasing version controllability. Furthermore, the programming abstractions provided by middleware can offer transparency regarding the distribution of applications in terms of location, concurrency, replication, fault tolerance, and mobility, enabling functionality that a single package cannot provide.

[0003] As an example of robot middleware, the Robot Operating System (ROS) is an open-source message exchange system suitable for rapid prototyping, but not for production. This is because it is compatible with... and Furthermore, it supports multiple programming languages, such as C / C++, ROS is widely used with Octave and Lisp. It also provides tools, libraries, and conventions, resulting in a large user community. Many useful simulators, visualization tools (Gazebo, RViz, rqt_plot, etc.) and open-source packages (e.g., navigation, canopen_master, rosserial, rosbridge_suite, MoveIt!, etc.) have been integrated into ROS.

[0004] However, ROS has its limitations. For example, ROS does not provide security against eavesdropping or adversarial attacks. Anyone who installs ROS can monitor internal messages and even input false information into the system. Furthermore, ROS is not designed for real-time message exchange. Additionally, ROS is unstable because its configuration and services are based on XMLRPC, and its transport is primarily based on Transmission Control Protocol (TCP). These aspects present problems when using ROS. Summary of the Invention

[0005] Therefore, the present invention provides a method for implementing functions for applications using middleware, which enables a more secure, stable and real-time message exchange system.

[0006] To address the aforementioned technical problems, this invention provides a method for implementing functions for an application using middleware on a computer. The method may include the following steps: initializing an application node in the middleware corresponding to the application, the initialization including binding the function to the application node and associating the application node with an encryptor; using the encryptor to connect the middleware to a remote device via a transport layer; publishing a message to invoke the function from the application node to the encryptor through the middleware; and, based on the published message, using the encryptor to communicate with the remote device to invoke the function.

[0007] Optionally, binding the function includes establishing one or more subscriber or publisher relationships between the function and the application node.

[0008] Optionally, the method further includes: receiving a function call to the function from the application, wherein publishing a message to call the function to the cipher via the middleware is executed in response to receiving the function call.

[0009] Optionally, the method further includes: in response to the encryptor receiving the published message from the application node, automatically serializing the published message to generate a serialized message and sending the serialized message through the transport layer.

[0010] Optionally, the method further includes: using the encryptor to receive a remote message from the remote device through the transport layer, and publishing the remote message to be received by the function at the application node through the middleware.

[0011] Optionally, the method further includes: in response to the encryptor receiving the remote message from the remote device through the transport layer, automatically deserializing the remote message to generate a deserialized message and sending the deserialized message through the middleware.

[0012] Optionally, the communication further includes performing a handshake between the encryptor and a remote encryptor of the remote device by exchanging middleware names and comparing the exchanged middleware names with locally registered middleware names.

[0013] Optionally, the middleware includes at least one additional application node and at least one additional encryptor, and the method further includes: exchanging published messages between the additional application node and the additional encryptor through the middleware to implement functions for the additional application node.

[0014] Optionally, each published message has an associated message type, and based on that associated message type, each published message is exchanged between an application node and an encryptor in the middleware.

[0015] The present invention also provides a system for implementing functions for an application using middleware on a computer, comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the processor to: initialize an application node in the middleware corresponding to the application, the initialization including binding the function to the application node and associating the application node with an encryptor; connect the middleware to a remote device via a transport layer using the encryptor; publish a message to invoke the function from the application node to the encryptor via the middleware; and, according to the published message, communicate with the remote device using the encryptor to invoke the function.

[0016] The present invention also provides a non-transient computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform a method for implementing a function for an application using middleware on a computer, the method comprising: initializing an application node in the middleware corresponding to the application, the initialization including binding the function to the application node and associating the application node with an encryptor; connecting the middleware to a remote device via a transport layer using the encryptor; publishing a message to invoke the function from the application node to the encryptor via the middleware; and communicating with the remote device using the encryptor to invoke the function according to the published message.

[0017] The technical solution of the present invention has the following advantages: the method of using middleware to implement functions for application programs provided by the present invention can realize a more secure, stable and real-time message exchange system. Attached Figure Description

[0018] The accompanying drawings, which are incorporated herein and form a part of this specification, illustrate embodiments of the present disclosure and, together with the description, further serve to explain the principles of the disclosure and enable those skilled in the art to make and use it. To more clearly illustrate the technical solutions in specific embodiments of the present invention or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of an exemplary middleware according to an embodiment of the present disclosure.

[0020] Figure 2This is an example of a robot controlled via a network according to embodiments of the present disclosure.

[0021] Figure 3 This is a block diagram of a robot controller middleware architecture according to an embodiment of the present disclosure.

[0022] Figure 4 This is a block diagram of a computer system for hosting a middleware computer according to an embodiment of the present disclosure.

[0023] Figure 5 This is a block diagram of the transport layer according to an embodiment of the present disclosure.

[0024] Figure 6 A block diagram illustrating the components of a message according to an embodiment of this disclosure.

[0025] Figure 7 A block diagram of the components of an encryptor according to an embodiment of the present disclosure.

[0026] Figure 8 This is a block diagram of the components of a node IO according to an embodiment of the present disclosure.

[0027] Figure 9 This is a block diagram of nodes according to an embodiment of the present disclosure.

[0028] Figure 10 A block diagram of other features of the middleware according to embodiments of the present disclosure.

[0029] Figure 11 This is a flowchart illustrating an exemplary method for implementing functions for an application using middleware on a computer, according to embodiments of the present disclosure.

[0030] Figure 12 This is a flowchart of an exemplary method for initializing and using middleware according to embodiments of the present disclosure.

[0031] Figure 13 This is a flowchart of an exemplary method for initializing and managing an encryptor according to embodiments of the present disclosure.

[0032] Figure 14 This is a flowchart of yet another exemplary method for initializing and using middleware according to embodiments of the present disclosure.

[0033] Figure 15 This is a flowchart of an exemplary method for initializing an application node according to embodiments of the present disclosure. Detailed Implementation

[0034] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Reference will now be made in detail to exemplary embodiments, examples of which are shown in the accompanying drawings. Where possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts.

[0036] Although specific configurations and arrangements have been discussed, it should be understood that this is for illustrative purposes only. Those skilled in the art will recognize that other configurations and arrangements can be used without departing from the spirit and scope of this disclosure. It will be apparent to those skilled in the art that this disclosure can also be used in a variety of other applications.

[0037] It should be noted that references to "one embodiment," "example embodiment," "some embodiments," "certain embodiments," etc., in the specification indicate that the described embodiments may include specific features, structures, or characteristics, but each embodiment may not necessarily include those specific features, structures, or characteristics. Furthermore, these phrases do not necessarily refer to the same embodiment. Additionally, when a specific feature, structure, or characteristic is described in connection with an embodiment, implementing such a feature, structure, or characteristic in relation to other embodiments is within the knowledge of those skilled in the art, whether explicitly described or not.

[0038] Generally, terms can be understood, at least in part, from their use in context. For example, depending at least in part on the context, the term “one or more” as used herein can be used to describe any feature, structure, or characteristic in a singular sense, or in a plural sense, to describe a combination of features, structures, or characteristics. Similarly, depending at least in part on the context, terms such as “a” or “the” can also be understood to convey either a singular or a plural usage. Furthermore, the term “based on” can be understood to not necessarily be intended to convey an exclusive set of factors, and can conversely allow for the presence of additional factors that are not necessarily explicitly described, again depending at least in part on the context.

[0039] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0040] As mentioned above, it would be beneficial to have a more secure, stable, and real-time message exchange system to replace ROS.

[0041] Embodiments of this disclosure address the aforementioned problems by providing methods and systems for implementing functions for applications using middleware constructed in a manner that provides a secure, stable, and real-time message exchange system.

[0042] Figure 1 The illustration shows a schematic diagram of an exemplary middleware according to an embodiment of the present disclosure. As noted above, this exemplary middleware is provided as an alternative middleware to replace the Robot Operating System (ROS).

[0043] Middleware includes the transport layer, messaging, node input / output, nodes, encryptors, and other components. These components are listed below. Figure 5-10 The definition is further detailed in the text. Figure 1 The basic components of these middleware parts are shown, including the functions defined in these parts. For example, the Transport Layer 500 component includes SSL Client 510, TCP Client 520, SSL Server 530, TCP Server 540, and Transport Layer 550. The Message 600 component includes Message 610, Navigation Message 620, Sensor Message 630, Diagnostic Message 640, Geometry Message 650, Standard Message 660, and Message Type 670. The Encryptor 700 component includes Encryptor 710, Subscriber Encryptor 720, Publisher Encryptor 730, and Remote Parameter 740. The Node IO 800 component includes Node Interface 810, Node IO 820, SubscriberInner 830, PublisherInner 840, and Publisher 850. The Node component 900 includes Middleware 910, Node Type 920, Node 930, and Node Handle 940. Other features 1000 include application class 1010, health monitor 1020, exception handling 1030, mathematical tools 1040, string tools 1050, and transformations 1060. These components of the middleware interact with each other in a specific way to facilitate message transmission over the infrastructure.

[0044] For a basic system build environment, middleware should at least be designed to... It runs on 18.04, and other implementations extend operating system support. 16.04 and As a non-restrictive example, for programming languages, middleware supports, for example, C++11 because it is fast. Scripting languages ​​like Ruby are often insufficient for real-time requirements. Because every line of code is visible, using scripting languages ​​can lead to security vulnerabilities, which can be minimized by further compiling the script code into binaries.

[0045] For the system architecture, the middleware generates one thread per node to maximize the utilization of computing resources. Alternatively, in one embodiment, the middleware uses a central master broker for shallow message relay. This embodiment may be susceptible to single points of failure, but as an alternative, the disclosed middleware can avoid this problem by adapting to a Data Distribution Service (DDS) framework / concept. Middleware generally avoids using open-source packages because it's difficult to guarantee their coding quality and security levels meet product requirements. That said, if sufficient research has been conducted before using these open-source libraries, the middleware can rely on them, as long as they are independent of ROS.

[0046] For example, for hardware drivers, simulators, and visualizers, middleware can call their Software Development Kit (SDK) or Application Programming Interface (API) as third-party libraries, or the middleware can allow nodes with only limited ROS access to achieve this using a middleware-ROS bridge. Furthermore, the middleware should provide some common robot-specific mathematical or string processing utility functions, which will be integrated with the following... Figure 10 Further discussion.

[0047] For robustness, middleware may include a health monitor as a fault protection feature, allowing it to dynamically regenerate nodes when they fail. Middleware should also catch failure exceptions and report them to the `std::exception` channel. These features are also combined with... Figure 10 Further discussion.

[0048] As an alternative, middleware can support the Data Distribution Standard (DDS) provided by the Object Management Group (OMG). For example, instead of a star-architecture using a central master broker, a distributed node discovery mechanism under a subscription / publication paradigm with various Quality of Service (QoS) measurements would be more beneficial. Furthermore, for remote transmissions with real-time requirements (e.g., control streams and audio / video streams for robotic arm manipulation), middleware should also support the Time-Sensitive Networking (TSN) standard (IEEE 802.1Q) to ensure very low latency and high availability.

[0049] Figure 2 An example of a network-controlled robot according to an embodiment of the present disclosure is illustrated. For example, Figure 2 An exemplary system architecture 200 for controlling a robot using middleware, according to an embodiment of the present disclosure, is illustrated.

[0050] like Figure 2 As shown, system architecture 200 may include robot 210, network 220, user 230, and middleware computer 240. Robot 210 can implement various functions for user 230. User 230 interacts with middleware computer 240, so that information is sent between robot 210 and middleware computer 240 representing user 230 via network 220. To facilitate interaction between robot 210 and user 230, middleware may be hosted by middleware computer 240, as discussed further below.

[0051] Network 220 provides a medium for a communication link between robot 210 and middleware computer 240, which interacts with user 230 to enable user 230 to control robot 210. Network 220 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0052] User 230 may want robot 210 to provide various services. For example, the user may want the robot to perform certain actions, or the robot to provide data to the user. Middleware computer 240, as described below, provides an improved way for user 230 to manage the exchange of information with robot 210, for example, by providing instructions to robot 210 or by receiving information generated by robot 210.

[0053] For example, robot 210 may be equipped with sensors 212 and actuators 214, as well as a controller 216. Robot 210 may also include a network interface 218 for connecting to a network 220, which in turn connects to a middleware computer 240. The middleware computer 240 facilitates the transmission of function calls, function parameters, and function return values ​​on behalf of the user 230 via the network 220, allowing the user 230 to interact more effectively with robot 210. Specifically, the middleware hosted by the middleware computer 240 makes it easier for the user 230 to program robot 210.

[0054] Sensor 212 and actuator 214 can transmit the collected information to controller 216 via various communication methods (e.g., wired or wireless communication links or fiber optic cables). As a non-limiting example, sensor 212 can be various electronic devices used to collect information about the surrounding environment, including but not limited to cameras and radar. The environmental information can be visual or auditory, but those skilled in the art will readily recognize that many other examples of sensor 212 are possible. Actuator 214 can be various electronic devices used to cause robot 210 to take physical actions to manipulate the environment, such as starting an electric motor to move a part of robot 210, but those skilled in the art will readily recognize that many other examples of sensor 212 are possible.

[0055] The controller 216 employs a robot operating system, such as an operating system designed to interact with middleware, and under the management of the middleware computer 240 and the controller 216, converts the surrounding environment information collected by the sensor 212 and the instructions provided by the user 230 to be sent through the network interface 218 and the middleware computer 240 via the network 220 into control information for the actuator 214, so that the robot 210 can easily interact with the user 230.

[0056] The controller 216 can also manage the transmission of information from the sensor 212 to the user 230. Such a controller 216 may include a processor and memory, and may have instructions for operating the robot 210 using software, or it may have hard-coded instructions for operating the controller 216 in a predetermined manner.

[0057] It should be understood that Figure 2 The number of robots 210 and users 230 is arbitrary. There can be any number of robots 210 and users 230, suitable as an example in practice.

[0058] Figure 3 A block diagram of a robot controller middleware architecture 300 (hereinafter referred to as "middleware 300") according to an embodiment of the present disclosure is shown. Figure 3 The interaction between transport layer 310, message 320, encryptor 330, node I / O 340, node 350, application class 360, and additional features 370 is illustrated. These aspects of middleware 300 are already somewhat related to the above. Figure 1 This was discussed. However, the overall interaction of these components of the middleware is about... Figure 3The transport layer 310 is responsible for end-to-end communication on network 220. In middleware 300, transport layer 310 provides the ability to send (write) strings to or receive (read) strings from a remote location, as discussed further below. Middleware 300 can transform messages entering or leaving transport layer 310 into message 320, providing the ability to pass message 320 through middleware 300.

[0059] Specifically, message 320 is transmitted from middleware 300 to encryptor 330. Encryptor 330 is a special type of node, which will be discussed further below. Encryptor 330 is designed to manage messages 320 sent to or received from transport layer 310. Once encryptor 330 receives message 320 from transport layer 310, it uses node IO 340 to share the message with other nodes 350, which are designed to interact with application class 360, providing a template for how user 230 interacts with middleware 300. Middleware 300 can also provide additional features 370 that extend its capabilities by providing additional capabilities when managing application execution.

[0060] Figure 3 Further details of the transport layer 310 shown are presented in Figure 5 Further details of message 320 are in... Figure 6 Presented in [the document / document]. Further details of the Encryptor 330 are in [the document / document / document]. Figure 7 Presented in [the document / document]. Further details of node 1O340 are in [the document / document ... Figure 8 Presented in [the document / document]. Further details of node 350 are in [the document / document ... Figure 9 Presented in [the document / platform]. Further details about the 360 ​​application class are in [the document / platform]. Figure 10 Presented in [the image / image]. Further details of additional feature 370 are also presented in [the image / image]. Figure 10 middle.

[0061] Figure 4 The illustration shows a block diagram of a computer system capable of hosting middleware according to an embodiment of the present disclosure. (Refer to...) Figure 4 This diagram illustrates a schematic structural diagram of a computer system 400 suitable for implementing a middleware computer according to one or more embodiments. For example, the computer system 400 can be used as an implementation of a middleware computer 240.

[0062] like Figure 4As shown, the computer system 400 includes a processor 410, which can execute various appropriate actions and processes according to a program. The program is stored in a read-only memory (ROM) 420; or the program is loaded into the memory 420 from a random access memory (RAM) 430. Memory 420 and memory 430 also store various programs and data required for the operation of the computer system 400. The processor 410, memory 420, and memory 430 are interconnected via a bus 440. Input / output (I / O) 450 and a network interface 460 are also connected to the bus 440.

[0063] like Figure 4 As shown, computer system 400 may include processor 410. Although only one processor is shown, it is understood that multiple processors may be included. Processor 410 may include microprocessors, microcontrollers, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functions described throughout this disclosure. Processor 410 may be a hardware device having one or more processing cores. Processor 410 may execute software. Software should be interpreted broadly as instructions, instruction sets, code, code segments, program code, programs, subroutines, software modules, application programs, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., firmware, middleware, microcode, hardware description languages, or others, whether or not they are referred to as software. Software may include computer instructions written in interpreted languages, compiled languages, or machine code, although, as mentioned above, there may be certain problems with using interpreted languages. Other techniques for indicating hardware are also permitted under the broad category of software.

[0064] like Figure 4As shown, computer system 400 may also include memory 420 and storage 430. Although only one memory 420 is shown, it should be understood that multiple memories may be included. Memory 420 generally refers to working memory used for interacting with processor 410, while memory 430 refers to hardware used for long-term data maintenance. For example, memory 420 may include random access memory (RAM), read-only memory (ROM), static RAM (SRAM), dynamic RAM (DRAM), and / or ferroelectric RAM (FRAM). Memory 430 may include electrically erasable programmable ROM (EEPROM), CD-ROM or other optical disc storage, hard disk drive (HDD) (e.g., disk storage or other magnetic storage device), flash drive, solid-state drive (SSD), and / or any other medium that can be used to carry or store required program code in the form of instructions accessible and executed by processor 410. Generally, memory 420 may contain any computer-readable medium suitable for storing information for processor 410, and memory 430 may contain any non-volatile computer-readable medium, such as non-transitory computer-readable medium.

[0065] Furthermore, the computer system 400 may include input / output (I / O) components 450. For example, all or part of a keyboard, mouse, microphone, camera, touchscreen, etc., may be connected as input to the I / O 450. However, these are merely non-limiting examples, and other inputs may be used. For instance, other inputs that provide information such as visual information, audio information, tactile information, or digital information may be used.

[0066] For example, monitors / displays including cathode ray tubes (CRTs), liquid crystal displays (LCDs), light-emitting diode displays (LEDs), or other output devices (such as speakers) can be connected as outputs to IO 450. However, these are merely non-limiting examples, and other outputs may be used. For instance, other outputs may be used that provide visual information, audio information, tactile information, or digital information, or generate two-dimensional (2-D) or three-dimensional (3-D) printed outputs.

[0067] Network interface 460 includes network interface cards, such as wireless adapters, Ethernet adapters, LAN cards, and / or modems. Network interface 460 performs communication processes over network 220 (e.g., the Internet), although other types of networks such as local area networks (LANs) or wide area networks (WANs) may also be used. Other types of network architectures may also be used, such as client-server and peer-to-peer networks. The network may be based on wired networks, wireless networks, fiber optics, or any other medium that allows information to be exchanged between computing devices.

[0068] Figure 5 A block diagram of a transport layer 500 according to an embodiment of the present disclosure is shown.

[0069] In the Open Systems Interconnection (OSI) model of computer networks, the transport layer is responsible for end-to-end communication on the network. In the middleware 300 disclosed herein, transport layer 500 refers to a set of classes labeled transport layer 550 that have the ability to losslessly send or write strings to and from remote devices. As a non-limiting example, examples of protocols that can be used in transport layer 500 include Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Secure Sockets Layer (SSL), and Control Area Network (CAN). However, these are merely examples, and other protocols may be used, provided they meet the requirements of the transport layer class. As a transport layer 550 class, the following virtual functions should be implemented in a non-blocking manner such that the implementation satisfies the conditional timeout threshold = 0, indicated by the function description.

[0070] If system conditions permit, the function `bool Create() = 0` will instantiate the transport layer object and return a flag indicating whether the creation was successful. Object parameters can be configured elsewhere, such as in the transport layer object's constructor or `init()` function.

[0071] The function `int Connect() = 0` connects a transport layer object to a remote connection, such as a remote device on a network. Because some transport layer implementations allow multiple remote connections, this function returns `remoteIndex(0, 1, ...)` of the connected connections when the connection is successful, so that the system can later call `int Read / Write(..., remoteIndex)`. If there are no pending new connections, or if the connection capacity is reached, this function returns -1.

[0072] The function `bool Close(int remoteIndex) = 0` closes the connection at the given remote index and returns a flag indicating whether the disconnection was successful. If `remoteIndex` is -1 when this function is called, it will close all open connections.

[0073] The function `int Read(char str[], size_t size, size_tremoteIndex) = 0` reads a string `str` of length `size` and the given `remoteIndex` into the transport layer buffer and returns the number of bytes actually received. If `remoteIndex` is invalid, the function returns -2. If there is no message, the function returns -1. If the remote connection is lost or the device is disconnected, the function returns 0.

[0074] The function `int Write(char str[], size_t size, size_t clientIndex) = 0` sends a string `str` of length `size` and the given `remoteIndex`, and returns the number of bytes actually received. If `remoteIndex` is invalid, the function returns -2. If the message cannot be sent successfully, the function returns -1.

[0075] The function int Capacity() const = 0 is a getter function that returns the maximum number of remote connections that this transport layer class can make.

[0076] The function `int MaxBuffSize() const = 0` is a getter function that returns the maximum buffer size that can be read or written at one time for this transport layer class.

[0077] For example, in the middleware, the transport layer class supports TCP communication at TCP server 540 and TCP client 520, as well as SSL communication based on the TCP layer at SSL server 530 and SSL client 510. Furthermore, in other examples, the transport layer class can be configured to support UDP and UDPSSL as placeholders for Time-Sensitive Networking (TSN) development, such as for real-time applications like video / audio streaming and robotic arm manipulation control flows. Additionally, the Control Area Network (CAN) transceiver is designed for lower-level heterogeneous hardware connectivity.

[0078] Figure 6 A block diagram illustrating the components of a message according to an embodiment of the present disclosure is shown.

[0079] In some embodiments, middleware messages are associated with ROS messages. Each ROS message definition corresponds to one middleware message definition. For example, std_msgs::Header and stdMsgs::Header, or geometry_msgs::Pose and geometryMsgs::Pose.

[0080] Figure 6 The definitions of many different message types 610 are shown, including navigation messages 620, sensor messages 630, diagnostic messages 640, geometry messages 650, and standard messages 660. Messages are also associated with services (stdSrvs, navSrvs, sensorSrvs, diagnosticSrvs) 680 and message types 670.

[0081] In some embodiments, a unique message type 670 value can be assigned to each message, a C++ enumeration, to further facilitate the cipher's handshake process or exception handling. Furthermore, each middleware message is an extension of a class called message610, which defines virtual functions for serialization, deserialization, and message type retrieval. Regarding serialization, the middleware relies on the third-party library msgpack-c for message serialization and deserialization. According to experimental results, this library is four times faster than Google's protobuf and provides abstractions for various programming languages ​​such as Python (msgpack-python), Java (msgpack-java), and C# (msgpack-cli).

[0082] The virtual functions defined in message 610 are as follows, as shown in the example C++ implementation.

[0083] The function void serialize(msgpack_packer*) serializes the message into an initialized msgpack-c packer.

[0084] The function `msgpack_unpack_return deserialize(char* str, size_t strLen, size_t&offset)` deserializes the string `str` of length `strLen` based on an automatically maintained variable offset. Calling this function and returning a successful unpacking flag will correctly initialize message 610.

[0085] The function MessageType getType() is a getter function that returns the message type 670 assigned to message 610.

[0086] This design decouples the instantiation of these message classes from the actual implementation of serialization and deserialization functions. For example, stdMsgs::Header can be instantiated and published / subscribed to in a shared memory paradigm without actually implementing the serialization function for this class. However, if the message is transmitted through a transport layer, the serialization function will be implemented.

[0087] Figure 7 A block diagram illustrating the components of an encryptor according to an embodiment of the present disclosure is shown.

[0088] Encryptor 710 is a special node 930 that automatically serializes messages and sends the serialized string through the transport layer when it receives an incoming message from subscriber encryptor 720. Whenever an incoming message string is received from the transport layer, it deserializes the string and then publishes the deserialized message to the middleware system through publisher encryptor 730. For encryptor 710, the user can manually call the following functions. Encryptor 710 is also associated with remote parameters 740, which is a special class containing remote middleware parameters such as name, rate, subscriber list, and publisher list.

[0089] The function `bool registerPlatfonns(vector<int>)` <string>Platforms register an array of remote middleware names to associate with a given encryptor 710. When two encryptors connect remotely, a handshake process occurs, during which the middleware names of the encryptors are exchanged. If a remote middleware name does not match any locally registered name, the remote encryptor will disconnect. However, if the handshake is successful and the names match, the encryptors are able to establish a connection.

[0090] The function `CarrrierType getCarrierType()` is a getter function that returns the transport layer type of the encryptor 710. As a non-limiting example, the encryptor 710 could be a TCP server 540, a TCP client 520, an SSL server 530, and an SSL client 510, such as... Figure 5 Further discussion is needed.

[0091] The function `bool disconnect(string platformName)` disconnects a remote middleware with the given name `platformName`.

[0092] The function `void subscribe(string topic, size_t qSize)` is a type-defined function for message type 670. When this function is called, the cipher subscribes to the specified topic and automatically serializes it.

[0093] The function `void Advertisement(string topic, size_t qSize)` is also a type definition function for message type 670. After this function is called, the encryptor will publish the message to the specified topic whenever a message is received from the transport layer.

[0094] The function void setMiddlewareName(string) sets the name of the middleware that this cipher will use during a remote handshake.

[0095] Figure 8 A block diagram illustrating the components of a node IO according to an embodiment of the present disclosure is shown.

[0096] The Node IO (or Node Interface) 800 defines a collection of classes that handle various multi-threaded processes of the node, such as mutex locking, message shared pointer encapsulation, and message queue handling of application nodes. As long as the quality management of these classes is successful (e.g., no memory leaks, the classes are fast and stable), the rest of the system can focus on in-thread processing as if each thread were a single-threaded one. The Node Interface 810 is a message type invariant class that defines the following virtual functions.

[0097] The function std::shared_ptr<const void> Read() = 0 returns a shared pointer to the oldest message in the message queue. If there is no message in the queue, the function will return a shared pointer to nullptr.

[0098] The function void Write(const std::shared_ptr<const void>& msg) = 0 enqueues a message in the message queue for later processing.

[0099] The functions Void Process(…) = 0 and void initialize(…) = 0 are functions for subscribers and for publishers, which will be introduced later.

[0100] The Node IO 820 inherited from the Node Interface 810 is a class defined for message type 670. The message queue and mutex are defined here. For example, SubscriberInner 830, PublisherInner 840, Subscriber Encryptor 720, and Publisher Encryptor 730 all inherit from the Node IO 820, so they are all for message type 670. This design allows the node to dynamically specify the message type for internal subscribers / publishers at compile time. Users do not interact directly with these classes. Instead, users can define a callback function to bind to each subscriber, and then the subscriber will automatically call such a callback function when it receives a message.

[0101] The class that will directly interact with the user is the Publisher 850, which is a shared pointer encapsulation version of PublisherInner 840. Users can use this Publisher 850 class to publish a deep-copied message by calling void publish(Message msg). Alternatively, users can first manually encapsulate the message into a constant shared pointer and then publish a shallow-copied message pointer by calling void publishShallow(std::shared_ptr<const Message> msgPtr).

[0102] In simple terms, during development, users interact with the Node.IO class through user-defined callback functions and publisher-provided publish functions. These usages are described in more detail in the sections discussing the middleware Node.930 and Encryptor.710.

[0103] Figure 9 A block diagram illustrating the components of a node according to an embodiment of the present disclosure is shown.

[0104] Node 930 is the finest-grained or basic unit of computation in middleware. Node 930 inherits from a more basic class called node type 920, which implements the function int64_tsleep() at a given rate. Users most often interact with the following functions.

[0105] Function void subscribe(const string&topic, size_t qSize, void(NodeName::*callback)(std::shared_ptr <messageconst>&), NodeName* obj) instantiates a SubscriberInner object and binds the user-defined callback function to it, where the callback function has the input type std::shared_ptr<Message const>& and the output type void. Different from ROS, this function does not return anything to the user because the user will interact with the received messages through the callback function. Although ROS::NodeHandle::subscribe returns a ROS::subscriber to the user, the ROS::subscriber is basically useless, so the current middleware provides a different and advantageous method.

[0106] Function Publisher <message>The `Advertisement(const string&topic, size_tqSize)` method instantiates a `PublisherInner 840` object and wraps its shared pointer to a version of `Publisher 850`, which is then returned to the user.

[0107] The function `void bindProcess(void(NodeName::*process)(void), NodeName*obj, FunctionType type)` is a distinguishing feature that differentiates this middleware from ROS. Similar to subscription callbacks, `bindProcess` binds a user-defined function pointer (both input and output are void) to the three FunctionTypes: Initialize, Terminate, and Process.

[0108] If Initialize() is bound, the system will call the bound function pointer to correctly initialize it after the node is started. Users can use this function to open file streams or preload datasets.

[0109] If Terminate() is bound, the system will call the bound function pointer to properly terminate the node before closing it. Users can use this function to close file streams or free memory.

[0110] If `Process()` is bound, the system will call the bound function pointer at the same rate as the node it is bound to after all subscriber callbacks have been invoked. Users can use this feature to implement filtering (Kalman filtering, particle filtering as unrestricted examples), or as unrestricted examples of conditioning control, state machine transitions, topic publishing, etc.

[0111] Furthermore, Middleware 910 is also a class that inherits from NodeType 920. By default, Middleware 910 can run at 1000Hz as a non-limiting example, but this is just an exemplary default operation rate; other default values ​​are possible. Alternatively, users can manually set the operation rate of Middleware 910 by calling setRate(double). This Middleware 910 class is the central broker for all internal message pointer transfers. Therefore, only one Middleware 910 object needs to be declared in each main() function, and users can manually call the following functions to prepare and use Middleware 910.

[0112] The function `bool registerNode(string nodeName, Node*)` registers a user-defined node 930 into middleware 910, so that the node can be dynamically launched later using `1aunch()`.

[0113] The function `bool launch(const string nodeName)` starts a thread to include a node with the given `nodeName`. If the name is not already registered, `launch()` will return `false`.

[0114] The function `bool shutdown(const string nodeName)` shuts down and joins the thread containing the node with the given `nodeName`. Similarly, `shutdown()` returns `false` if the name is not already started.

[0115] The function `bool terminate()` shuts down each node thread associated with middleware 910.

[0116] The function void spinOnce() iterates through each publisher in middleware 910 and moves the messages they publish to the corresponding subscribers.

[0117] The function void sleep() will take computation time into account and sleep the middleware 910 accordingly to achieve the target rate.

[0118] Node handle 940 is a class that wraps node 930 with a shared pointer. The purpose of using node handle 940 is to simplify the user API. The use of node handle 940 is... Figure 11-15 Further discussion is provided in the flowchart shown.

[0119] Figure 10 A block diagram illustrating components of other features of the middleware according to embodiments of the present disclosure is shown.

[0120] Figure 10 Application class 1010 is shown. Such an application class provides users with a way to build middleware and use it to allow applications to invoke function calls.

[0121] Despite what is below Figure 15 The application class 1010 is further discussed in the text. Figure 10 This describes some other aspects of the middleware. For example, Figure 10 The display shows Health Monitor 1020, Anomaly 1030, Mathematical Tools 1040, String Tools 1050, and Transformation 1060.

[0122] Here, the health monitor 1020 is used to monitor node failures or any system anomalies. The monitor can then terminate and restart the node accordingly. The exception handler 1030 is a class that catches internal failures. The mathematical tool 1040 provides enhanced mathematical calculation processing, while the string tool 1050 provides enhanced string manipulation processing.

[0123] Transformer 1060 provides a class that offers various utility functions and is associated with ROS::tf and ROS:tf2. Here, tf is a package that allows users to track multiple coordinate systems over time. tf maintains the relationships between coordinate systems in a time-buffered tree structure and allows users to transform points, vectors, etc., between any two coordinate systems at any desired point in time. Therefore, Transformer 1060 provides similar features in the context of this embodiment.

[0124] In other aspects, such as real-time applications, if the goal is to stream robotic arm control commands or video / audio in real time, one can first refer to the IEEE Time-Sensitive Networking (TSN) definition and then implement a UDP server, UDP client, UDP SSL client, and UDP SSL client. Typically, UDP is used here because it can be used in applications where speed is more critical than reliability. For example, it might be better to use UDP in an application that provides real-time interaction between robot 210 and user 230. Here, losing some data points is acceptable as long as the goal of maintaining real-time interaction is met.

[0125] If the middleware is lightweight enough to run on low-level boards like STM32, then a CANTransceiver can be implemented. The Controller Area Network (CAN) bus is a powerful bus standard designed to allow microcontrollers and devices to communicate with each other's applications without a master. It's a message-based protocol originally designed for multiplexing electrical wiring in automobiles to save copper, but it can also be used in many other environments. For each device, data in a frame is transmitted sequentially, but if multiple devices transmit simultaneously, the highest priority device can continue transmitting, while other devices back off. Frames are received by all devices, including the transmitting device.

[0126] Other targets might include using YAML (an acronym for "YAML Ain't Markup Language"), a human-readable data serialization language. YAML is commonly used for configuration files and applications that store or transfer data. YAML targets many of the same communication applications as Extensible Markup Language (XML), but with a minimal syntax that is intentionally different from standard generalized markup languages. For example, a parameter server can be implemented using YAML-cpp.

[0127] Furthermore, the middleware can implement asynchronous server / client pairing. Additionally, it can implement a generic state machine pattern to replace ROS::action. As yet another feature, the middleware can implement a node state visualizer. For such a visualizer, node state can be health, location, failure, mobility, or other information about node operation that can be visualized in a useful way.

[0128] Figure 11 The illustration shows a flowchart of an exemplary method 1100 for implementing functions for an application using middleware on a computer, according to an embodiment of the present disclosure.

[0129] In step S1102, the method receives function calls to functions. For example, an application might want to call instructions to cause robot 210 to perform actions. For instance, an application might want to call a function such as `NavigateRobot()` to make robot 210 move in a certain way, `ManufactureArm()` to make robot 210 move its arm, `DetectObject()` to make robot 210 detect objects using sensors, `RecognitizeHuman()` to make robot 210 recognize people within its field of vision, or `Converse()` to make robot 210 perform dialogue, such as by acting as a chatbot. However, these are just examples and should not be considered limitations. The application can call other functions that robot 210 performs. These functions can also generate results returned to the application.

[0130] Before the application makes a function call, it is assumed that a class has already been defined for the application, such as robot navigation, robotic arm manipulation, object recognition, etc. More details about the application's call definition are discussed below. Figure 15 Describe it.

[0131] In step S1104, the method initializes the application node by binding a function and associating that function with the encryptor. Combined with... Figure 15 The initialization process is described in more detail. Typically, initialization involves declaring the application node using the node handle variable 940, binding functions, and establishing a publisher relationship with the cipher that prepares the application to define how the application and remote devices (such as robot 210) can communicate.

[0132] In step S1106, the method uses a cryptographic handshake to connect the middleware to the remote device via the transport layer. As described above, each cryptographic device has a name associated with it, and the remote cryptographic device checks whether the cryptographic device's name is already registered. Once the remote cryptographic device confirms registration, the handshake is successful, and the middleware connects to the cryptographic device.

[0133] In step S1108, the method publishes a message (also called a published message) to invoke a function in the encryptor. Once the application node is bound to the function and associated with the encryptor, the method publishes the published message to the encryptor. For example, using a publisher-subscriber relationship, middleware can pass messages to the encryptor.

[0134] In step S1110, the method uses an encryptor to communicate with a remote device based on the published message. Specifically, the encryptor automatically serializes the published message to generate a serialized message, and sends the serialized message through the transport layer in response to the published message received by the encryptor from the application node.

[0135] In step S1112, the method uses an encryptor to receive a message from a remote device and returns the message to the function via middleware. Step S1112 may include: receiving a remote message from the remote device via the transport layer using an encryptor, and publishing the remote message via middleware for the function and application node to receive. Step S1112 may also include: automatically deserializing the remote message to generate a deserialized message, and sending the deserialized message via middleware in response to the encryptor receiving the remote message from the remote device via the transport layer.

[0136] Figure 12 A flowchart illustrating an exemplary method 1200 for initializing and using middleware according to an embodiment of the present disclosure is shown.

[0137] In step S1202, the method declares the application node object. As mentioned above, this is done by declaring the node handle variable 940, which is... Figure 15 Step S1502 is further described in the text.

[0138] In step S1204, the method initializes the application node object. This is performed through binding functions, preparing the publisher, etc. Figure 15 Steps S1504, S1506 and S1508 are further described.

[0139] In step S1206, the method declares an encryptor object. Typically, this step involves initializing the transport layer, constructing the encryptor using the transport layer, and invoking various aspects of the encryptor, as described above. Figure 7 The following is discussed in [the text]. Figure 13 The method is further described in S1206.

[0140] In step S1208, the method obtains a unique global middleware pointer. For example, this is achieved by calling `shared_ptr`. <middleware>getMiddlewarePtr() retrieves the pointer to the global middleware 910.

[0141] In step S1210, the method registers the cipher and application nodes using a global middleware pointer. For example, the method calls Middleware::registerNode(...) to register each cipher and application node. However, registration does not apply to nodes that use the node handle variable 940, because when NodeHandle::registerNode(...) is called, the function automatically registers the node to a unique global middleware object.

[0142] In step S1212, the method launches the node and the encryptor. To perform such a launch, the middleware calls Middleware::launch(...). This launch spawns a thread with a default thread function to contain the application node.

[0143] In step S1214, the method calls `spinOnce()`. Specifically, step S1214 invokes `Middleware::spinOnce()`. As mentioned above, `spinOnce()` iterates through each publisher and moves its published messages to the corresponding subscribers. Therefore, the role of `spinOnce()` is to ensure that each time it is called, the message is propagated from the publisher to the subscriber in the middleware. Since the preceding parts of the middleware have already established the structure and path that these messages should follow, step S1214 ensures that the messages are actually successfully published and received.

[0144] In step S1216, the method calls `sleep()`, then loops back to step S1216 to perform more operations in an infinite while loop. Specifically, step S1216 calls `Middleware::sleep()`. As mentioned above, `sleep()` ensures that computation time is considered to produce the target rate. At this point, within the method, certain nodes can also be programmed to dynamically shut down or restart as needed by calling `Middleware::shutdown(...)` and `Middleware::launch(...)`.

[0145] Figure 13 A flowchart of an exemplary method 1300 for initializing and managing an encryptor according to an embodiment of the present invention is shown.

[0146] In step S1302, the method initializes the transport layer class. For example, initializing the transport layer class includes obtaining a shared pointer to the transport layer class.

[0147] In step S1304, the method passes a shared pointer of the transport layer class to the cipher constructor. This shared pointer is used to construct the cipher in the constructor call, thereby establishing an initial relationship between the cipher and the transport layer.

[0148] In step S1306, the method calls functions to define the cryptographer node. For example, the method accordingly calls Cryptor::register-Platforms(...), Cryptor::subscribe(...), Cryptor::advertise(...), and Cryptor::setMiddlewareName(...).

[0149] In step S1308, the method manually disconnects the remote connection as needed. For example, the method calls Cryptor::disconnect(...) for a given cipher.

[0150] Figure 14 A flowchart illustrating another exemplary method 1400 for initializing and using middleware according to an embodiment of the present disclosure is shown.

[0151] In step S1402, the method uses a shared pointer of the remote device to declare the encryptor. As described above, step S1402 is related to... Figure 13 correspond.

[0152] In step S1404, the method registers the encryptor and application node to the global pointer. Step S1404 corresponds to S1210.

[0153] In step S1406, the method defines publisher nodes and subscriber nodes. For example, in step S1406, the nodes and encryptors can invoke subscribe(...) and / or advertise(...). The method can also define publisher 850, subscriberInner 830, publisherInner 840, subscriber encryptor 720, and publisher encryptor 730 objects for sharing messages through middleware.

[0154] In step S1408, the method starts the application node or the encryptor. Step S1408 corresponds to S1212.

[0155] In step S1410, the method iterates through the publisher nodes and moves the message to the subscriber nodes. Step S1408 corresponds to S1214. Figure 14 In addition, steps corresponding to S1216 can be performed such that when the method moves a message from the publisher node to the subscriber node, the method includes synchronization steps to help the method maintain a specific running rate.

[0156] Figure 15 A flowchart illustrating an exemplary method 1500 for initializing an application node according to an embodiment of the present disclosure is shown.

[0157] In step S1502, the method declares a node handle variable 940. As described above, the middleware defines a class for the specified application before initializing the application node. The node handle variable 940 can be declared as public or private. The name and rate of the node variable are initialized by calling NodeHandle::Initialize(...) as a constructor. By declaring the node handle variable, it provides the application with a way to make application calls using the middleware's application programming interface (API), which allows the user 230 to conveniently remotely call functions at robot 210.

[0158] In step S1504, the method binds callback functions. Binding is performed by calling NodeHandle::subscribe(...) for each callback function. By performing the binding, it allows the application to pass callback functions, facilitating function calls.

[0159] In step S1506, the method declares and initializes the publisher. The method declares the necessary publisher variables and calls NodeHandle::advertise(...) to initialize them. By taking these steps, it prepares the middleware so that when a message is ready to be published, Publisher::publish(...) or Publisher::publishShallow(...) can be called accordingly.

[0160] In step S1508, the method may optionally define a process function that will be called at the same rate as the node, and call NodeHandle::bindProcess(...).

[0161] Another aspect of this disclosure relates to a non-transitory computer-readable medium storing instructions that, when executed, cause one or more processors to perform the methods described above. The computer-readable medium may include volatile or non-volatile, magnetic, semiconductor, magnetic tape, optical, removable, non-removable, or other types of computer-readable media or computer-readable storage devices. For example, as disclosed, the computer-readable medium may be a storage device or memory module on which computer instructions are stored. In some embodiments, the computer-readable medium may be a disk or flash drive on which computer instructions are stored.

[0162] It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed systems and related methods. Other embodiments will be readily apparent to those skilled in the art, given the detailed description and practice of the disclosed systems and related methods.

[0163] The specification and examples are intended to be illustrative only, and the true scope is indicated by the claims and their equivalents.

[0164] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.< / middleware> < / message> < / messageconst> < / string>

Claims

1. A method for implementing functions for an application using middleware on a computer, characterized in that, include: The application receives a function call to the function from the application, initializes the application node corresponding to the application in the middleware on the computer, the initialization includes binding the function to the application node and associating the application node with the encryptor, wherein the function is used to make the robot perform the corresponding action, and the encryptor is a node with encryption function; Use this encryptor to connect the middleware to the robot via the transport layer; In response to the received function call, a message is published to invoke the function from the application node to the cipher via the middleware; and According to the released message, the encryptor is used to communicate with the robot so that the robot can remotely call the function to make the robot perform the corresponding action; The robot is equipped with a remote encryptor and a remote middleware name. The encryptor is registered locally on the computer with an array of remote middleware names associated with it. Communicating with the robot using the encryptor includes: exchanging middleware names and comparing the exchanged remote middleware names with the array of remote middleware names registered locally on the computer, and performing a handshake between the encryptor and the robot's remote encryptor.

2. The method according to claim 1, wherein, Binding this function involves establishing one or more subscriber or publisher relationships between the function and the application node.

3. The method according to claim 1, wherein, The method further includes: in response to the encryptor receiving the published message from the application node, automatically serializing the published message to generate a serialized message and sending the serialized message through the transport layer.

4. The method according to claim 1, further comprising: The encryptor is used to receive remote messages from the robot through the transport layer and to publish remote messages to be received by functions at the application node through the middleware.

5. The method according to claim 4, wherein, The method further includes: in response to the encryptor receiving the remote message from the robot through the transport layer, automatically deserializing the remote message to generate a deserialized message and sending the deserialized message through the middleware.

6. The method according to claim 1, wherein, The middleware includes at least one additional application node and at least one additional encryptor, and the method further includes: exchanging published messages between the additional application node and the additional encryptor through the middleware to implement functions for the additional application node.

7. A system for implementing functions for an application using middleware on a computer, characterized in that, include: At least one processor; as well as A memory that stores instructions, which, when executed by the at least one processor, enables the at least one processor to: The application receives a function call to the function from the application, initializes the application node corresponding to the application in the middleware on the computer, the initialization includes binding the function to the application node and associating the application node with the encryptor, wherein the function is used to make the robot perform the corresponding action, and the encryptor is a node with encryption function; Use this encryptor to connect the middleware to the robot via the transport layer; In response to the received function call, a message is published to invoke the function from the application node to the cipher via the middleware; and According to the released message, the encryptor is used to communicate with the robot so that the robot can remotely call the function to make the robot perform the corresponding action; The robot is equipped with a remote encryptor and a remote middleware name. The encryptor is registered locally on the computer with an array of remote middleware names associated with it. Communicating with the robot using the encryptor includes: exchanging middleware names and comparing the exchanged remote middleware names with the array of remote middleware names registered locally on the computer, and performing a handshake between the encryptor and the robot's remote encryptor.

8. A non-transitory computer-readable medium storing instructions, which, when executed by at least one processor, cause the at least one processor to perform a method for implementing functions for an application using middleware on a computer, characterized in that, The method includes: The application receives a function call to the function from the application, initializes the application node corresponding to the application in the middleware on the computer, the initialization includes binding the function to the application node and associating the application node with the encryptor, wherein the function is used to make the robot perform the corresponding action, and the encryptor is a node with encryption function; Use this encryptor to connect the middleware to the robot via the transport layer; In response to the received function call, a message is published to invoke the function from the application node to the cipher via the middleware; and According to the released message, the encryptor is used to communicate with the robot so that the robot can remotely call the function to make the robot perform the corresponding action; The robot is equipped with a remote encryptor and a remote middleware name. The encryptor is registered locally on the computer with an array of remote middleware names associated with it. Communicating with the robot using the encryptor includes: exchanging middleware names and comparing the exchanged remote middleware names with the array of remote middleware names registered locally on the computer, and performing a handshake between the encryptor and the robot's remote encryptor.

Images