Robot programming system for terminal user
Through a robot programming system for end users, using voice interaction and localized programming, the problems of high difficulty in user programming, single functions and strong network dependence in the existing technology are solved, and multiple scenarios with user-defined functions and cost-effectiveness are realized. application.
Patent Information
- Application Number
- CN202411417450.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-11
- Publication Date
- 2024-12-03
AI Technical Summary
The existing robot operating system is not convenient for users to participate in programming, has a single function and cannot be customized according to user needs, and it is difficult to meet diverse daily needs, and it is difficult to obtain large model training data, there are hallucinations problems and strong network dependencies, which are restricted. The flexibility and reliability of robots in multi-scenario applications.
Provide a robot programming system for end users, using interactive units, hardware access units, batch task planning execution units and learning units, allowing users to directly program through voice interaction, supporting users to customize tasks, reducing dependence on the network, and realizing Run locally.
Implement user-defined functions, reduce the emergence and hallucination problems of large models, improve the flexibility and reliability of robots in multi-scene applications, enhance user engagement, simplify the programming process, improve performance and operation reliability .
Smart Images

Figure CN119057792A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of robot technology, and in particular to a robot programming system for terminal users. Background Art
[0002] With the continuous reduction of robot hardware costs and the continuous iteration and maturity of related technologies, the current robot industry has made great progress. Coupled with the amazing ability of large language models in language interaction, it is expected to give the robot body a higher thinking ability. Nowadays, various types of robots have appeared in people's vision, especially service robots, as an important category of robots, are more used in commercial and family scenarios. For example, they have been used or are about to be put into use in scenarios such as on-site customer service Q&A, content explanation and guidance, hotel catering services, as well as housework and elderly care.
[0003] The key technologies of service robots mainly include the brain, cerebellum, limb movements, load and endurance. The robot's operating system can be regarded as the brain part, which is an important part supporting the intelligent operation of the robot. Most of the operating systems currently used by robots are ROS2 evolved from ROS1, and there are also robot manufacturers who have developed their own operating systems. Although it is an important hub connecting robot software and hardware, and plays an extremely important role in various functional applications of robots, there are still two main problems with existing robot operating systems:
[0004] First, it is not convenient for users to participate in programming. The existing robot operating system is still at the stage similar to the DOS operating system of early computers. It is mainly for programming engineers with a certain programming language foundation. Moreover, it is difficult for programming engineers to modify and maintain existing programs. In this case, it is extremely difficult for ordinary users to let the robot perform functions according to their wishes (users program themselves).
[0005] Secondly, most existing robots have a single function and cannot achieve multi-functionality. The functions of existing robots are all solidified programs before leaving the factory, and their functional capabilities are relatively limited. Manufacturers will not and cannot customize and develop for each user's special needs. At present, most user needs point to general robots that can perform corresponding functions according to the user's wishes. Users can design and adjust their functions after purchasing. Users hope that it can help them do more daily things, such as washing clothes, cooking, cleaning the room, etc.
[0006] In response to the above problems, both the robotics academic and industrial communities have done a lot of work. The current mainstream research direction is to strive to realize a large robot model, hoping that by providing the robot with sufficient training data, a robot model with strong generalization ability can be trained on the basis of a certain computing power, so that it can be adapted to various application scenarios. However, this technical route has also encountered a series of difficulties: (1) It is still uncertain whether the emergence phenomenon in the large language model training can appear on the robot large model, and the effect of the corresponding function cannot be guaranteed at present. (2) It is currently very difficult to obtain data for robot training. Robot behavior data cannot be easily obtained on the Internet like large language models, and the number of robots serving in real scenarios is originally small, so it is difficult to form effective training data. (3) The robot large model cannot solve the hallucination problem, because as long as the robot large model is essentially based on the probabilistic neural network node data for calculation, the hallucination problem of the large model will always exist, which will seriously affect the application of robots in some scenarios. (4) The robot model may not be able to directly meet the personalized requirements of users, because the results of training on the training data are more inclined to commonality, and users need personalized robots rather than common robots. For household robots, even for the same function, the scenarios and needs of each household may be different, and even the needs in the morning may change in the evening. It is difficult for the robot model to effectively and in real time adapt to the personalized needs of all users. (5) At present, the large model is still insufficient in reasoning ability and interpretability. Although the large model has a certain reasoning ability, its performance often deteriorates seriously when it is used in long-chain reasoning. When the current black box large model cannot provide effective interpretability for the output results, it will be limited when it is used in scenarios with high requirements for reasoning ability and interpretability. (6) Existing robots all have the problem of users being unable to participate or having limited participation, because the system itself does not open a window for users to participate. Users do not have effective communication channels and naturally there is no communication result; and once users find errors during use, it is difficult to modify them in the first time, missing the best opportunity to improve the robot. (7) The local operation of large robot models faces challenges. Currently, the operation of most large models depends on servers in the cloud. Although this can solve the problem of insufficient local computing power, it is difficult to ensure that the robot can work well without a network. How to reduce the dependence on the network as much as possible is of great significance to the economy, reliability and data security of robot operation. This has always been an important research topic in the development of robots. In short, there is still a lot of uncertainty and the risk of waiting longer in the current research direction of large robot models. Summary of the invention
[0007] In view of the shortcomings of the robot technology driven by large models proposed above, the present invention provides a robot programming system for end users. The present invention adopts a completely different technical route from the large model technology and provides a human-computer interaction method for ordinary users to participate in robot programming. It can effectively meet the personalized needs of ordinary users for robot functions.
[0008] The technical means adopted by the present invention are as follows:
[0009] A robot programming system for end users, comprising:
[0010] An interactive unit, the interactive unit is used to provide an interactive interface between the robot and the user, the interactive interface includes an interactive main interface and a program setting interface, the program setting interface is used to select a default task name for the interactive main interface, start the execution and debugging of the selected task in response to the user's interactive action, start adding a task, editing a task and deleting a task in response to the user's interactive action, and jump to the voice interactive learning interface;
[0011] A hardware access unit, the hardware access unit is used to access the robot's embodied hardware according to the robot's own program and the program written by the user, the robot's embodied hardware includes a sensor module for the robot to sense external information and a drive module for controlling the robot's execution component actions;
[0012] A batch task planning execution unit, the batch task planning execution unit is used to plan multiple subtasks to be executed at different specified times to be started by calling the batch task name only once;
[0013] A learning unit is used to learn user input sentences and corresponding responses through voice interaction.
[0014] Furthermore, learning the user input sentence and the corresponding response through voice interaction includes:
[0015] In the program setting interface, in response to the user's interactive action to start the learning function, the voice interactive learning interface is entered, and the voice interactive learning interface includes a [Start / Stop Learning] button and an [Exit] button. The [Start / Stop Learning] button is used to set whether the current mode is the learning mode or the question-and-answer mode after learning, and the [Exit] button is used to exit the current learning interface;
[0016] After the user triggers [Start / Stop Learning] to enter the learning mode, the learning content is output to the robot through voice, and information interaction is achieved in combination with the robot's feedback voice. The learning content includes question and answer content, action content, task content, picture content and video content.
[0017] Furthermore, learning the user input sentence and the corresponding response through voice interaction also includes:
[0018] An input question Q that converts a speech signal into text information;
[0019] Questions are matched one by one in the question answer QA library. When the match is successful, the response body content corresponding to the question is output. The response body is a combination of multimodal information used to define the response result. Each input question Q corresponds to a response body. The question answer QA library is used to store the input question Q and the corresponding response body.
[0020] Furthermore, learning the user input sentence and the corresponding response through voice interaction also includes:
[0021] If the entire QA library fails to match, the process jumps to calling the large model to perform a generalized matching operation on the user input question Q. When the generalized matching score is lower than the set threshold, the user is asked to confirm by asking a question in reverse.
[0022] Furthermore, the program setting interface is used to start adding tasks, editing tasks, and deleting tasks in response to user interaction actions, including:
[0023] In response to the user triggering the [Add] button on the program setting interface, a new task dialog box pops up, and the user completes the new task by interacting with the information in the new task dialog box;
[0024] In response to the user triggering the [Edit] button on the program setting interface, an edit task dialog box pops up, and the user completes the user task editing by interacting with the information in the edit task dialog box;
[0025] In response to the user triggering the [Delete] button on the program setting interface, a delete task dialog box pops up, and the user completes the deletion task by interacting with the information in the delete task dialog box.
[0026] Furthermore, when the user writes a task through the task editing dialog box, the user uses the native language for programming.
[0027] Furthermore, the hardware access unit includes a first interface adapter, which converts different functions of different robot manufacturers for the same action into a unified function expressed in the user's native language by searching a first dedicated conversion interface library.
[0028] Furthermore, the system also includes an external resource access unit, and the external resource access unit is used to access third-party resources according to an external interface.
[0029] Furthermore, the external resource access unit includes a second interface adapter, and the second interface adapter uniformly converts external resources from a third party into a unified function expressed in the user's native language by searching a second dedicated conversion interface library.
[0030] Furthermore, the system also includes a wireless communication unit, and the wireless communication unit is used to perform program writing and debugging with a PC or a PAD in a wireless manner.
[0031] Compared with the prior art, the present invention has the following advantages:
[0032] The present invention enables users to customize robot functions through direct programming by users, and can avoid the emergence problem and illusion problem of existing robot large models. The present invention is easy to realize multi-functionality and high cost performance of the robot. By executing different programs written by users, different intelligent tasks of the robot are completed, and the hardware computing power requirement is not high. Therefore, it is easier to meet the personalized needs of users for robots. Users can write designated tasks in their native language, which makes it easier to realize the personalized functions of the product.
[0033] The present invention is easy to establish a mapping between user voice and robot behavior. A large robot model needs to be trained with a large amount of scene data to establish a rich voice-robot behavior mapping. Scene data is not easy to obtain, and even the common results obtained through training may not meet the same scene. The present invention can quickly establish a user's own unique voice-robot behavior mapping at any time through a "learn and use" method.
[0034] The present invention allows users to easily participate in robot programming. By opening up a DIY programming window and programming in the user's native language, as well as using Python, an easy-to-learn and easy-to-use programming language, ordinary users can easily participate in robot programming, solving the problem of difficult user participation in existing robot systems. The programming results are easy to understand and spread, so that users can more easily maintain the growth of the robot.
[0035] The present invention can help improve the performance of robot agents. At present, some agents are affected by the performance of large models after being written directly by users in natural language. To further improve the performance, users need to use special programming languages to improve and refine, but ordinary users are often unable to do so. In the present invention, a task written by the user is equivalent to an agent, and can fully manage the robot's embodied hardware, and can also call external API resources and even large models, and all these operation processes can be implemented by users programming in their native language.
[0036] The present invention makes it easier for the robot intelligent operating system to run locally. Because it adopts a programming mode rather than a massive neural network node calculation mode, except for a few necessary functions that require network dependence, most of the work of the entire robot can be run locally, reducing dependence on network cloud servers. This also improves operational reliability and reduces operating costs.
[0037] The present invention provides users with a special robot interface library, so that the robot system can be compatible with the call of various large models and various external knowledge bases, so as to better use mature external resources to further improve the intelligence level of the robot. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0039] Figure 1 This is a schematic diagram of the main interface of a robot programming system for end users of the present invention.
[0040] Figure 2 The present invention is a schematic diagram of a program setting interface of a robot programming system for end users.
[0041] Figure 3 A new task interface is added to a robot programming system for end users of the present invention.
[0042] Figure 4 A task name interface is newly added to a robot programming system for end users of the present invention.
[0043] Figure 5 The present invention discloses a robot programming system editing task interface for terminal users.
[0044] Figure 6 The present invention discloses a robot programming system deletion task interface for terminal users.
[0045] Figure 7 The present invention discloses a task name deletion interface of a robot programming system for terminal users.
[0046] Figure 8 This is a programming example of a robot programming system for end users of the present invention.
[0047] Fig. 9 The present invention is a schematic diagram of the workflow of a first interface adapter of a robot programming system for end users.
[0048] Fig.10 This is an example of batch task programming of a robot programming system for end users of the present invention.
[0049] Fig.11 The present invention is a batch task execution flow chart of a robot programming system for end users.
[0050] Fig.12 The present invention is a schematic diagram of the workflow of a second interface adapter of a robot programming system for end users.
[0051] Fig.13 This is a communication connection diagram of a wireless communication unit of a robot programming system for end users of the present invention.
[0052] Fig.14 The present invention is a communication flow chart of a wireless communication unit of a robot programming system for end users.
[0053] Fig.15 The present invention discloses a question answer QA library data structure for a robot programming system facing end users.
[0054] Fig.16 The present invention is a flowchart of processing user input sentences and responses of a robot programming system for terminal users.
[0055] Fig.17 The present invention provides a learning interface for terminal users to input sentences and responses of a robot programming system.
[0056] Fig.18 The present invention is a learning flow chart of a robot programming system for terminal users, which includes user input sentences and responses. DETAILED DESCRIPTION
[0057] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0058] The present invention provides a robot programming system for terminal users, including an interaction unit, a hardware access unit, a batch task planning and execution unit, an external resource access unit, a wireless communication unit and a learning unit. In this application, the Python robot program written by the terminal user in his own native language is collectively referred to as a task.
[0059] The interaction unit is used to provide an interaction interface between the robot and the user, and the interaction interface includes an interaction main interface and a program setting interface. The program setting interface is used to select the default task name for the interaction main interface, start the execution and debugging of the selected task in response to the user's interaction action, start adding tasks, editing tasks and deleting tasks in response to the user's interaction action, and jump to the voice interaction learning interface.
[0060] Specifically, if Figure 1 As shown in the figure, a schematic diagram of the main interface is given. The top of the interface is the toolbar area, including the image logo, product name, task name, and the [Status] indicator light for the running status, as well as the [Voice] indicator light that allows voice input; the [Run] and [Stop] buttons for operating the task; the [Exit] button for exiting the system operation, the [Input Content] dialog box for typing input, and the [Dialogue] button for operation after inputting content. The middle part is the robot facial expression / picture / video multi-function display area, which is used to display the robot's facial expressions and also display information such as pictures and videos; the bottom is the print display area for running information, which displays relevant output information during operation.
[0061] Main interface related button operations
[0062] When the status light is green, it indicates that the task can be executed; when it is blue, it indicates that the task is being executed; when it is red, it indicates that the currently executed task has been stopped; when it is yellow, it indicates that the execution time of the subtask is waiting to arrive when executing a batch task.
[0063] When the status light is green, click the [Run] button, the currently selected task (the task name displayed in the task name area) starts to execute, the status indicator light is blue, and the information bar at the bottom of the interface displays the content and time of the task being executed.
[0064] During the execution of a task, click the [Stop] button to stop the current task execution. The status indicator turns red, and the information bar at the bottom of the interface displays the content and time of the currently stopped task.
[0065] After the user inputs a language input sentence by voice, the recognized content is displayed in the "Input Content" window, and the matching response to the language input sentence is output. If it is a question and answer, the answer is directly displayed in the "Operation Information Print Display Area" and announced by voice at the same time; if it is to execute an action or task, the action and task (user-programmed content) are executed directly.
[0066] When it is inconvenient for the user to input by voice, he can manually type the content in the "Input Content" window and click the [Dialogue] button to output the matching response of the language input sentence. The process is the same as the content described above by voice input.
[0067] Users can directly enter the program setting interface by clicking the "Product Name" title.
[0068] The execution process of the program setting page can refer to the working process of the above main interface.
[0069] The program setting interface has a total of 4 functions: 1) select a default task name for the interactive main interface from the task list of the interface; 2) execute and debug the task name selected by the user in the interface, and the execution process is exactly the same as the description of the task execution operation in the interactive main interface; 3) write the user's own exclusive tasks in the interface, that is, user DIY programming, including operations such as adding tasks, editing tasks, and deleting tasks; 4) jump to the learning interface, and the user uses voice interaction to let the robot learn the content and related execution of the response to the user's voice input sentence.
[0070] Specifically, when executing functions 2) and 3) in the program setting interface, click the title "Product Name + Interface Name". After the pop-up dialog box for entering the management permission password appears, the buttons will only be displayed and operated after the correct password is entered, otherwise the relevant buttons will be hidden. Because the user's task content and learning content are very important information, outsiders without permission are not allowed to change them at will to avoid losses caused by wrong changes.
[0071] Program settings interface layout, such as Figure 2 . The interface continues to use the layout style of the main interface, but the area of different areas has been adjusted. The upper part is the toolbar area, including the product name + interface name, the [Status] indicator light that shows the running status, and the [Voice] indicator light that allows voice input; the [Run], [Stop], and [Pause] buttons for operating task operations; the [Back] button to return to the main interface, the [Input Content] dialog box for typing content, and the [Dialogue] button for operations after entering content. The middle part is mainly used to display the robot's facial expressions, and also displays information such as pictures and videos; the lower part is used to display printout information during operation. This interface has added some new content, and the [Add], [Edit], [Delete], [Learn], and [Extend] buttons for user DIY programming operations have been added to the far left. In addition, a task name list has been added to indicate the task names that already exist in the current robot environment.
[0072] The [Learn] button is used for on-site voice interactive learning to let the robot learn what the robot should respond to when the user inputs a sentence, including answering, performing actions and executing tasks, etc. The [Extend] button is used to jump to other extended function operations. The [Exit] button is used when you click to select a specified task name in the task name list. After clicking this button to exit to the main interface, the selected task name will be the default task name selected for the main interface.
[0073] Click the [Add] button in the program setting interface, and a dialog box "Please enter the name of the new task" will pop up. After entering the task name, for example, if you enter "Task Name 100", click the [Confirm] button to add the newly entered task name to the current task name list column. Figure 3 and Figure 4 .
[0074] In the program setting interface, select the task name to be edited, for example, select "Task Name 100", and then click the [Edit] button to enter the display interface where the task content can be edited using the soft keyboard. Figure 5 . When editing is completed, click the [Finish] button to complete the editing operation.
[0075] In the program setting interface, select the task name to be deleted, for example, select "Task Name 100", and then click the [Delete] button, and a dialog box "Are you sure you want to delete task: XXX?" will pop up. Figure 6 When confirming the deletion operation, click the [Confirm] button to complete the task deletion operation. Figure 7 The content in the dotted box in the figure indicates that it has been deleted and does not exist.
[0076] In order to allow users to easily write programs, it is undoubtedly the most convenient to choose the native language that the user is familiar with. At the same time, the style of the programming language should be simple, easy to learn, and quickly mastered. The new version of Python3 itself supports programming in dozens of languages, such as Chinese, Japanese, Korean, French, German and other languages. The variables and function names of the programming language can be directly named in the native language. At the same time, it is recognized as the most suitable programming language for beginners to master. In this way, ordinary users do not have to be limited to the language of the programming language, but only need to focus on the logical implementation of the program. The present invention selects this programming language as the user's target programming language, so that the user can directly use his or her native language for programming. A program sample using the Chinese language as an example is shown in Figure 8 .
[0077] It should be noted here that Figure 8 The "r" in the system is the abbreviation of the special interface library name in the present invention. The "r" comes from the first letter of "robot". All functions or attributes applicable to the r library are represented by "r.", such as "r.speech (content)",
[0078] "r. Sound playing (song name)", etc. The meaning of "r." appearing in the subsequent explanation is the same, so I will not repeat it here.
[0079] The hardware access unit is used to access the robot's embodied hardware according to the robot's own program and the program written by the user. The robot's embodied hardware includes a sensor module for the robot to sense external information and a drive module for controlling the robot's execution component actions.
[0080] Specifically, each robot has different embodied hardware, such as position sensors, temperature sensors, torque sensors, etc., which are used to receive external sensor information, as well as drivers for the robot's head, arms, legs, torso, etc. to perform actions. These embodied hardware are accessed through robot operating system instructions. Traditional embodied hardware access is implemented by writing function programs in English, and because each robot is different, the related instruction functions are different and do not have unified compatibility. When the type and model of the robot is changed, the previously used access functions are no longer applicable. If users are required to master them separately, it is both laborious and uneconomical. To this end, the present invention converts the same content but different description instructions of different robot manufacturers into a unified interface description instruction by establishing an interface adapter. That is, the present application designs a first dedicated conversion interface library, which includes conversion functions for conventional and extended robot actions, such as raising and lowering the head in the vertical direction, turning left and right in the horizontal direction of the head; horizontal and vertical movement of the left and right arms; taking, putting down and rotating the left and right hands; moving the whole body forward, backward, turning left, turning right and stopping; facial expressions such as smiling, angry, shy, and naughty; raising, putting down and rotating the left and right legs. As long as the user masters the unified interface description instruction, he can deal with various types of robots supported by the interface adapter. At the same time, the interface description instruction is written in the user's native language, so that the user only needs to master this interface description instruction to access the embodied hardware of multiple robots, realizing a one-stop access, and no longer worrying about the original access function being written in English, which greatly facilitates the user's programming and naturally increases the user's extensive participation. In addition, this also makes the program written by the user more applicable to multiple robots, reducing the user's development and maintenance costs. For example, the same "raise left upper arm" action is converted to the action functions of different robot manufacturers into a unified interface description of "r.Left upper arm raised (X, Y)", see Fig. 9 .
[0081] It should be noted here that by continuously tracking the hardware upgrades of different robot manufacturers, we can sustainably support more types of robot manufacturers, thereby continuously improving the performance of robots.
[0082] The batch task planning and execution unit is used to plan multiple subtasks to be executed at different specified times so as to be started by calling the batch task name only once.
[0083] Specifically, in actual operation, sometimes it is necessary to execute multiple tasks at different specified times by executing the task name only once. This way of executing tasks can make the robot a more valuable service expert. The present invention uses batch task processing (task names prefixed with "%" are defined as "batch tasks", and of course they can also be defined in other ways). The content of the batch task is a regular task name list, which contains the specified task name and its start execution time. The present invention calls the tasks in the list content of the batch task subtasks. For an example of the content description of a batch task, see Fig.10 , related flowchart, see Fig.11 .
[0084] Get the batch task periodic loop execution setting instruction "__ loop execution to__" (the first line of valid instructions), which is used to specify the periodic execution requirements of the current batch task. It can specify the current batch task to be executed by number, such as "__ loop execution to__2 times"; it can also specify the current batch task to be executed according to the deadline, such as "__ loop execution to__2024-8-31 / 0:00:00", that is, it will be executed until 0:0:00 on August 31, 2024 and will no longer be looped.
[0085] Get the name and execution time of the first subtask, such as "Make breakfast @-- / 5:20:00&&Late: Still execute" in the figure. The subtask name is "Make breakfast", and the start execution time is data led by "@", such as "-- / 5:20:00", which means it will start at 5:20:00 every day.
[0086] It should be noted here that the data date data "--" does not specify a date, indicating that it will be repeated every day. If a date is specified, such as "2024-8-28 / 5:20:00", it means that it will only be executed at the specified time on August 28, 2024. For data without a specified execution time, such as "-- / ::" or without writing this data by default, the execution time is immediate. The data led by "&& later:" is a processing strategy when the time when the task is started is later than the scheduled execution time at that time. During the robot's execution process, due to various reasons, the execution of the previous task may be delayed beyond the start execution time of the subsequent task, which will cause the subsequent tasks to be executed later. To this end, three late execution strategies are added: Late: Still execute (start execution even if it is late), Late: Do not execute and skip directly (skip the current delayed task and execute the task after it), Late: Send notification and wait for confirmation (send a notification to the user, including a voice prompt, and the user decides how to execute the current delayed task, either still execute the current delayed task or skip the current delayed task).
[0087] Execute according to the start execution time of the current subtask. If the time has not arrived, wait. If it is late, execute according to the specified delay strategy.
[0088] After executing one subtask, obtain the name and start time of the next subtask in sequence, and continue to execute according to the above steps until all subtasks in the batch task are completed.
[0089] The external resource access unit is used to access third-party resources according to the external interface.
[0090] Specifically, with the continuous development of robot technology, different technologies will appear in different periods. Various large model technologies and other emerging technologies will become more and more mature through continuous iterative development, and the cost will become lower and lower. These have injected strong vitality into the development of the robot industry. Although the present invention is essentially based on programming, it is not "only programming is respected, and other things are ignored". On the contrary, various new technologies that are currently available and will appear in the future can be fully called through the convenience of programming, so that there will be long-term development. For example, for the effective "manipulator grasping" function module currently obtained through various large model technologies or other technologies, we only need to add an interface instruction to achieve it, such as the instruction "r. Grab (XX target)". Similarly, for other mature functional modules, such as knowledge graph modules, or video production modules, etc., they can be introduced by continuously upgrading interface instructions and calling third-party tools, so that the intelligence of large models and other intelligent resources can be fully utilized to allow robots to do more valuable things. This application designs a second dedicated conversion interface library, which includes conversion functions for robots to call various external resources, such as module interfaces for speech recognition and synthesis, module interfaces for image and video recognition and synthesis, access and call interfaces for various external large models; external weather forecast class call interfaces; external news video resource call interfaces; external service class information call interfaces, etc. For related schematic diagrams, see Fig.12 .
[0091] The wireless communication unit is used for wireless programming and debugging with a PC or PAD.
[0092] Specifically, programming can be done using the robot's own screen, but compared to using a large PC screen and mouse and keyboard, touch buttons are not very suitable for efficient programming. Moreover, when debugging a program, if the robot itself has manipulator actions to perform, there may be a risk of harm to people. Therefore, programming or debugging on a PC or PAD through a wireless connection is very convenient and safe. To this end, the present invention specifically provides a function that supports debugging on a PC and PAD. This function supports synchronization of local and external devices for editing files, supports full-speed operation, single-step debugging, breakpoint operation, and execution of Python statements (statements sent from external devices), and can be used to access function variable values during debugging. See the schematic diagram of the communication connection. Fig.13 , the relevant flow chart can be seen in Fig.14 .
[0093] The learning unit is used to learn user input sentences and corresponding responses through voice interaction.
[0094] Specifically, user input sentences and their responses refer to when a user inputs a sentence through voice, which may be a question or a statement. For the convenience of narration, "question Q" will be used instead of "user input sentence" in the following text. The robot has behavioral responses to it, including text, voice, video, action and task. Due to different input sentences, its response is also different. For knowledge questions and answers or chat dialogues, just text and voice output is enough, but for some input sentences, such as asking the robot to perform a task, it is not as simple as text output. The response of some input sentences requires connecting to the network, such as encyclopedia knowledge questions and answers, directly accessing high-quality large models. And some input sentences are user-exclusive input sentences and their responses, which are all types that cannot be provided by network data, so it is necessary to learn them in a learning way, especially in a voice interactive learning way, which is more convenient. The learning process of the present invention is divided into two modes, one is a learning mode, in which the learning of user input sentences and their response content is completed; the other is a question-and-answer mode, in which the results after learning are tested.
[0095] The following first introduces the processing process of user input sentences and their responses (i.e., in the question-answering mode), and then introduces the learning process of user input sentences and their responses (i.e., in the learning mode).
[0096] a. Processing of user input sentences and responses
[0097] First, the voice signal is converted into an input question Q of text information, and then the questions are matched one by one in the local QA library. When the match is successful, the response body content corresponding to the question is output. The response result here is defined as a "response body", which includes a combination of multimodal information, but they are not necessarily filled with full information, but are completed as needed. Each input question Q corresponds to a response body, and a number of input questions Q and their response bodies constitute the question answer QA library. The internal structure of the user question answer QA library is shown in Fig.15 The response body includes the units of sequence number, QA library question Q, text answer, action, task, picture and video.
[0098] It should be noted that when users ask about encyclopedia knowledge, "Please XXX" or other distinguishing words are added before the input sentence to distinguish the results of local access or direct call of the big model. This can ensure that the questions and answers of various situations can be distinguished in an orderly manner, and at the same time, it can ensure that the target to be accessed (local library / external library) can be located as quickly as possible.
[0099] The flowchart of receiving the user input sentence (question sentence / command sentence) through speech recognition and obtaining the response output is shown in Fig.16. First, the user input question Q is matched with all the question Q in the QA library through the keyword algorithm. When the match is successful, the response body content of the input question Q is directly obtained. Then, the content in the response body is output in sequence, and the content without content is skipped. For example, if there is text content, the text is output, if there is action content, the action is executed, if there is picture content, the picture is played, if there is video content, the video is played, and if there is task content, the task content is executed. If the entire QA library fails to match, it jumps to calling the large model to perform a generalized matching operation of the user input question Q. The background shaded area in the figure shows: generalized matching of the user input question Q by calling the large model. Because even if the content of the QA library question Q is comprehensive, there will be inconsistencies with the user input sentence, and it is not necessary to include all kinds of sentences in terms of operating efficiency and economy. Therefore, here the user input sentence and the question sentence in the QA library are matched by calling the large model, and then the sequence number and matching score of the matching question sentence are obtained, and then the corresponding processing is performed. When the score is lower than the set threshold, the user is asked to confirm through a rhetorical question, thereby enhancing the accuracy of the question execution and greatly reducing the impact of the large model hallucination problem on the accuracy of the answer. Before performing an important action in an uncertain situation, by asking the user for confirmation, even if this rhetorical question slows down the speed of the immediate response, it is worth it compared to a more accurate execution effect, which is much better than a quick response but a wrong execution result. The large model is called when the direct local keyword match fails. The large model is not introduced every time, because if the local keyword match is successful, there is no need to call the generalized match of the large model. This can not only improve the matching efficiency, but also reduce the unnecessary overhead of large model calls.
[0100] b. Learning process of user input sentences and their responses
[0101] exist Figure 2 In the program setting interface, click the [Learn] button in the leftmost button area to enter the voice interactive learning interface. Fig.17 . The figure includes the typing input area and the [Input] button, which are mainly used for manual typing input when voice interaction is inconvenient in some situations. The rest include the "Please Speak", "Answer", "Action", "Task", "Picture" and "Video" related areas corresponding to voice input, as well as the learning status information display area; the [Start / Stop Learning] button is used to set whether the current mode is learning mode (the button color is blue) or the question-and-answer mode after learning (the button color is green). When entering the learning interface for the first time, the default is the question-and-answer mode, that is, the button color is green. The [Exit] button is used to exit the current learning interface.
[0102] It should be noted that areas such as "Action", "Task", "Picture" and "Video" are all automatically loaded by the system as candidates for the existing file names corresponding to their classes, and are displayed in the area in the form of a sliding scroll bar, divided into three partitions: upper, middle and lower. The middle partition is the content selected by the end user. If not set, the content of the partition is empty (i.e., not selected).
[0103] The following example illustrates the learning process of "learning and using" through voice interaction. The relevant flow chart is shown in Fig.18 The "Xiaobao" in the example here is the nickname of the robot, and it can also be other nicknames.
[0104] In the voice learning interface, the user directly issues the voice command "Xiaobao starts learning", and the robot responds with "OK, Xiaobao starts learning". At the same time, the color of the [Start / Stop Learning] button changes from green to blue, indicating that it has entered the learning mode.
[0105] The user asks the question (or command) "Please tell me which floor the respiratory department of the hospital is on?". If the voice recognition is correct, "Please tell me which floor the respiratory department of the hospital is on?" will be displayed in the corresponding area of "Please tell me". There will also be a voice response of "Xiaobao has learned the question "Please tell me which floor the respiratory department of the hospital is on?", indicating that the question has been learned. Please note that the preceding "Please tell me" is a prefix of the guiding sentence and must be added. Otherwise, there will be a voice output of "Tip, please start with 'Please tell me' when learning" to guide the user to follow the standard learning process.
[0106] The user says the answer sentence "The respiratory department is on the third floor" by voice. If the voice recognition is correct, "The respiratory department is on the third floor" will be displayed directly in the "Answer" corresponding area, and there will be a voice response of "Xiaobao has learned to answer the respiratory department on the third floor", indicating that the answer sentence has been learned. Note that the "answer" in front is a prefix of the guide and must be added, otherwise there will be a voice prompt "Tip, please start with 'please say', 'answer', 'action', etc. when learning."
[0107] The user says the action "head nod" in voice. If the voice recognition is correct, "head nod" will be displayed directly in the "action" corresponding area, and there will be a voice response of "Xiaobao has learned the action head nod", indicating that the action has been learned. Note that the "action" in front is also a guide prefix, similar to the previous "please speak" and "answer". The other guide prefix operations are completely similar to this and will not be repeated.
[0108] It should be noted that the action content must be an action that already exists in the current system. Otherwise, when the system cannot find it, a voice prompt will be given: "Hint: XX action does not exist. Please reconfirm or add it before operating!" In addition, you can also select the corresponding action content by sliding the scroll wheel in this area. This method can help you quickly understand and select when you are not sure what actions are available.
[0109] The user says the content of the task "Task: Make breakfast" by voice. If the voice recognition is correct, "Make breakfast" will be displayed directly in the "Task" corresponding area, and there will be a voice response of "Xiaobao has learned the task of making breakfast", indicating that the task has been learned.
[0110] It should be noted that the task content must be a task that already exists in the current system (the user has created it and it exists in the task list column). Otherwise, when the system cannot find it, it will give a voice prompt "Tip, XX task does not exist, please reconfirm or add it before proceeding!" In addition, you can also select the corresponding task content by sliding the scroll wheel of the task target column, so that you can quickly understand and select when you are not sure what tasks are available.
[0111] The user speaks the content of the picture "Picture ABC". If the voice recognition is correct, "ABC" will be directly displayed in the corresponding area of "Picture", and there will be a voice response of "Xiaobao has learned picture ABC", indicating that the picture has been learned.
[0112] It should be noted that the picture content must be a picture in the current system. Otherwise, when the system cannot retrieve it, a voice prompt will be given: "Hint: XX picture does not exist. Please reconfirm or add it before operation!" In addition, after double-clicking the bar, the picture selection pops up directly. There are two ways: one is to take a photo on the spot, and the other is to select from existing picture files. These are common and simple operations, and the relevant descriptions are omitted here.
[0113] The user says the content of the video "Video 123" in voice. If the voice recognition is correct, "123" will be directly displayed in the corresponding area of "Video", and there will be a voice response of "Xiaobao has learned video 123", indicating that the video has been learned.
[0114] It should be noted that the video content must be a video that already exists in the current system. Otherwise, when the system cannot retrieve it, a voice prompt will be given: "Hint: XX video does not exist. Please reconfirm or add it!" In addition, after double-clicking the bar, a video selection pops up directly. There are two ways: one is to shoot on the spot, and the other is to select from existing video files. These are common and simple operations, and the relevant descriptions are omitted here.
[0115] Not all of the items listed in the response body corresponding to an input question Q need to be learned. Except for question Q, which must exist, more items can be selected based on at least one selection, depending on demand.
[0116] When the content of the above learning items is entered, the user inputs the voice "Xiaobao has finished learning". When the voice recognition is correct, it will automatically respond with "Xiaobao has learned OK, you can learn the next content". You can also manually press the [Learning Completed] button to achieve the effect of learning completion. After the learning is completed, you can start a new learning by entering "Please say XXX" again.
[0117] When the sentence learning is completed, the learning result status information is then displayed on the screen to guide the current learning status verification. The learning result status includes user information, learning time and learning content, etc., so that the information can be traced and used in the future.
[0118] When the input question Q and its response content are learned, the user inputs the voice "Xiaobao quits learning". When it is recognized correctly, or the [Start / Stop Learning] button is pressed manually, it will automatically respond with "Xiaobao has quit learning". At the same time, the color of the [Start / Stop Learning] button changes from blue to green, indicating that the current learning mode has been exited, that is, entering the question-and-answer mode after learning is completed.
[0119] In the question-and-answer mode, the learning effect can be directly tested. For example, when the user inputs "Which floor is the respiratory department of the hospital on?", the voice output should be "The respiratory department of the hospital is on the third floor" and displayed in the response area. When there are matching actions or other items, there will be corresponding manifestations at the same time. When the output response content is incorrect, the output matching status information can be used to check or modify the error.
[0120] When all learning content has been completed, click the [Exit] button to exit the current learning interface and return to the previous interface of the learning interface. The entire learning process is completed.
[0121] The input sentence and response body learning is realized through voice interaction, which solves the inconvenience of direct typing input and is easier for ordinary users to accept.
[0122] In summary, through the above introduction to the user input sentence and its response processing process and the voice interactive learning method, it is explained that the present invention solves the problem that the existing robot has limited response content and lacks a user-friendly on-site learning method in responding to the corresponding user input language content. In addition to conventional multimodal information such as text, voice, pictures, and videos, its response content also includes action execution and even task content execution. The user only needs to move his mouth (input a sentence) to correspond to the robot's various tasks, so that the robot has the ability to solve complex problems. The on-site voice interactive learning method is easy to operate and does not require contact typing, making it easier for the robot to learn input sentences and their response content anytime and anywhere, so that the robot can solve more user problems as the learning content continues to increase.
[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A robot programming system for end users, It is characterized in that include: An interactive unit, the interactive unit is used to provide an interactive interface between the robot and the user, the interactive interface includes an interactive main interface and a program setting interface, the program setting interface is used to select a default task name for the interactive main interface, start the execution and debugging of the selected task in response to the user's interactive action, start adding a task, editing a task and deleting a task in response to the user's interactive action, and jump to the voice interactive learning interface; A hardware access unit, the hardware access unit is used to access the robot's embodied hardware according to the robot's own program and the program written by the user, the robot's embodied hardware includes a sensor module for the robot to sense external information and a drive module for controlling the robot's execution component actions; A batch task planning execution unit, the batch task planning execution unit is used to plan multiple subtasks to be executed at different specified times to be started by calling the batch task name only once; A learning unit is used to learn user input sentences and corresponding responses through voice interaction.
2. A robot programming system for end users according to claim 1, It is characterized in that Learning user input sentences and corresponding responses through voice interaction includes: In the program setting interface, in response to the user's interactive action to start the learning function, the voice interactive learning interface is entered, and the voice interactive learning interface includes a [Start / Stop Learning] button and an [Exit] button. The [Start / Stop Learning] button is used to set whether the current mode is the learning mode or the question-and-answer mode after learning, and the [Exit] button is used to exit the current learning interface; After the user triggers [Start / Stop Learning] to enter the learning mode, the learning content is output to the robot through voice, and information interaction is achieved in combination with the robot's feedback voice. The learning content includes question and answer content, action content, task content, picture content and video content.
3. A robot programming system for end users according to claim 2, It is characterized in that Learning user input sentences and corresponding responses through voice interaction also includes: An input question Q that converts a speech signal into text information; Questions are matched one by one in the question answer QA library. When the match is successful, the response body content corresponding to the question is output. The response body is a combination of multimodal information used to define the response result. Each input question Q corresponds to a response body. The question answer QA library is used to store the input question Q and the corresponding response body.
4. A robot programming system for end users according to claim 3, It is characterized in that Learning user input sentences and corresponding responses through voice interaction also includes: If the entire QA library fails to match, the process jumps to calling the large model to perform a generalized matching operation on the user input question Q. When the generalized matching score is lower than the set threshold, the user is asked to confirm by asking a question in reverse.
5. A robot programming system for end users according to claim 1, It is characterized in that The program setting interface is used to start adding tasks, editing tasks, and deleting tasks in response to user interactions, including: In response to the user triggering the [Add] button on the program setting interface, a new task dialog box pops up, and the user completes the new task by interacting with the information in the new task dialog box; In response to the user triggering the [Edit] button on the program setting interface, an edit task dialog box pops up, and the user completes the edit task by interacting with the information in the edit task dialog box; In response to the user triggering the [Delete] button on the program setting interface, a delete task dialog box pops up, and the user completes the deletion task by interacting with the information in the delete task dialog box.
6. A robot programming system for end users according to claim 1, It is characterized in that When the user performs information interaction actions through the editing task dialog box, programming is performed in the native language.
7. A robot programming system for end users according to claim 1, It is characterized in that The hardware access unit includes a first interface adapter, which converts different functions of different robot manufacturers for the same action into a unified function expressed in the user's native language by searching a first dedicated conversion interface library.
8. A robot programming system for end users according to claim 1, It is characterized in that The system further comprises an external resource access unit, and the external resource access unit is used for accessing third-party resources according to an external interface.
9. A robot programming system for end users according to claim 6, It is characterized in that The external resource access unit includes a second interface adapter, which uniformly converts external resources from a third party into a unified function expressed in the user's native language by searching a second dedicated conversion interface library.
10. A robot programming system for end users according to claim 1, It is characterized in that The system also includes a wireless communication unit, which is used to perform program writing and debugging with a PC or a PAD in a wireless manner.
Citation Information
Patent Citations
Method and system for establishing and performing task program based on functional unit
CN102214103A
Interface edition and operation system and method for ordinary users
CN103425485A
Graphic and interactive machine vision system and operation method thereof
CN106293748A
Method for robot intelligent programming
CN107678743A
Graphical development platform based on ROS (Robot Operating System)
CN107678746A
