System with voice recognition and verbal command control

Abstract

The invention relates to the field of digital information technology. A hardware and software system for controlling a robotic system (RS) includes an actuator and an RS control device. The RS control device includes a voice recognition unit which is capable of recognizing voice commands and is connected to a command vocabulary storage unit, a robot command identification unit for identifying commands on the basis of recognized vocabulary and information stored in the storage unit, a communication unit that transmits the command to the RS control device, and a feedback device. The claimed solution lends universality to the verbal command control of robotic systems.

Description

[0001] A hardware and software system with multilingual speech recognition and verbal command control of robots

[0002] The utility model relates to the field of digital information technology, namely to the field of verbal control of a robot through speech recognition in English, Kazakh and Russian languages, and can be applied to the control of robots on production lines, automation of assembly, welding and packaging processes and, in general, improvement of interaction between people and machines to increase production efficiency.

[0003] Modern robotics has made great strides. Robots have become an integral part of our daily lives. Today, they can serve not only as physical but also as intellectual assistants to humans. Their presence and use in more and more places and professions has significantly improved many production and technological processes, making them more efficient and safer. A new trend for modern robots is their use in education, medicine, entertainment, advertising, retail, space exploration, the military, and emergency response.

[0004] In everyday life, we increasingly encounter anthropomorphic (humanoid and android) robots capable of interactive verbal (vocal) and non-verbal (gesture) communication with people. Many of them possess elements of artificial intelligence, enabling them to understand, respond to questions, speak, and even engage in small talk. However, they are generally oriented primarily toward English and Chinese, and to a lesser extent, toward Japanese, Korean, Russian, and other languages. To date, Kazakhstani researchers have developed methods and software for intelligent robotic systems capable of recognizing, understanding, and reproducing speech in the Kazakh language.Computer models have been developed and experimental studies are being conducted on the practical application of verbal (voice) command control of various types of robots: anthropomorphic, manipulative, mobile (Orken Mamyrbayev, Mussa Turdalyuly, Nurbapa Mekebayev, Keylan Alimhan, Aizat Kydyrbekova, Tolganay Turdalykyzy. Automatic Recognition of Kazakh Speech Using Deep Neural Networks / / Asian Conference on Intelligent Information and Database Systems, 2019).

[0005] A mobile robot (US Patent No. 6,841,963) contains multiple contactless sensors arranged in a row along the bottom of its body. These sensors are used to orient the robot in space. The robot is positioned using metal guide strips placed in the floor of the work area. The position of the robot and its working elements is calculated by the control unit (CU) using an image of the work area stored in the CU's memory.

[0006] A disadvantage of the known technical solution is the lack of a verbal control system, which may limit the scope of its application.

[0007] A known anthropomorphic robot (RU Patent No. 2361726) has a control system comprising a computer housed within the robot's body, with pre-trained neural networks installed, a vision unit consisting of cameras mounted on the robot's head, and a video signal processing unit. Tactile sensors are located on the inner surfaces of the robot's hands and connected to the microcontroller's inputs. A light beacon is located on the index finger of one of the robot's hands. The robot's operation involves pre-training three neural networks to perform tasks related to grasping and holding various objects, generating a final sample of images of the relative positions of the robot and the object, and selecting corresponding rotation angles for the body, manipulators, and hands, as well as finger flexion and palm curvature.The computer is then given a task, such as "grasping" an object. The first neural network sends a signal to the microcontroller with the values ​​of the rotation angles of the robot's body and manipulators to achieve contact with the object. The second neural network corrects these angles, and the third neural network sends the values ​​of the rotation angle of the hand, the degree of flexion of the fingers, and the curvature of the palm to the microcontroller.

[0008] The disadvantage of this solution is its limited functionality due to the inability to recognize commands in languages ​​other than Russian; moreover, there is no verbal control system for the robot.

[0009] The closest analogue to the present utility model is a hardware and software system for controlling a robot (Japanese Patent JP2006289514A), which proposes a robotic mechanism comprising at least one actuator, a robot control device that controls the robot's operation, and a voice interface that transmits a command to the control device. The voice interface recognizes user voice input and converts it into a robot command. The robot control system includes a speech recognition unit, a command dictionary storage unit that stores information linking the recognized dictionary to the robot command, a command definition unit that determines the robot command based on the recognized dictionary and information stored in the command dictionary storage unit, and a first communication unit that transmits the command to the robot control device.The robot control unit receives a command transmitted via the voice interface, and the command control unit determines whether to execute or ignore the command, depending on the robot's operational state. The authorized command storage unit stores commands that can be executed, and the robot control unit controls the robot's operation accordingly.

[0010] A disadvantage of the known technical solution is the limited functionality of the voice interface, which does not allow for the execution of all possible commands; in addition, the voice recognition system is implemented only in English.

[0011] Human speech recognition is a complex technical task, requiring sound (acoustic) signals to be analyzed and structured into a hierarchy of words, phrases, and sentences. The main challenges in speech recognition include acoustic and temporal variability, when the same audio files have different characteristics in terms of amplitude, frequency, pitch, duration, and other variables at different points in time.

[0012] The utility model is based on the goal of universality and multilingualism for verbal command control during the operation of robotic systems (RS) using modern technologies. Universality and multilingualism are achieved through the use of an intelligent speech recognition system in Kazakh, English, and Russian.

[0013] The declared hardware and software complex (HSC) has a high level of capability for recognizing and executing commands not only in Kazakh, Russian, and English, but also in other languages ​​of the world, which makes the HSC multilingual and more universal when using verbal command control of robotic systems of various types and purposes.

[0014] The technical result of this utility model is the creation of a hardware and software system that is versatile, has expanded functionality, is highly efficient, and is convenient for programming and controlling various types of robotic systems: anthropomorphic, manipulative, and mobile robots, thanks to the ability to recognize voice commands in Kazakh, English, Russian, and other languages.

[0015] The technical result is achieved through the use of the following in the PAK:

[0016] 1. A voice recognition device that uses advanced algorithms and machine learning methods to recognize voice commands in multiple languages, including Kazakh. The device analyzes the incoming voice signal, identifies the language, and converts the voice command into text.

[0017] 2. Verbal command control devices: Provides an interface for user interaction with the robotic system via voice commands. The user (operator) can issue commands in any of the three supported languages, including Kazakh.

[0018] 3. Processor: processes and interprets recognized commands, determining the appropriate actions to carry them out.

[0019] 4. Command Execution Device: includes mechanisms and devices that execute commands recognized and interpreted by the processor. These may include: actuators, manipulators, drives, sensors, detectors, and other PC components.

[0020] 5. Machine learning and adaptation capabilities enable an intelligent robotic system to independently improve its ability to recognize and execute commands based on accumulated experience and interactions with the user and the external environment. The system analyzes errors and successes in recognizing and executing commands and adjusts its algorithms.

[0021] 6. Device for connecting to external devices and networks: enables data exchange with other devices and systems, as well as access to cloud services for updating and expanding the system's functionality. The essence of the utility model is illustrated by the following figures.

[0022] Fig. 1 shows a block diagram of the operation of the hardware and software complex.

[0023] Fig. 2 - Simulator of a collaborative robot.

[0024] Fig. 3 - Simulator of a collaborative robot for commands: "left", "right".

[0025] Fig. 4 - Simulator of a collaborative robot for commands: "lower", "grab".

[0026] Fig. 5 - Simulator of a collaborative robot for commands: "lift", "put down".

[0027] Fig. 6 shows the results of verbal control of the robot in three languages.

[0028] A hardware and software control system for a robotic system, including at least one drive, a control device for the robotic system, which controls the operation of the robotic system and includes a speech recognition unit in Kazakh, Russian and English languages, connected to a command dictionary storage unit, which stores information linking the recognized dictionary with the robot command, a command determination unit, which determines the robotic system command based on the recognized dictionary and information stored in the command dictionary storage unit and a communication unit, which transmits the command to the control device for the robotic system, as well as a feedback device, the speech recognition unit is configured to recognize voice commands in Kazakh and Russian languages.

[0029] The PC system control module implements voice command recognition programs in Kazakh, Russian, and English. The keywords in the control module database are shown in Table 1.

[0030] Table 1. Basic robot control commands:

[0031] In this paper, the Word Error Rate (WER) metric is used to determine the effectiveness of speech recognition systems. It is based on counting insertions, substitutions and deletions of words (Orken Mamyrbayev, Mussa Turdalyuly, Nurbapa Mekebayev, Kuralay Mukhsina, Alimukhan Keylan, Bagher BabaAli, Gulnaz Nabieva, Aigerim Duisenbayeva, Bekturgan Akhmetov. Continuous Speech Recognition of Kazakh Language / / ITM Web of Conferences No. 24, 2019). Where

[0032] S - the number of replaced words

[0033] D - the number of deleted words

[0034] 7 - the number of inserted words

[0035] That is, the higher the WER, the less accurate the recognition. It is usually expressed as a percentage.

[0036] There are several well-known open-source speech recognition systems: CMU Sphinx, NTK, Kaldi, and Julius. The current version is sphinx4-5prealpha. Sphinx-4 is a modular framework. This modular structure allows for customization of the system's parameters based on the needs of a specific task. Three main modules are distinguished: FrontEnd, Decoder, and Linguist (CMU Sphinx Project by Carnegie Mellon University. hup:.ycrnusphinx.&3urceforge.n€i / ).

[0037] The essence of the utility model will be clear from the following detailed explanation, using a specific example of an experiment involving verbal control of an ABB industrial robot in Kazakh, Russian, and English. A number of approaches and technologies can be used to integrate a verbal command control system with robots. The main objective is to enable the speech recognition and command interpretation system to interact with the control unit of a given robot.

[0038] The integration of verbal command control with robots consists of the following steps (Fig. 1):

[0039] 1. Voice recognition and command interpretation tool:

[0040] ■ Includes CMU Sphinx for speech recognition in multiple languages, including Kazakh.

[0041] ■ The command interpretation module converts recognized text commands into specific actions.

[0042] 2. Interface for communication with the robot:

[0043] ■ The developed interface (software or middleware) allows the speech recognition module to transmit commands to the ABB robot controller.

[0044] 3. Communication protocol:

[0045] ■ Uses standard communication protocols such as TCP / IP, OPC UA, or RESTful API to exchange data between the module and the robot controller.

[0046] 4. Programming the robot's actions: ■ Based on the interpreted commands, corresponding commands will be generated that the robot controller can execute.

[0047] For the experiments, audio data consisting of 1,480 recordings was recorded: 20 columns with 74-75 recordings. Each recording consists of phrases in three languages, averaging 6 seconds in length. To identify the speaker, the following data was collected: name, gender, place of birth, and year of birth.

[0048] All audio materials have the same characteristics:

[0049] - file extension: wav;

[0050] - digital conversion method: PCM;

[0051] - discrete frequency: 44.1 kHz;

[0052] - bit depth: 16 bits;

[0053] - number of audio channels: one (mono).

[0054] The audio and recording of one speaker took on average 40-50 minutes, including the time needed to prepare the speaker, equipment and doubling, which equates to 74-75 resulting files, totaling 7-8 minutes for each speaker.

[0055] To conduct scientific research and test the performance of the developed algorithms and software, the Robo DK 3D modeling software simulator was used.

[0056] CMU Sphinx is implemented in C / C++. The intelligent verbal system was developed in Python with the CMU Sphinx communication module and a graphical PyOpenGL command execution simulator. ABB robots were chosen as the basis for the 3D modeling. The graphical results of the study are shown in Figs. 2-5.

[0057] The methodology is based on the average integration technique of acoustic phonemes for three languages ​​(Kazakh, Russian, and English). The methodology is fundamentally the same for English, Russian, and Kazakh. The goal of the study was to test the methodology and the quality of speech recognition in the three languages. The obtained WER results showed a range of ~61-65%, which is quite good for early studies on language identification and integration (Figure 6).

Claims

FORMULA 1. A hardware and software system with multilingual voice recognition and verbal command control of robots, comprising at least one drive, a robotic system control device that controls the operation of the robotic system and includes an English language speech recognition unit connected to a command dictionary storage unit that stores information linking the recognized dictionary with the robot command, a command determination unit that determines the robotic system command based on the recognized dictionary and information stored in the command dictionary storage unit, and a communication unit that transmits the command to the robotic system control device, as well as a feedback device, characterized in that the speech recognition unit is designed with the ability to recognize voice commands in any language of the world.

Images