Robot, voice control method and apparatus therefor, and storage medium

Abstract

A robot, a voice control method and apparatus therefor, and a storage medium. The voice control method for a robot comprises: obtaining a voice input of a user (102); processing the voice input of the user to obtain a task suitable for being executed by a robot (104); and controlling the robot to execute the task (106). The voice input of the user is processed, so that the task suitable for being executed by the robot can be obtained, and then the robot is controlled to operate on the basis of the task suitable for being executed by the robot, thereby improving execution efficiency and flexibility of the robot, improving accuracy of task execution of the robot in the subsequent application process, and helping improve user experience.

Description

Robot, voice control method, device and storage medium thereof

[0001] This application claims priority to Chinese patent application No. 202311252440.1 filed on September 26, 2023 and Chinese patent application No. 202410286909.1 filed on March 13, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of control technology, and in particular to a robot and a voice control method, device, and storage medium thereof. Background Art

[0003] In related technical solutions, more and more devices support voice control. The emergence of voice control technology provides convenience for the interaction between users and devices.

[0004] However, in related technical solutions, the voice interaction solutions supported by the device are mostly: using a voice recognition model to perform voice recognition on the voice input by the user, and then controlling according to the recognition result.

[0005] The above voice interaction solution has the following disadvantages:

[0006] When the user's voice input is relatively complex, the recognition results output by the speech recognition model are unsatisfactory. Its semantic understanding ability is limited, and there are still challenges in handling complex contexts and fuzzy semantics. It is difficult to accurately understand the user's needs, which affects the accuracy and efficiency of task execution.

[0007] Summary of the Invention

[0008] This application aims to at least solve the technical problems existing in the existing technology or related technology, that is, when the user's voice input is relatively complex, the recognition results output by the speech recognition model are unsatisfactory, its semantic understanding ability is limited, there are still challenges in processing complex contexts and fuzzy semantics, and it is difficult to accurately understand the user's needs, thereby affecting the accuracy and efficiency of task execution.

[0009] To this end, a first aspect of the present application is to provide a robot voice control method.

[0010] A second aspect of the present application is to provide a robot voice control device.

[0011] A third aspect of the present application is to provide another robot voice control device.

[0012] A fourth aspect of the present application is to provide a readable storage medium.

[0013] A fifth aspect of the present application is to provide a device.

[0014] A sixth aspect of the present application is to provide a robot.

[0015] A seventh aspect of the present application is to provide a computer program product.

[0016] In view of this, according to the first aspect of the present application, the present application provides a robot voice control method, which includes: obtaining the user's voice input; processing the user's voice input to obtain a task suitable for the robot to perform; and controlling the robot to perform the task.

[0017] In some technical solutions, the user's voice input is processed to obtain a task suitable for the robot to perform, specifically including: performing voice recognition on the voice input to obtain text information; using a preset language model to parse the text information to obtain task information; and determining the task corresponding to the task information.

[0018] In some technical solutions, in the process of generating task information, the robot voice control method also includes: generating natural language feedback information based on a preset language model; before controlling the robot to perform the task, the robot voice control method also includes: synthesizing voice information based on the feedback information; and playing the voice information.

[0019] In some technical solutions, voice recognition is performed on the voice input to obtain text information, specifically including: using a deep learning-based voice recognition model to perform voice recognition on the voice input to obtain text information.

[0020] In some technical solutions, voice information is synthesized based on feedback information, specifically including: performing voice synthesis on the feedback information based on an end-to-end voice synthesis model based on deep learning to obtain voice information.

[0021] In some technical solutions, task information includes at least one task time and a corresponding task execution location; controlling the robot to perform the task specifically includes: determining a task execution path based on at least one task time and a corresponding task execution location; and executing the task corresponding to the task information based on the task execution path.

[0022] In some technical solutions, the robot is a cleaning robot or a service robot, and the robot voice control method also includes: obtaining the current position information of the robot; determining a navigation path based on the current position information and the task execution path; wherein the robot walks based on the navigation path.

[0023] In some technical solutions, the robot voice control method further includes: obtaining environmental information of the robot's environment; and updating the navigation path based on the environmental information.

[0024] In some technical solutions, the robot voice control method also includes: when the task information includes keywords used to indicate repeated execution, determining the start time, end time and target task associated with the keywords; and determining the periodic task based on the start time, end time and target task.

[0025] In some technical solutions, a preset language model is used to parse text information to obtain task information, including: sending the text information to a preset language model deployed on a server; and receiving the task information output from the preset language model.

[0026] In some technical solutions, the user's voice input is processed to obtain a task suitable for the robot to perform, specifically including: recognizing and converting the acquired user's voice input to obtain a target control instruction; controlling the robot to perform the task, specifically including: controlling the robot based on the target control instruction and obtaining feedback information; optimizing the target algorithm parameters based on the feedback information; the target algorithm parameters include at least one of the model parameters for recognizing the voice input and the control strategy for controlling the robot based on the target control instruction.

[0027] According to the robot voice control method of the present application, the acquired user voice input is recognized and converted, that is, the user task instructions are recognized and converted, and the target control instructions that can be recognized by the robot are obtained. The target algorithm parameters are optimized based on the feedback information corresponding to the execution results. It can be applied to complex and flexible application scenarios, improve the execution efficiency and flexibility of the robot, and improve the accuracy of the robot's task execution in subsequent applications, which helps to improve the user experience.

[0028] In some technical solutions, the user's voice input is recognized and converted to obtain target control instructions, including: inputting the user's voice input into a large language model to obtain intermediate instructions output by the large language model; and converting the intermediate instructions to obtain target control instructions.

[0029] In some technical solutions, the intermediate instructions are converted to obtain target control instructions, including: standardizing the basic information included in the intermediate instructions to generate standardized data; based on the task information included in the intermediate instructions, matching the target function corresponding to the task information from multiple candidate functions; and generating the target control instruction based on the standardized data and the target function.

[0030] In some technical solutions, when a target control instruction includes multiple sub-control instructions, controlling the robot based on the target control instruction includes: determining a target execution order of the multiple sub-control instructions based on the priority of each sub-control instruction; storing the multiple sub-control instructions in a task stack based on the target execution order; and deleting the target sub-control instruction from the task stack after executing the target sub-control instruction among the multiple sub-control instructions based on the task stack.

[0031] In some technical solutions, controlling a robot based on target control instructions includes: generating a target control strategy based on image information collected by the robot and the target control instructions; and controlling the robot based on the target control strategy.

[0032] In some technical solutions, optimizing target algorithm parameters based on feedback information includes optimizing model parameters based on at least part of the feedback information within the first time period and user task instructions corresponding to the feedback information.

[0033] In some technical solutions, optimizing target algorithm parameters based on feedback information includes optimizing a control strategy based on at least part of the feedback information within the second time period and a target control instruction corresponding to the feedback information.

[0034] In some technical solutions, the robot is controlled based on target control instructions, including: predicting risk information based on sensor data collected by the robot and the robot's posture information; and executing target operations based on the risk information.

[0035] According to the second aspect of the present application, the present application provides a robot voice control device, which includes: an acquisition module for acquiring the user's voice input; a processing module for processing the user's voice input to obtain a task suitable for the robot to perform; and a control module for controlling the robot to perform the task.

[0036] In some technical solutions, the processing module is a speech recognition module, which is used to perform speech recognition on speech input, obtain text information, and determine a task corresponding to the task information.

[0037] In some technical solutions, the control module is an execution module, which is used to control the robot to perform tasks.

[0038] In some technical solutions, the acquisition module can be understood as a human-computer interaction module, which is used to obtain the user's voice input.

[0039] In some technical solutions, in the process of generating task information, the generation module is used to: generate natural language feedback information based on a preset language model; before controlling the robot to perform the task, the robot voice control device also includes: a speech synthesis module, used to synthesize voice information based on the feedback information; and play the voice information.

[0040] In some technical solutions, the speech recognition module is specifically used to: use a deep learning-based speech recognition model to perform speech recognition on speech input to obtain text information.

[0041] In some technical solutions, the speech synthesis module is specifically used to: perform speech synthesis on the feedback information based on an end-to-end speech synthesis model based on deep learning to obtain speech information.

[0042] In some technical solutions, task information includes at least one task time and a corresponding task execution location; the execution module is specifically used to: determine the task execution path based on at least one task time and the corresponding task execution location; and execute the task corresponding to the task information based on the task execution path.

[0043] In some technical solutions, the robot is a cleaning robot or a service robot, and the robot voice control device also includes: a position information module, used to obtain the current position information of the robot; a path planning module, used to determine the navigation path based on the current position information and the task execution path; wherein the robot walks based on the navigation path.

[0044] In some technical solutions, it also includes: a sensor module for obtaining environmental information of the robot's environment; and a path planning module for updating the navigation path based on the environmental information.

[0045] In some technical solutions, it also includes: a time management module, which is used to determine the start time, end time and target task associated with the keyword when the task information includes a keyword used to indicate repeated execution; and determine the periodic task based on the start time, end time and target task.

[0046] In some technical solutions, optionally, it further includes: a cloud computing module, used to send text information to a preset language model deployed on a server; and receive task information output from the preset language model.

[0047] In some technical solutions, the processing module is a first processing module, which is used to recognize and convert the user's voice input to obtain target control instructions; the control module includes a second processing module and a third processing module, the second processing module is used to control the robot based on the target control instructions and obtain feedback information; the third processing module is used to optimize the target algorithm parameters based on the feedback information; the target algorithm parameters include at least one of the model parameters for recognizing voice input and the control strategy for controlling the robot based on the target control instructions.

[0048] According to the robot voice control device of the present application, the target control instructions that can be recognized by the robot are obtained by recognizing and converting the acquired user voice input, that is, recognizing and converting the task instructions, and optimizing the target algorithm parameters based on the feedback information corresponding to the execution results. It can be applied to complex and flexible application scenarios, improve the execution efficiency and flexibility of the robot, and improve the accuracy of the robot's task execution in subsequent applications, which helps to improve the user experience.

[0049] According to the third aspect of the present application, the present application provides a robot voice control device, including a processor and a memory, the memory storing programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the robot voice control method as described in any one of the above are implemented.

[0050] According to the fourth aspect of the present application, the present application provides a readable storage medium, which stores a program or instruction. When the program or instruction is executed by a processor, the steps of the robot voice control method as described above are implemented.

[0051] According to a fifth aspect of the present application, the present application provides a device comprising: a robot voice control device as described in any one of the above items; and / or a readable storage medium as described above.

[0052] According to the sixth aspect of the present application, the present application provides a robot, comprising: a robotic arm; and a robot voice control device such as any of the above, the robot voice control device being electrically connected to the robotic arm.

[0053] According to a seventh aspect of the present application, the present application provides a computer program product, comprising a computer program, which implements any of the above-mentioned robot voice control methods when executed by a processor.

[0054] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become obvious from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0056] FIG1 shows a flow chart of a robot voice control method according to an embodiment of the present invention;

[0057] FIG2 shows a second flow chart of the robot voice control method in an embodiment of the present application;

[0058] FIG3 shows a schematic block diagram of a robot voice control device according to an embodiment of the present application;

[0059] FIG4 shows a schematic block diagram of the interaction between the mobile phone software, the server and the device in an embodiment of the present application;

[0060] FIG5 shows a schematic block diagram of another robot voice control device according to an embodiment of the present application;

[0061] FIG6 is a third flow chart of the robot voice control method provided in an embodiment of the present application;

[0062] FIG7 is a fourth flow chart of the robot voice control method provided in an embodiment of the present application;

[0063] FIG8 is a schematic structural diagram of a robot voice control device provided in an embodiment of the present application;

[0064] FIG9 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to more clearly understand the above aspects, features and advantages of the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features therein can be combined with each other in the absence of conflict.

[0066] In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present application is not limited to the specific embodiments disclosed below.

[0067] In one embodiment of the present application, as shown in FIG1 , a robot voice control method is provided, comprising:

[0068] Step 102: Obtain user's voice input;

[0069] Step 104: Process the user's voice input to obtain a task suitable for the robot to perform;

[0070] Step 106: Control the robot to perform the task.

[0071] In this embodiment, by processing the user's voice input, tasks suitable for the robot to perform can be obtained, and then the robot can be controlled to run based on tasks suitable for the robot to perform, thereby improving the robot's execution efficiency and flexibility, and improving the accuracy of the robot's task execution in subsequent applications, which helps to improve the user experience.

[0072] In one embodiment of the present application, the user's voice input is processed to obtain a task suitable for the robot to perform, specifically including: performing voice recognition on the voice input to obtain text information; parsing the text information using a preset language model to obtain task information; and determining the task corresponding to the task information.

[0073] Specifically, as shown in FIG2 , a robot voice control method is provided, comprising:

[0074] Step 202: Obtain user's voice input;

[0075] Step 204: Perform voice recognition on the voice input to obtain text information;

[0076] Step 206: parse the text information using a preset language model to obtain task information;

[0077] Step 208: Execute the task corresponding to the task information.

[0078] The embodiments of the present application propose a robot voice control method. By running the above control method, the recognition ability of voice input can be improved, and thus it has better processing performance in processing complex contexts and fuzzy semantics, can accurately understand user needs, and ensure the accuracy and efficiency of task execution.

[0079] The embodiments of this application are based on the following principles: specifically, upon receiving voice input, voice recognition is performed on the voice input to obtain text information corresponding to the voice input. This text information is then parsed using a preset language model to obtain task information. During this process, text information is input to the preset language model. Compared to solutions that directly use voice input, there is no need to train the preset language model for voice recognition, which reduces the cost of model training.

[0080] Among them, the preset language model is a large language model, that is, a large language model (LLM) is an artificial intelligence model designed to understand and generate human language. They are trained on a large amount of text data and can perform a wide range of tasks, including text summarization, translation, and sentiment analysis. In the embodiments of the present application, a large language model is used to accurately understand user needs and ensure the accuracy of the output task information, thereby improving the success rate of task execution.

[0081] In some embodiments, optionally, the large language model is a large pre-trained language model that has been fine-tuned and optimized, which can be a Chat Generative Pre-trained Transformer (Chat GPT), which can be understood as a chat robot program. T is a natural language processing tool driven by artificial intelligence technology. It can conduct conversations by understanding and learning human language, and can also interact according to the context of the chat, truly chatting and communicating like a human, and even completing tasks such as writing emails, video scripts, copywriting, translation, code, and writing papers. In some embodiments, optionally, a large pre-trained language model is used to improve the model's ability to understand complex contexts and fuzzy semantics through fine-tuning and optimization, overcome the problem of insufficient semantic understanding ability, thereby more accurately understanding user needs and improving the success rate of task execution. In some embodiments, optionally, the task information is a structured task instruction.

[0082] In some embodiments, optionally, in the process of generating task information, the robot voice control method further includes: generating natural language feedback information based on a preset language model; before controlling the robot to perform the task, the robot voice control method further includes: synthesizing voice information based on the feedback information; and playing the voice information.

[0083] In this embodiment, a preset language model is used to generate feedback information in natural language. This process leverages the advantages of the preset language model to ensure that the output feedback information closely matches the user's voice input. Voice information synthesized based on the feedback information is played to enable voice interaction between the robot and the user. This process allows for smooth interaction between the robot and the user, thereby enhancing the interactive experience with the robot.

[0084] In some embodiments, optionally, speech synthesis is to convert text information into a human-understandable speech signal. In this application, speech synthesis is used to convert natural language feedback information into a speech signal (i.e., speech information) for the convenience of user listening.

[0085] In some embodiments, the robot voice control method further includes: outputting feedback information. In this embodiment, the feedback information can be output in the form of text, so that the robot can display the feedback information to the user in different forms, thereby meeting the interaction needs of different scenarios.

[0086] In some embodiments, optionally, performing speech recognition on the speech input to obtain text information specifically includes: using a deep learning-based speech recognition model to perform speech recognition on the speech input to obtain text information.

[0087] In this embodiment, the speech recognition model based on deep learning has better speech synthesis capabilities, and the above-mentioned speech training model is adopted so that the text information obtained by recognition can accurately express the user's ideas. Compared with the recognition scheme that uses word-by-word text recognition, the embodiment of the present application can more accurately express the meaning expressed by the voice input, and thus fully understand the meaning of the voice input. In some embodiments, optionally, speech recognition is the process of converting human voice signals into text information that can be processed by a computer. In this application, speech recognition technology is used to convert the user's voice input into text, which facilitates subsequent interaction with a preset language model.

[0088] In some embodiments, optionally, the speech recognition model can be a long short-term memory network (Long Short-Term Memory, LSTM) or a natural language processing model. Specifically, an advanced speech recognition model based on deep learning (such as a model based on an LSTM network or a Transformer) is adopted to improve the accuracy and robustness of speech recognition, so that the system can effectively recognize user voice input in various environments. Among them, natural language processing (NLP), natural language processing is an intersection of computer science, artificial intelligence and linguistics, and is committed to enabling computers to understand, parse and generate human language. In this application, natural language processing technology is used to parse the user's voice input, extract key information, and interact with a preset language model to generate structured task instructions (i.e., task information) and natural language feedback (i.e., feedback information).

[0089] In some embodiments, voice information may be synthesized based on the feedback information, specifically including: synthesizing the feedback information using an end-to-end speech synthesis model based on deep learning to generate the voice information. In this embodiment, synthesizing the voice information enables user-robot interaction via voice, overcoming the reliance on manual input or a fixed command set for robot control commands in related embodiments, thereby improving the user-robot interaction experience.

[0090] In some embodiments, optionally, the end-to-end speech synthesis model can be a speech synthesis model based on Tacotron. Tacotron is a neural network speech synthesis framework. In this embodiment, the above-mentioned end-to-end speech synthesis model is adopted to make the sentences of the generated voice information more natural and fluent, making the interaction between the user and the robot more natural and convenient, improving user experience and acceptance. In the above embodiment, through voice input and output and natural language processing technology, efficient communication between the user and the robot is achieved, improving interaction efficiency and user experience.

[0091] In some embodiments, optionally, the task information includes at least one task time and a corresponding task execution location; controlling the robot to perform the task specifically includes: determining a task execution path based on at least one task time and a corresponding task execution location; and executing the task corresponding to the task information based on the task execution path.

[0092] In this embodiment, when the voice input contains multiple tasks, the preset language model can parse out the task time corresponding to each task and the task execution location corresponding to the task time during the parsing of text information, and then determine the task execution path based on the parsed task time and task execution location, so that the robot can use the task execution path as a guide for task execution during the operation process, thereby more accurately understanding user needs and improving the success rate of task execution.

[0093] In the above embodiments, the task execution path can be understood as the order in which multiple tasks are executed in sequence. In some embodiments, the task execution path uses the task time corresponding to each task as the sorting condition, where the task time includes the start time and end time of the task execution, so as to improve the reliability of task execution.

[0094] In some embodiments, the robot is optionally a cleaning robot or a service robot, and the robot voice control method further includes: obtaining the robot's current position information; determining a navigation path based on the current position information and the task execution path; wherein the robot moves based on the navigation path. In this embodiment, during the operation of the robot, the current position information is obtained so that the robot's movement is guided based on the current position information and the task execution path. During this process, the robot can achieve autonomous movement and automatically perform tasks. In this process, the user does not need to participate in the movement of the robot, overcoming the disadvantage of requiring manual user intervention for the robot to operate and improving operation efficiency.

[0095] In some embodiments, the service robot may optionally be a hotel room robot or a mobile robot used to serve food in a restaurant. The service robot may be configured with different functions depending on its usage scenario, which will not be described in detail here. In some embodiments, the current position information may optionally be obtained by a position sensor provided on the robot. The position sensor may be selected based on the actual usage scenario, and its specific selection will not be described in detail here.

[0096] In some embodiments, the method may further include: obtaining environmental information of the robot's environment; and updating the navigation path based on the environmental information. In this embodiment, by obtaining environmental information, the robot can update the navigation path according to the environmental information during its walking process, thereby reducing collisions with other objects in the environment, thereby achieving dynamic obstacle avoidance, maximizing the robot's own safety, and reducing the impact of the robot's operation on other objects in the environment. In some embodiments, the environmental information may optionally include placement information of other objects in the environment in addition to the robot. In some embodiments, the environmental information may optionally include posture information of other objects in addition to placement information of other objects. In this embodiment, by obtaining the above information, obstacle avoidance can be performed to the greatest extent possible. In some embodiments, the environmental information may optionally be acquired and sensed using one or more of a camera, a lidar, and an infrared sensor.

[0097] In some embodiments, optionally, it further includes: when the task information includes keywords for indicating repeated execution, determining the start time, end time and target task associated with the keywords; determining the periodic task based on the start time, end time and target task. In this embodiment, the keywords can be words that can express repeated execution, such as weekly, every interval, monthly, and daily. They can be selected according to the actual usage scenario and will not be described in detail here. In the above embodiment, the start time, end time and target task can be obtained by recognizing and parsing the voice input, and will not be described in detail here. In the above embodiment, by determining the periodic task so that the robot can perform the task periodically, there is no need for the user to repeatedly issue voice input instructions, which facilitates reducing the frequency of interaction between the user and the robot, thereby improving the user experience of the robot.

[0098] In some embodiments, a preset language model is optionally used to parse text information to obtain task information, including: sending the text information to a preset language model deployed on a server; and receiving task information output from the preset language model. In this embodiment, the text information is sent so that the preset language model deployed on the server can process the received text information to obtain the task information. In this process, the processing of the text information can be transferred from the local device side to the cloud, that is, the server side, and the server's large and sufficient computing power is used to perform data processing, thereby increasing the speed of data processing to meet different needs in different scenarios.

[0099] In some embodiments, a robot voice control device is proposed, including: an acquisition module for acquiring a user's voice input; a processing module for processing the user's voice input to obtain a task suitable for the robot to perform; and a control module for controlling the robot to perform the task.

[0100] Specifically, in some embodiments, as shown in Figure 3, the present application provides a robot voice control device 300, including: a human-computer interaction module 302, used to obtain the user's voice input; a voice recognition module 304, used to perform voice recognition on the voice input to obtain text information; a generation module 306, used to use a preset language model to parse the text information to obtain task information; and an execution module 308, used to execute the task corresponding to the task information.

[0101] The embodiment of the present application proposes a robot voice control device 300, which can improve the recognition ability of voice input, and thus has better processing performance in processing complex contexts and fuzzy semantics, and thus can accurately understand the needs of users, ensuring the accuracy and efficiency of task execution. The embodiment of the present application is implemented based on the following principles. Specifically, when voice input is obtained, voice recognition is performed on the voice input to obtain text information corresponding to the voice input, and then the text information is parsed using a preset language model to obtain task information. In this process, what is input into the preset language model is text information. Compared with the solution of directly using voice input, there is no need to train the preset language model for voice recognition, which reduces the cost of model training.

[0102] In some embodiments, optionally, during the process of generating task information, the generation module is further configured to: generate natural language feedback information based on a preset language model; and before controlling the robot to perform the task, the robot voice control device further comprises: a speech synthesis module configured to synthesize speech information based on the feedback information; and play the speech information. In this embodiment, the feedback information is generated in natural language using the preset language model. During this process, the advantages of the preset language model can be leveraged to ensure that the output feedback information closely matches the user's voice input. By playing the speech information synthesized based on the feedback information, voice interaction between the robot and the user is achieved. This process allows for smooth interaction between the robot and the user, thereby enhancing the robot's interactive experience.

[0103] In some embodiments, speech synthesis optionally converts text information into human-understandable speech signals. In the present application, speech synthesis is used to convert natural language feedback information into speech signals (i.e., speech information) for the user to hear. In some embodiments, the robot voice control method further includes: outputting feedback information. In this embodiment, the feedback information can be output in the form of text, so that the robot can display the feedback information to the user in different forms, thereby meeting the interaction needs of different scenarios.

[0104] In some embodiments, optionally, the speech recognition module is specifically used to: use a deep learning-based speech recognition model to perform speech recognition on the speech input to obtain text information.

[0105] In this embodiment, the deep learning-based speech recognition model has superior speech synthesis capabilities. By using the aforementioned speech training model, the text information obtained through recognition can accurately express the user's thoughts. Compared to recognition solutions that use word-by-word text recognition, the embodiments of this application can more accurately express the meaning of the speech input, thereby fully understanding the meaning of the speech input.

[0106] In some embodiments, speech recognition is optionally the process of converting human speech signals into text information that can be processed by a computer. In this application, speech recognition technology is used to convert the user's speech input into text to facilitate subsequent interaction with a preset language model.

[0107] In some embodiments, the speech synthesis module is optionally configured to synthesize the feedback information based on a deep learning-based end-to-end speech synthesis model to generate speech information. In this embodiment, synthesizing speech information enables user-robot interaction via voice, overcoming the reliance on manual input or a fixed command set for robot control commands in related embodiments, thereby improving the user-robot interaction experience.

[0108] In some embodiments, the end-to-end speech synthesis model can optionally be a Tacotron-based speech synthesis model. Tacotron is a neural network speech synthesis framework. In this embodiment, the use of this end-to-end speech synthesis model makes the generated speech information more natural and fluent, making the interaction between the user and the robot more natural and convenient, thereby improving user experience and acceptance.

[0109] Specifically, the device takes a robot as an example, for example:

[0110] 1. The user sends a navigation request to the system through voice, for example: "Take me to the nearest coffee shop."

[0111] 2. The speech recognition module receives speech input and converts it into text: "Take me to the nearest coffee shop."

[0112] 3. The generation module parses the text and generates structured task instructions, such as: {"action":"navigate","target":"nearest_coffee_shop"}, and natural language feedback, such as: "OK, we are planning a route to the nearest coffee shop for you."

[0113] 4. The speech synthesis module converts natural language feedback into speech signals.

[0114] 5. The human-computer interaction module transmits voice signals to the user through the speaker, informing the user that the route is being planned.

[0115] 6. The robot's task execution module receives the structured task instructions, parses the instructions, and plans a navigation path. The robot then begins executing the navigation task according to the planned path.

[0116] 7. During the execution of the task, the system can provide natural language feedback based on the real-time situation, for example: "Turn left 100 meters ahead", and convert it into a voice signal through the speech synthesis module, and finally convey it to the user through the human-computer interaction module.

[0117] This embodiment effectively addresses the shortcomings of existing technologies in speech recognition, semantic understanding, and human-computer interaction. At the same time, the system provides a more natural and convenient human-computer interaction experience, allowing users to easily complete navigation tasks in various scenarios. For example, in indoor environments, the embodiments of this application can also be applied to the implementation of complex time- and task-related functions, such as the following voice input: "Clean the bedroom at 3 p.m., mop the living room and kitchen before 5 p.m., and then go back to dry the rags." The steps are as follows:

[0118] 1. The voice recognition module converts the voice command into text: "Clean the bedroom at 3 pm, mop the living room and kitchen before 5 pm, and then go back to dry the rags."

[0119] 2. The generation module parses the text and generates structured task instructions, for example:

[0120] jsonCopy code[{"action":"clean","location":"bedroom","time":"15:00"},{"action":"mop","location":["living_room","kitchen"],"deadline":"17:00"},{"action":"dry","object":"cleaning_cloth","after_previous":true}], while generating natural language feedback: "OK, I will clean the bedroom at 3 pm, finish mopping the living room and kitchen before 5 pm, and dry the mop after finishing."

[0121] 3. The speech synthesis module converts natural language feedback into speech signals.

[0122] 4. The human-computer interaction module transmits voice signals to the user through the speaker to inform the user of the task plan.

[0123] 5. The execution module performs the corresponding tasks at the specified time and place according to the generated structured task instructions.

[0124] Specifically, first, perform the bedroom cleaning task at 3:00 PM; then, complete the mopping task in the living room and kitchen before 5:00 PM; and finally, dry the rags according to the task order. In the above embodiment, an advanced deep learning-based speech recognition model is used to effectively recognize user voice input in different accents, speaking speeds, and noisy environments. Targeted training enables highly accurate speech recognition, improving the accuracy and robustness of speech recognition, enabling the system to better understand user needs. By fine-tuning and optimizing large pre-trained language models (such as Chat GPT), the model's ability to understand complex contexts and fuzzy semantics is improved. Accurate semantic understanding and task instruction generation enable accurate structured task instructions and natural language feedback based on user input, providing users with precise task execution. Advanced speech synthesis technology is used to convert natural language feedback into speech signals, providing a natural and smooth human-computer interaction experience. Users can communicate with the system through simple voice commands, achieving simple and efficient operation and natural and smooth human-computer interaction. The execution module parses and plans the path based on the structured task instructions, adjusting the robot's motion, controlling sensors, and other equipment to meet user needs. This optimized task execution method can improve the robot's work efficiency, reduce energy consumption, achieve more efficient task completion, and realize flexible indoor and outdoor navigation task execution. Combining multiple modules such as speech recognition, semantic understanding, human-computer interaction and task execution, it provides users with a more natural and convenient navigation experience. Users can easily complete navigation tasks in various scenarios, improving their quality of life and work efficiency. In the above embodiment, the above embodiment improves the accuracy and naturalness of speech recognition, semantic understanding and human-computer interaction, and can also realize flexible and efficient task execution in various indoor and outdoor scenarios. At the same time, the optimized task execution module helps to improve the robot's work efficiency and reduce energy consumption, achieve more efficient task completion, and bring a better experience to users.

[0125] The embodiments proposed in this application utilize a modular design, ensuring excellent interoperability and compatibility between modules, making them easy to expand and upgrade. This allows the system to be easily expanded and upgraded in the future to adapt to changing user needs and technological advancements. Furthermore, the above embodiments are not only applicable to indoor navigation tasks in home environments but also support multi-scenario applications. For example, they can be widely applied in various fields, such as commercial, industrial, and medical, enabling voice-based navigation in various scenarios. This further enhances the practicality and market value of this application. Furthermore, by optimizing the robot's task execution module, more precise navigation and obstacle avoidance capabilities are achieved, effectively reducing potential safety risks during task execution. Furthermore, the human-computer interaction module provides real-time feedback on task execution status and abnormalities, improving the user's ability to monitor the robot's operation, ensuring safety during use, and enhancing security. It should be noted that natural language interaction reduces the difficulty of communication between the user and the robot. Users do not need to learn complex operating interfaces and commands; they can complete task settings through simple voice commands, significantly lowering the barrier to entry and reducing the difficulty of use.

[0126] In some embodiments, the task information optionally includes at least one task time and a corresponding task execution location; the execution module is specifically configured to: determine a task execution path based on the at least one task time and the corresponding task execution location; and execute the task corresponding to the task information based on the task execution path. In this embodiment, when the voice input contains multiple tasks, the preset language model can parse out the task time corresponding to each task and the task execution location corresponding to the task time during the parsing of the text information, and then determine the task execution path based on the parsed task time and task execution location, so that the device can use the task execution path as a guide for task execution during operation, thereby more accurately understanding user needs and improving the success rate of task execution.

[0127] In the above embodiments, the task execution path can be understood as the order in which multiple tasks are executed in sequence. In some embodiments, the task execution path uses the task time corresponding to each task as the sorting condition, where the task time includes the start time and end time of the task execution, so as to improve the reliability of task execution. In addition, the modular design ensures good coordination and compatibility between the modules, allowing for easy expansion and upgrade to adapt to changing user needs and technological advancements.

[0128] In some embodiments, the robot is optionally a cleaning robot or a service robot, and the robot voice control device further includes: a position information module for obtaining the robot's current position information; and a path planning module for determining a navigation path based on the current position information and the task execution path; wherein the robot moves based on the navigation path. In this embodiment, during the operation of the robot, the current position information is obtained so that the robot's movement is guided based on the current position information and the task execution path. During this process, the robot can achieve autonomous walking and automatically perform tasks. In this process, the user does not need to participate in the movement of the robot, overcoming the disadvantage of the robot requiring manual user participation to operate, thereby improving operation efficiency.

[0129] In some embodiments, optionally, the service robot can be a guest room robot in a hotel, or a mobile robot used to serve food in a restaurant. The service robot can deploy different functions according to its different usage scenarios, which will not be elaborated here.

[0130] In some embodiments, the current position information can optionally be obtained by a position sensor provided on the robot. The position sensor can be selected based on the actual usage scenario, and its specific selection is not described here. In this embodiment, it is not only suitable for indoor navigation tasks in a home environment, but can also be widely used in multiple fields such as commerce, industry, and medical care to achieve voice semantic navigation in various scenarios. This further improves practicality and market value.

[0131] In some embodiments, the system optionally further includes a sensor module for acquiring environmental information about the robot's environment; and a path planning module for updating the navigation path based on the environmental information. In this embodiment, by acquiring environmental information, the robot can update its navigation path based on the environmental information during its movement, thereby reducing collisions with other objects in the environment. This allows for dynamic obstacle avoidance, maximizing the robot's own safety while minimizing the impact of the robot's operation on other objects in the environment.

[0132] In some embodiments, the environmental information optionally includes information about the placement of objects in the environment other than the robot. In some embodiments, the environmental information optionally includes information about the placement of other objects as well as their position and orientation. In this embodiment, obtaining this information enables maximum obstacle avoidance.

[0133] In some embodiments, environmental information can optionally be acquired and sensed using one or more of a camera, lidar, or infrared sensor. In this embodiment, module optimization enables more precise navigation and obstacle avoidance, effectively reducing potential safety risks during mission execution. Furthermore, the human-machine interaction module can provide real-time feedback on mission execution and abnormalities, enhancing the user's ability to monitor robot operations and ensuring safety during use.

[0134] In some embodiments, optionally, it further includes: a time management module for determining the start time, end time and target task associated with the keyword when the task information includes a keyword for indicating repeated execution; and determining the periodic task based on the start time, end time and target task. In this embodiment, the keyword can be a word that can express repeated execution, such as weekly, every interval, monthly, and daily. It can be selected according to the actual usage scenario and will not be described in detail here. In the above embodiment, the start time, end time and target task can be obtained by recognizing and parsing the voice input, and will not be described in detail here. In the above embodiment, by determining the periodic task so that the robot can perform the task periodically, there is no need for the user to repeatedly issue voice input instructions, which facilitates reducing the frequency of interaction between the user and the robot, thereby improving the user experience of the robot.

[0135] In some embodiments, the system optionally further includes: a cloud computing module for sending text information to a preset language model deployed on a server; and receiving task information output from the preset language model. In this embodiment, the text information is sent so that the preset language model deployed on the server can process the received text information and thereby obtain the task information. In this process, the processing of the text information can be transferred from the local device to the cloud, that is, the server side, and the server's large and sufficient computing power can be used for data processing, thereby increasing the speed of data processing to meet the different needs in different scenarios.

[0136] In some embodiments, as shown in Figure 4, a mobile phone application (APP), software deployed on a device, or a webpage acquires voice input and outputs it to a server. The Flask backend (a framework) in the server outputs it to ChatGPT, which, combined with Low-Rank Adaptation of Large Language Models (LORA) (a model), feeds back feedback information and task information. The device's Manager outputs this to the Planner (planning module), where AI (artificial intelligence) manages the framework. OpenAI is open artificial intelligence, and a token is a string of characters generated by the server.

[0137] In some embodiments, as shown in Figure 5, the present application provides a robot voice control device 500, including a processor 502 and a memory 504, the memory 504 stores programs or instructions that can be run on the processor 502, and when the programs or instructions are executed by the processor 502, the steps of the robot voice control method as described in any of the above embodiments are implemented.

[0138] In some embodiments, the present application provides a readable storage medium, which stores a program or instruction. When the program or instruction is executed by a processor, the steps of the robot voice control method as described in any one of the above embodiments are implemented.

[0139] In some embodiments, the present application provides a device comprising: a robot voice control device as in any of the above aspects; and / or a readable storage medium as described above.

[0140] In some embodiments, optionally, the device comprises one of the following: a cleaning robot or a service robot.

[0141] In some embodiments, the robot voice control method, robot voice control device, robot, and readable storage medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios. Among them, the robot voice control method can be applied to a terminal, and can be specifically executed by hardware or software in the terminal. The terminal includes but is not limited to a portable communication device such as a mobile phone or tablet computer with a touch-sensitive surface (e.g., a touch screen display and / or a touchpad). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., a touch screen display and / or a touchpad).

[0142] In the following embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse, and a joystick.

[0143] The robot voice control method provided in the embodiment of the present application may be executed by a robot, or a functional module or functional entity in the robot that can implement the robot voice control method, or an electronic device that is communicatively connected to the robot. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The robot voice control method provided in the embodiment of the present application is described below using a robot as an example of the execution subject.

[0144] As shown in FIG6 , the robot voice control method includes: step 610 , step 620 and step 630 .

[0145] Step 610: Recognize and convert the acquired user's voice input to obtain a target control instruction.

[0146] In this step, the user's voice input, that is, the user task instruction is an instruction for controlling the robot to perform the corresponding action. The user task instruction can be a voice instruction. The user task instruction can be a simple word instruction such as "sweep the floor" or "mop the floor", etc., and the user task instruction can also be a more complex language instruction, such as a paragraph instruction such as "clean up the trash can in the living room at 3 o'clock this afternoon". The target control instruction is an instruction that can be recognized by the robot and is used to control the robot to perform an action corresponding to the user task instruction. In the actual execution process, conventional semantic recognition technology can be used to perform semantic recognition on the user task instruction and convert it into a language that can be recognized by the robot to obtain the target control instruction. Of course, in other embodiments, the user task instruction can also be other instructions besides voice instructions, such as instructions input through the human-computer interaction panel or action instructions collected by the image sensor, etc., which are not limited in this application.

[0147] Step 620: Control the robot based on the target control instruction and obtain feedback information.

[0148] In this step, the robot can be any type of robot, including but not limited to household robots, industrial robots, or robots used in the service field. Feedback information may include: the execution results returned by the robot when controlling the robot to perform the relevant task based on the target control instruction, as well as the user's feedback on the robot's performance of the relevant task. In some embodiments, the execution result may include completion or error reporting.

[0149] Step 630: Optimize target algorithm parameters based on the feedback information; the target algorithm parameters include at least one of model parameters for recognizing voice input and a control strategy for controlling the robot based on the target control instruction.

[0150] The model corresponding to the model parameters can be any network model, such as a large language model. The optimized target algorithm parameters can be used for the next robot control process. For example, when the target algorithm parameters are model parameters for recognizing voice input, the next time a user task instruction is received, the user task instruction can be semantically recognized based on the optimized model. For another example, when the target algorithm parameters are a control strategy for controlling the robot based on a target control instruction, after the user task instruction is converted into a target control instruction next time, the robot can be controlled to perform the corresponding action based on the target control instruction and the optimized target algorithm parameters.

[0151] In this application, by identifying and converting the acquired user task instructions, target control instructions that the robot can recognize are obtained, so that the robot can recognize complex user instructions, and thus can be applied to complex and flexible application scenarios, thereby improving the robot's execution efficiency and flexibility, and improving the user experience; in addition, the target algorithm parameters are optimized through feedback information to achieve autonomous optimization and learning of the target algorithm parameters, so as to improve the accuracy of the robot's task execution in subsequent applications, and further improve the user experience.

[0152] According to the robot voice control method provided in the embodiment of the present application, the acquired user task instructions are identified and converted to obtain target control instructions that can be recognized by the robot, and the target algorithm parameters are optimized based on the feedback information corresponding to the execution results. It can be applied to complex and flexible application scenarios, improve the execution efficiency and flexibility of the robot, and improve the accuracy of the robot's task execution in subsequent applications, which helps to improve the user experience.

[0153] As shown in Figure 7, in some embodiments, step 610 may include: inputting the user's voice input into the large language model, obtaining intermediate instructions output by the large language model; and converting the intermediate instructions to obtain target control instructions. In this embodiment, the large language model (LLM) is a deep learning model trained on massive text data using advanced natural language processing technology. It can generate natural language text, deeply understand the meaning of text, and handle various natural language tasks.

[0154] In some embodiments, the large language model can be a deep learning model, including but not limited to GPT or BERT. The intermediate instructions are obtained by semantically recognizing the user task instructions. It is understandable that the intermediate instructions are a language in an intermediate state and cannot be directly recognized by the machine. Among them, BERT, Bidirectional Encoder Representations from Transformers is a pre-trained deep learning model. For example, for the user task instruction "Clean up the trash can in the living room at 3 o'clock this afternoon", the keywords can be extracted through the large language model to obtain intermediate instructions: "3 o'clock in the afternoon", "living room", "trash can" and "clean up" and other keywords used to represent time, place and tasks.

[0155] In some embodiments, the large language model can also identify and correct common language errors, such as colloquial expressions and grammatical errors, to ensure correct understanding of the instructions. The target control instruction is a machine-recognizable language instruction obtained by standardizing the intermediate instructions.

[0156] In some embodiments, converting the intermediate instructions to obtain the target control instructions may include: standardizing the basic information included in the intermediate instructions to generate standardized data; matching the target function corresponding to the task information from multiple candidate functions based on the task information included in the intermediate instructions; and generating the target control instructions based on the standardized data and the target function. In this embodiment, the basic information is attribute information used to characterize the task, including but not limited to: time and location. Standardized data is data that can be recognized by the robot. Task information is information related to the task to be performed, such as sweeping, dragging, grabbing, and opening doors. Candidate functions are pre-set control instructions for performing different actions. During the actual execution process, an association table between task information and candidate functions can be established in advance to facilitate matching of candidate functions corresponding to the task information based on the association table. Continuing with the user task instruction "Clean the trash can in the living room at 3 o'clock this afternoon" as an example, the large language model can extract the keywords to obtain intermediate instructions: "3 o'clock in the afternoon", "living room", "trash can" and "clean" and other intermediate instructions. The time information can be further converted into "2023010115:00", and then search whether there is a living room in the map corresponding to the room and whether there is a trash can in the living room. If the trash can is found, match the unique code (Identity Document, ID) corresponding to the trash can in the map to obtain the location information of the trash can. Then, based on the task information "clean", the target function "sweep" corresponding to the task information "clean" is matched from multiple pre-set candidate functions, and "clean" is mapped to the "sweep" function.

[0157] In some embodiments, when a target control instruction includes multiple sub-control instructions, controlling a robot based on the target control instruction may include: determining a target execution order for the multiple sub-control instructions based on the priority of each sub-control instruction; storing the multiple sub-control instructions in a task stack based on the target execution order; and deleting the target sub-control instruction from the task stack after the target sub-control instruction among the multiple sub-control instructions has been executed based on the task stack. In this embodiment, the priority level can be determined based on the importance of the task and the execution order. The priority level can be user-defined or automatically generated by the system, and this application is not limited here. The target execution order is the execution order of the sub-control instructions. The task stack is a container for activity instances in a stack format, following the first-in, last-out principle, where activity instances correspond one-to-one to sub-control instructions. An activity instance is a component in Android. The target sub-control instruction can be any sub-control instruction among the multiple sub-control instructions, and the target sub-control instruction is the subtask instruction currently being executed in the task stack. After the target sub-control instruction has been executed, it can be deleted from the task stack. During the actual execution process, the robot can count all the sub-control instructions to be executed the next day the night before, and then determine the execution order corresponding to each sub-control instruction to obtain the target execution order, and then store multiple sub-control instructions in the task stack based on the target execution order; on the second day, the robot executes each sub-control instruction in sequence based on the storage order of the task stack. After executing a sub-control instruction, the sub-control instruction can be deleted from the task stack until all sub-control instructions in the task stack are cleared.

[0158] In some embodiments, when a new sub-control instruction is added, the new sub-control instruction can be placed in a task stack to wait for the robot to execute until the robot has completed executing all sub-control instructions. According to the robot voice control method provided in the embodiment of the application, by storing sub-control instructions in the form of a task stack, it is possible to achieve automatic task hosting, so that the robot can execute various tasks in sequence, improve the stability of the robot's operation and the orderliness of task execution, and further improve the level of intelligence.

[0159] Continuing with Figure 7, in some embodiments, controlling a robot based on target control instructions may include: generating a target control strategy based on image information collected by the robot and the target control instructions; and controlling the robot based on the target control strategy. In this embodiment, the target control strategy is a specific task execution strategy corresponding to the target control instructions that the robot can execute. It is used to plan the robot's path and the corresponding position and posture at each point along the path. Image information is used to represent environmental information, obstacles, and task objectives during the robot's operation.

[0160] In some embodiments, environmental information may include lighting and complex backgrounds. Image information can be acquired based on an image sensor mounted on the robot. In actual implementation, image recognition can be performed on the image information to perform object recognition, environmental analysis, spatial positioning, and semantic segmentation. Based on the recognition results and combined with target control instructions, the robot is then controlled to run along the determined path and achieve the corresponding posture at the corresponding location.

[0161] In some embodiments, the target control strategy is generated based on the image information and target control instructions collected by the robot, and the method may also include: generating the target control strategy based on the image information, target control instructions and other sensor data. In this embodiment, other sensor data may include but is not limited to tactile, sound, and infrared sensor data. By combining visual data with other sensor data from other sensors to obtain the target control strategy, the precision and accuracy of the target control strategy can be further improved so that the robot can make more accurate decisions. According to the robot voice control method provided in the embodiment of the present application, by combining the visual perception results to determine the target control strategy to control the robot to perform the corresponding task, the efficiency and accuracy of the robot in performing the task can be improved.

[0162] In some embodiments, the method may further include: performing feature recognition on the captured image information to obtain target features and corresponding position information of the target features; and matching a target control strategy from a plurality of control strategies based on the target features and corresponding position information. In this embodiment, the target features may be objects identified by the robot and to be processed. The control strategies may be pre-set strategies for the robot to perform different actions or functions.

[0163] Taking a robot vacuum as an example, the control strategies may include sweeping, vacuuming, mopping, and navigation.

[0164] It is understandable that the strategies to be executed by the robot may also be different depending on the target features. For example, when the target feature is melon seed shells, the strategy to be executed may be sweeping the floor; when the target feature is water stains, the strategy to be executed may be mopping the floor. Position information is used to characterize the actual position of the target feature in the current scene. During the actual execution process, if the robot recognizes that there are a bunch of melon seed shells in the collected image information, the melon seed shells can be determined as the target feature, and the position information of the melon seed shells can be identified. The melon seed shells and the corresponding position information of the melon seed shells are then sent to the large language model. The large language model matches the corresponding target control strategy based on the received information, such as matching the control strategy corresponding to sweeping the floor. The large language model can be pre-trained to determine whether the received target feature needs to be processed, and if it is determined that it needs to be processed, further match the corresponding control strategy.

[0165] According to the robot voice control method provided in the embodiment of the present application, by combining the visual perception results to collect the on-site environment image, the corresponding control strategy is automatically matched based on the image analysis results to control the robot to execute the relevant strategy without the need for the user to issue instructions, and it has a higher degree of automation and intelligence.

[0166] Continuing with FIG. 7 , in some embodiments, step 630 may include optimizing model parameters based on at least a portion of the feedback information within a first time period and the user task instructions corresponding to the feedback information. In this embodiment, the first time period may be user-defined, such as set to one month, six months, or one year, though this application is not limiting. The feedback information may include user evaluation information. User evaluation information is user evaluation information on the robot's performance of the user task instructions, such as satisfaction with cleaning performance or coffee making. For example, the robot collects user evaluation information based on the results of each user task instruction after executing the user task instruction. Six months later, the robot optimizes the large language model based on the user evaluation information collected within the six-month period and the user task instructions corresponding to the user evaluation information, thereby improving the accuracy and precision of the model parameters. This ensures that the intermediate instructions recognized by the large language model are closer to the user's true intentions. According to the robot voice control method provided in the embodiments of the present application, optimizing model parameters based on feedback information can improve the accuracy and precision of recognizing complex user instructions, enabling the execution of more complex or varied tasks, broadening application scenarios and functionality.

[0167] Continuing with Figure 7, in some embodiments, step 630 may include optimizing the control strategy based on at least a portion of the feedback information within a second duration and the target control instructions corresponding to the feedback information. In this embodiment, the second duration may be user-defined, such as one day, two days, or one week, though this application is not limiting. In some embodiments, the second duration may be shorter than the first duration. Feedback information may include user evaluation information and error reporting information. Error reporting information is information reported by the robot when it fails to execute the target control instructions. Error reporting information may include information such as the fault status, cause of the fault, and time of the fault. For example, the robot may collect error reporting information each time it fails to execute a task. At the end of the day, data recognition and learning are performed on one or more of the collected error reporting information to optimize the control strategy to reduce the probability of task failure the next time the same task is executed. For another example, based on user evaluation information on each task execution result, the robot may learn user preferences, such as cleaning frequency or coffee strength, to optimize the control strategy so that the final task execution result more closely matches the user's preferences.

[0168] According to the robot voice control method provided in the embodiment of the present application, the control strategy is optimized through the feedback information during the robot's task execution, and autonomous optimization and learning of the control strategy are achieved. This can not only improve the accuracy and precision of the control strategy and reduce the error rate of subsequent task execution; it can also learn the user's preferences to make the control strategy closer to the user's preferences, realize user personalized and stylized services, help improve the user experience, and have a high degree of intelligence.

[0169] Continuing with reference to FIG7 , in some embodiments, controlling a robot based on target control instructions may further include: predicting risk information based on sensor data collected by the robot and the robot's posture information; and performing a target operation based on the risk information. In this embodiment, the posture information includes the robot's position information and posture information. The sensor data may be data collected by any one or more sensors. The sensor data may include, but is not limited to, image data, radar data, infrared data, audio information, and tactile data. The risk information may be potential safety risk information, including, but not limited to, collision risk, operational error risk, and other risks. The target operation may include, but is not limited to, outputting warning information to the user, controlling the robot to pause the execution of a task, and other operations.

[0170] In the process of the robot executing the target control instruction, it can also collect sensor data in the environment in real time to identify and analyze the sensor data, such as collision analysis and error operation analysis, etc., to determine whether there will be risks such as collision when the robot continues to execute the target control instruction based on the current target control strategy; when it is predicted that there is risk information, corresponding measures can be taken based on the risk information to avoid the risk, such as controlling the robot to stop running when it is detected that a collision is about to occur. When it is predicted that there is no risk information, the robot can be controlled to maintain the current target control strategy and continue to execute the target control instruction. According to the robot voice control method provided in the embodiment of the present application, risk information is predicted by sensor data, and potential safety risks can be detected so that when it is determined that there is a safety risk, corresponding operations can be taken to avoid the risk, thereby improving the safety of the robot operation and the service life of the robot.

[0171] Continuing to refer to FIG. 7 , in the actual implementation process, the modules shown in FIG. 7 may be set to execute corresponding steps, thereby realizing the control of the robot.

[0172] The robot voice control method provided in the embodiment of the present application can be executed by a robot voice control device. In the embodiment of the present application, the robot voice control method is executed by a robot voice control device as an example to illustrate that the robot voice control device provided in the embodiment of the present application is provided.

[0173] The present application also provides a robot voice control device. As shown in Figure 8, the robot voice control device includes a first processing module 810, a second processing module 820, and a third processing module 830. The first processing module 810 is configured to recognize and convert the user's voice input to obtain a target control instruction; the second processing module 820 is configured to control the robot based on the target control instruction and obtain feedback information; and the third processing module 830 is configured to optimize target algorithm parameters based on the feedback information. The target algorithm parameters include at least one of a model parameter for recognizing the voice input and a control strategy for controlling the robot based on the target control instruction.

[0174] According to the robot voice control device provided in the embodiment of the present application, the user's voice input, that is, the user's task instructions, is recognized and converted to obtain target control instructions that the robot can recognize, and the target algorithm parameters are optimized based on the feedback information corresponding to the execution results. It can be applied to complex and flexible application scenarios, improve the execution efficiency and flexibility of the robot, and improve the accuracy of the robot's task execution in subsequent applications, which helps to improve the user experience.

[0175] In some embodiments, the first processing module 810 may also be used to: input the user task instruction into the large language model to obtain the intermediate instruction output by the large language model; and convert the intermediate instruction to obtain the target control instruction.

[0176] In some embodiments, the first processing module 810 can also be used to: standardize the basic information included in the intermediate instructions to generate standardized data; based on the task information included in the intermediate instructions, match the target function corresponding to the task information from multiple candidate functions; and generate target control instructions based on the standardized data and the target function.

[0177] In some embodiments, the second processing module 820 can also be used to: determine the target execution order of multiple sub-control instructions based on the priority level of each sub-control instruction; store the multiple sub-control instructions in the task stack based on the target execution order; and delete the target sub-control instruction from the task stack when the target sub-control instruction among the multiple sub-control instructions is executed based on the task stack.

[0178] In some embodiments, the second processing module 820 may also be used to: generate a target control strategy based on image information and target control instructions collected by the robot; and control the robot based on the target control strategy.

[0179] In some embodiments, the third processing module 830 may also be configured to optimize model parameters based on at least a portion of the feedback information within the first duration and a user task instruction corresponding to the feedback information.

[0180] In some embodiments, the third processing module 830 may also be configured to optimize the control strategy based on at least part of the feedback information within the second time period and the target control instruction corresponding to the feedback information.

[0181] In some embodiments, the second processing module 820 may also be used to: predict risk information based on sensor data collected by the robot and the robot's posture information; and perform target operations based on the risk information.

[0182] The robot voice control device in the embodiment of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or chip. The electronic device can be a terminal or other device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not specifically limit it.

[0183] The robot voice control device in the embodiment of the present application can be a device having an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0184] The robot voice control device provided in the embodiment of the present application can implement each process implemented in the method embodiments of Figures 6 to 7. To avoid repetition, they are not described here.

[0185] In some embodiments, the embodiments of the present application further provide a robot, comprising: a robotic arm and a robot voice control device as described in any of the above embodiments. In this embodiment, the robot voice control device is electrically connected to the robotic arm. The robot voice control device is used to execute the robot voice control method as described in any of the above embodiments. According to the robot provided in the embodiments of the present application, by recognizing and converting the acquired user's voice input, a target control instruction that the robot can recognize is obtained, and the target algorithm parameters are optimized based on the feedback information corresponding to the execution result. It can be applied to complex and flexible application scenarios, improve the execution efficiency and flexibility of the robot, and improve the accuracy of the robot's task execution in subsequent applications, which helps to improve the user experience.

[0186] As shown in Figure 9, the embodiment of the present application further provides an electronic device 900, including a processor 901, a memory 902, and a computer program stored in the memory 902 and executable on the processor 901. When the program is executed by the processor 901, each process of the above-mentioned robot voice control method embodiment is implemented, and the same technical effect is achieved. To avoid repetition, it is not described here. It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0187] The present application also provides a readable storage medium, which is a non-transitory computer-readable storage medium. A computer program is stored on the non-transitory computer-readable storage medium. When the computer program is executed by a processor, the various processes of the above-mentioned robot voice control method embodiment are implemented, and the same technical effects are achieved. To avoid repetition, the details are not repeated here. The processor is the processor in the electronic device in the above-mentioned embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0188] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the aforementioned robot voice control method. The processor is the processor in the electronic device described in the aforementioned embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0189] The present application also provides a chip, which includes a processor and a communication interface, the communication interface and the processor being coupled, and the processor being used to execute programs or instructions to implement the various processes of the above-mentioned robot voice control method embodiment, and to achieve the same technical effects. To avoid repetition, the description is omitted here. It should be understood that the chip mentioned in the present application embodiment can also be referred to as a system-on-chip, system-on-chip, chip system, or system-on-chip chip, etc.

[0190] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0191] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the embodiments of the present application are essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0192] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

[0193] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0194] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and intent of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A robot voice control method, wherein: include: Get the user's voice input; Processing the user's voice input to obtain a task suitable for the robot to perform; Control the robot to perform the task.

2. The robot voice control method according to claim 1, wherein: The processing of the user's voice input to obtain a task suitable for the robot to perform specifically includes: Performing speech recognition on the speech input to obtain text information; Parsing the text information using a preset language model to obtain task information; and A task corresponding to the task information is determined.

3. The robot voice control method according to claim 2, wherein: In the process of generating the task information, the robot voice control method further includes: Generate natural language feedback information based on the preset language model; Before controlling the robot to perform the task, the robot voice control method further includes: synthesizing speech information based on the feedback information; Play the voice message.

4. The robot voice control method according to claim 2, wherein: The performing speech recognition on the speech input to obtain text information specifically includes: The speech input is subjected to speech recognition using a speech recognition model based on deep learning to obtain the text information.

5. The robot voice control method according to claim 3, wherein: The synthesizing speech information based on the feedback information specifically includes: Based on an end-to-end speech synthesis model of deep learning, speech synthesis is performed on the feedback information to obtain the speech information.

6. The robot voice control method according to any one of claims 2 to 5, wherein: The task information includes at least one task time and a corresponding task execution location; The controlling the robot to perform the task specifically includes: Determine a task execution path based on at least one of the task time and the corresponding task execution location; Based on the task execution path, a task corresponding to the task information is executed.

7. The robot voice control method according to claim 6, wherein: The robot is a cleaning robot or a service robot, and the robot voice control method further comprises: Obtaining current position information of the robot; Determine a navigation path according to the current location information and the task execution path; Wherein, the robot walks based on the navigation path.

8. The robot voice control method according to claim 7, wherein: The robot voice control method also includes: Acquiring environmental information of the environment in which the robot is located; The navigation path is updated based on the environmental information.

9. The robot voice control method according to claim 6, wherein: The robot voice control method also includes: In a case where the task information includes a keyword for indicating repeated execution, determining a start time, an end time, and a target task associated with the keyword; A periodic task is determined based on the start time, the end time, and the target task.

10. The robot voice control method according to claim 2, wherein: The adopting a preset language model to parse the text information to obtain task information includes: Sending the text information to the preset language model deployed on the server; Receive task information output from the preset language model.

11. The robot voice control method according to claim 1, wherein: The processing of the user's voice input to obtain a task suitable for the robot to perform specifically includes: Recognize and convert the acquired voice input of the user to obtain a target control instruction; The controlling the robot to perform the task specifically includes: Controlling the robot based on the target control instruction and obtaining feedback information; Based on the feedback information, target algorithm parameters are optimized; the target algorithm parameters include at least one of a model parameter for recognizing the voice input and a control strategy for controlling the robot based on the target control instruction.

12. The robot voice control method according to claim 11, wherein: The step of recognizing and converting the acquired voice input of the user to obtain a target control instruction includes: Inputting the user's voice input into a large language model, and obtaining an intermediate instruction output by the large language model; The intermediate instruction is converted to obtain the target control instruction.

13. The robot voice control method according to claim 12, wherein: The converting process of the intermediate instruction to obtain the target control instruction includes: Standardizing the basic information included in the intermediate instruction to generate standardized data; Based on the task information included in the intermediate instruction, matching a target function corresponding to the task information from a plurality of candidate functions; The target control instruction is generated based on the standardized data and the target function.

14. The robot voice control method according to any one of claims 11 to 13, wherein: In a case where the target control instruction includes a plurality of sub-control instructions, controlling the robot based on the target control instruction includes: Determining a target execution order of the plurality of sub-control instructions based on a priority level of each of the sub-control instructions; storing the plurality of sub-control instructions into a task stack based on the target execution order; When the target sub-control instruction among the plurality of sub-control instructions is executed based on the task stack, the target sub-control instruction is deleted from the task stack.

15. The robot voice control method according to any one of claims 11 to 13, wherein: The controlling the robot based on the target control instruction comprises: Generate a target control strategy based on the image information collected by the robot and the target control instruction; Based on the target control strategy, the robot is controlled.

16. The robot voice control method according to any one of claims 11 to 13, wherein: The optimizing the target algorithm parameters based on the feedback information includes: Based on at least part of the feedback information within the first duration and the user task instruction corresponding to the feedback information, the Model parameters.

17. The robot voice control method according to any one of claims 11 to 13, wherein: The optimizing the target algorithm parameters based on the feedback information includes: The control strategy is optimized based on at least part of the feedback information within a second duration and a target control instruction corresponding to the feedback information.

18. The robot voice control method according to any one of claims 11 to 13, wherein: The controlling the robot based on the target control instruction comprises: Predicting risk information based on sensor data collected by the robot and position information of the robot; Based on the risk information, a target operation is performed.

19. A robot voice control device, wherein: include: An acquisition module is used to obtain the user's voice input; A processing module, used to process the user's voice input to obtain a task suitable for the robot to perform; A control module is used to control the robot to perform the task.

20. The robot voice control device according to claim 19, wherein: The processing module is a speech recognition module, and the speech recognition module is used to perform speech recognition on the speech input to obtain text information; and A task corresponding to the task information is determined.

21. The robot voice control device according to claim 19, wherein: The processing module is a first processing module, which is used to recognize and convert the acquired user's voice input to obtain a target control instruction; The control module includes a second processing module and a third processing module, wherein the second processing module is used to control the robot based on the target control instruction and obtain feedback information; The third processing module is used to optimize target algorithm parameters based on the feedback information; the target algorithm parameters include at least one of a model parameter for identifying the voice input and a control strategy for controlling the robot based on the target control instruction.

22. A robot voice control device, wherein: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the robot voice control method according to any one of claims 1 to 18 are implemented.

23. A readable storage medium, wherein: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the robot voice control method according to any one of claims 1 to 18 are implemented.

24. A robot, wherein: include: Robotic arm; The robot voice control device according to any one of claims 19 to 21, wherein the robot voice control device is electrically connected to the robot arm.

Images