Control instruction response method and device, robot and storage medium

Abstract

The invention relates to a control instruction response method and device, a robot and a storage medium. The method comprises the steps of collecting an audio signal when a preset triggering condition is met; performing semantic recognition processing on the audio signal to obtain corresponding text information; the text information is judged through a pre-established large language model agent, a response mode is determined, and the response mode comprises action execution and / or question answering; rehearsing the response mode based on the constructed robot model, and evaluating a rehearsing result of the response mode through a large language model agent to obtain a corresponding evaluation result; and when the evaluation result meets the requirement, controlling the robot to broadcast a question answer generated by the robot model during rehearsal, and / or execute an action sequence generated by the robot model during rehearsal. Therefore, the response mode can be adaptively determined according to the text content corresponding to the audio signal, so that the response mode is more diversified, the response feedback is more intelligent, and the interaction experience is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a control instruction response method, device, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of artificial intelligence technology, more and more electronic devices can support human-computer interaction functions. Among them, the human-computer interaction function supports users to wake up voice assistants, send voice commands to electronic devices, have conversations and ask questions with electronic devices, etc., so that users can quickly acquire knowledge and control devices.

[0003] In traditional technology, robots generally respond to control commands in the form of voice by giving direct voice answers. For example, after recognizing the voice, they obtain the corresponding answer from the knowledge base and then broadcast the answer. This response mode has a low degree of intelligence, especially when applied to humanoid robots, where users prefer robots to respond to user control commands in an anthropomorphic manner.

[0004] However, when applying traditional technology and robot Q&A, the robot's response method is single and the interactivity is poor. Summary of the invention

[0005] Based on this, it is necessary to provide a control instruction response method, device, computer equipment, computer-readable storage medium and computer program product that can enrich the response mode and enhance interactivity in response to the above technical problems.

[0006] In a first aspect, the present application provides a control instruction response method, comprising:

[0007] When the preset trigger conditions are met, the audio signal is collected;

[0008] Performing semantic recognition processing on the audio signal to obtain corresponding text information;

[0009] The text information is judged by a pre-established large language model agent to determine a response method, wherein the response method includes: performing an action and / or answering a question;

[0010] Previewing the response method based on the constructed robot model, and evaluating the preview result of the response method through the large language model agent to obtain a corresponding evaluation result;

[0011] When the evaluation result meets the requirements, the robot is controlled to broadcast the answer to the question generated by the robot model during the preview, and / or execute the action sequence generated by the robot model during the preview.

[0012] In one embodiment, the preset trigger condition includes at least one of the following:

[0013] An object is detected entering and / or a human face is detected within a preset detection range;

[0014] It is detected that the decibel value of the audio signal in the environment is greater than a preset value;

[0015] Detecting that the audio signal contains a target wake-up word;

[0016] The pre-established visual model collects continuous video frames containing the lip area of ​​the human face, and the image recognition results of the continuous video frames indicate the presence of a lip opening and closing movement.

[0017] In one embodiment, the performing semantic recognition processing on the audio signal to obtain corresponding text information includes:

[0018] Using an end-to-end speech recognition model to segment the audio signal according to a preset duration, and converting the segmented audio signal into a feature vector;

[0019] The feature vector is decoded and text information is output.

[0020] In one embodiment, the large language model agent includes: a thinking model, a reasoning and action model, wherein:

[0021] The thinking model is used to determine a response method according to the text information;

[0022] The reasoning and action model is used to generate a control instruction for the robot model according to the response mode, and evaluate the preview result of the robot model executing the control instruction to obtain a corresponding evaluation result;

[0023] The reasoning and action model is further used to regenerate control instructions for the robot model when the evaluation result does not meet the requirements, and evaluate the preview results of the robot model executing the control instructions until the evaluation result meets the requirements;

[0024] The reasoning and action model is also used to feed back response completion indication information to the thinking model when the evaluation result meets the requirements.

[0025] In one embodiment, when the response method includes answering a question, the large language model agent evaluates the preview result of the response method to obtain a corresponding evaluation result, including:

[0026] The reasoning and action model generates a control instruction according to the text information, and sends the control instruction to the robot model, wherein the control instruction is used to instruct the robot model to broadcast the answer to the question;

[0027] The answer to the question is evaluated by the reasoning and action model to determine whether the answer to the question is complete and / or correct.

[0028] In one embodiment, when the response method includes executing an action, evaluating the preview result of the response method by the large language model agent to obtain a corresponding evaluation result includes:

[0029] The reasoning and action model generates a control instruction according to the text information, and sends the control instruction to the robot model, wherein the control instruction is used to instruct the robot model to execute an action sequence;

[0030] The reasoning and action model evaluates the execution result of the action sequence to determine whether the action corresponding to the action sequence is completed and / or correct.

[0031] In one embodiment, before controlling the robot to announce the answers to the questions generated by the robot model during the preview and executing the action sequence generated by the robot model during the preview, the method further includes:

[0032] Setting the answers to the questions generated by the robot model during the preview and the action sequences generated by the robot model during the preview to a number of timestamps;

[0033] Through the correspondence between the timestamps, the text content of the answer to the question and the action sequence are aligned, so that the robot synchronously executes the corresponding action sequence when broadcasting the answer to the question.

[0034] In a second aspect, the present application further provides a control instruction response device, comprising:

[0035] A collection module, used to collect audio signals when a preset trigger condition is met;

[0036] A recognition module, used to perform semantic recognition processing on the audio signal to obtain corresponding text information;

[0037] A determination module, used to identify the text information through a pre-established large language model agent and determine a response method, wherein the response method includes: performing an action and / or answering a question;

[0038] An evaluation module, used for previewing the response mode based on the constructed robot model, and evaluating the preview result of the response mode through the large language model agent to obtain a corresponding evaluation result;

[0039] The control module is used to control the robot to broadcast the answer to the question generated by the robot model during the preview and / or execute the action sequence generated by the robot model during the preview when the evaluation result meets the requirements.

[0040] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0041] When the preset trigger conditions are met, the audio signal is collected;

[0042] Performing semantic recognition processing on the audio signal to obtain corresponding text information;

[0043] The text information is judged by a pre-established large language model agent to determine a response method, wherein the response method includes: performing an action and / or answering a question;

[0044] Previewing the response method based on the constructed robot model, and evaluating the preview result of the response method through the large language model agent to obtain a corresponding evaluation result;

[0045] When the evaluation result meets the requirements, the robot is controlled to broadcast the answer to the question generated by the robot model during the preview, and / or execute the action sequence generated by the robot model during the preview.

[0046] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0047] When the preset trigger conditions are met, the audio signal is collected;

[0048] Performing semantic recognition processing on the audio signal to obtain corresponding text information;

[0049] The text information is judged by a pre-established large language model agent to determine a response method, wherein the response method includes: performing an action and / or answering a question;

[0050] Previewing the response method based on the constructed robot model, and evaluating the preview result of the response method through the large language model agent to obtain a corresponding evaluation result;

[0051] When the evaluation result meets the requirements, the robot is controlled to broadcast the answer to the question generated by the robot model during the preview, and / or execute the action sequence generated by the robot model during the preview.

[0052] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0053] When the preset trigger conditions are met, the audio signal is collected;

[0054] Performing semantic recognition processing on the audio signal to obtain corresponding text information;

[0055] The text information is judged by a pre-established large language model agent to determine a response method, wherein the response method includes: performing an action and / or answering a question;

[0056] Previewing the response method based on the constructed robot model, and evaluating the preview result of the response method through the large language model agent to obtain a corresponding evaluation result;

[0057] When the evaluation result meets the requirements, the robot is controlled to broadcast the answer to the question generated by the robot model during the preview, and / or execute the action sequence generated by the robot model during the preview.

[0058] The control instruction response method, device, computer equipment, computer-readable storage medium and computer program product described above collect audio signals when the preset trigger conditions are met; thus, the audio signals can be actively collected when the preset trigger conditions are met, avoiding long-term monitoring of voice signals in the environment and reducing power consumption. The audio signals are semantically recognized and processed to obtain corresponding text information; the text information is judged by a pre-established large language model agent to determine the response method, which includes: executing actions and / or answering questions; thus, the response method can be determined according to the text corresponding to the voice information, making the response method more abundant and intelligent. The response method is previewed based on the constructed robot model, and the preview result of the response method is evaluated by the large language model agent to obtain the corresponding evaluation result; thus, the result of the response method can be previewed and evaluated in advance, making the control instruction finally executed by the robot more accurate. When the evaluation result meets the requirements, the robot is controlled to broadcast the answer to the question generated by the robot model during the preview, and / or execute the action sequence generated by the robot model during the preview. In this way, the response method can be adaptively determined according to the text content corresponding to the audio signal, making the response method more diversified, the response feedback more intelligent, and improving the interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 An application environment diagram of a control instruction response method in an embodiment;

[0061] Figure 2 A schematic diagram of a flow chart of a control instruction response method in an embodiment;

[0062] Figure 3 A schematic diagram of the working principle of a large language model agent in one embodiment;

[0063] Figure 4 A schematic diagram of a flow chart of a control instruction response method in another embodiment;

[0064] Figure 5 is a structural block diagram of a control instruction response device in an embodiment;

[0065] Figure 6 is a structural block diagram of a control instruction response device in another embodiment;

[0066] Figure 7 FIG. 4 is a diagram showing the internal structure of a processing system of a robot in one embodiment. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0068] The control instruction response method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the robot 101 communicates with the server through the network. An offline knowledge base is stored in the local memory of the robot 101, and the robot 101 can also obtain data in the cloud knowledge base from the server through the network. Exemplarily, each robot 101 has its own control range (as shown in the dotted box). When someone enters this control range and meets the preset trigger conditions, the robot 101 collects audio signals; performs semantic recognition processing on the audio signals to obtain corresponding text information; discriminates the text information through the pre-established large language model agent and determines the response method, which includes: executing actions and / or answering questions; previewing the response method based on the constructed robot model, and evaluating the preview results of the response method through the large language model agent to obtain corresponding evaluation results; when the evaluation results meet the requirements, the robot 101 is controlled to broadcast the answers to the questions generated by the robot model during the preview, and / or execute the action sequence generated by the robot model during the preview.

[0069] In an exemplary embodiment, Figure 2 As shown, a control instruction response method is provided, which is applied to Figure 1 The robot in the example is used to illustrate, including the following steps 201 to 205. Among them:

[0070] Step 201: When a preset trigger condition is met, an audio signal is collected.

[0071] In this embodiment, the audio signal is collected only when a preset trigger condition is met, thereby reducing the power consumption of the robot and preventing the robot from being in the audio signal collection state for a long time.

[0072] Exemplarily, the preset trigger condition includes at least one of the following:

[0073] 1) An object is detected entering the preset detection range and / or a human face is detected.

[0074] 2) The decibel value of the audio signal in the environment is detected to be greater than the preset value.

[0075] 3) Detect that the audio signal contains the target wake-up word.

[0076] 4) The pre-established visual model collects continuous video frames containing the lip area of ​​the human face, and the image recognition results of the continuous video frames indicate the presence of lip opening and closing movements.

[0077] For method 1), various distance sensors can be used to detect whether an object enters the preset detection range. For example, ultrasonic sensors, infrared sensors, laser sensors, and radar sensors are all distance sensors, that is, they detect objects within a certain range from the robot.

[0078] For method 1), a visual sensor can also be used to detect whether a face image appears within a preset detection range. If a face image exists, it means that someone is approaching.

[0079] For example, when the robot uses a visual sensor, target person recognition can also be set, that is, the robot is awakened only when the target person enters the control range of the robot. For example, when the visual sensor detects that a person has entered the control range, a face image of the person who has entered the control range is collected, and the face image is compared with a reference target face image. If the comparison is successful, the robot is awakened to collect audio signals.

[0080] Regarding method 2), when the robot collects audio signals from the surrounding environment, it does not parse the content of the audio signals, but only judges the decibel value of the audio signals. When the decibel value is greater than the preset value, the robot is triggered to collect audio signals for a long period of time. At this time, the collected audio signals need to be processed for semantic recognition to obtain the corresponding text information.

[0081] For method 3), the robot is awakened to collect audio signals only when the target wake-up word is detected.

[0082] Regarding method 4), compared with other methods, it is a more accurate detection method. Through the pre-established visual model, continuous video frames containing the lip area of ​​the face are collected, and the continuous video frames are recognized and processed to determine whether there is a lip opening and closing movement; if there is a lip opening and closing movement, a narrow beam is used to more accurately receive the audio signal.

[0083] Step 202: Perform semantic recognition processing on the audio signal to obtain corresponding text information.

[0084] Exemplarily, an end-to-end speech recognition model may be used to segment the audio signal according to a preset duration, and the segmented audio signal may be converted into a feature vector; the feature vector may be decoded and processed to output text information.

[0085] In this embodiment, the latest end-to-end Paraformer model can be used to achieve efficient and accurate speech recognition. It can convert the audio signal into a feature vector, and use the acoustic model and language model to process and decode it, and finally output the text. This semantic recognition method has the ability of multi-language support and real-time reasoning, and is suitable for efficient speech recognition in a low-latency environment.

[0086] Step 203: The text information is judged by the pre-established large language model agent to determine the response method.

[0087] The response method includes: performing an action and / or answering a question.

[0088] In this embodiment, the large language model intelligent agent includes: a thinking model and a reasoning and action model, wherein: the thinking model is used to determine a response method according to the text information; the reasoning and action model is used to generate a control instruction for the robot model according to the response method, and evaluate the preview result of the robot model executing the control instruction to obtain a corresponding evaluation result; the reasoning and action model is also used to regenerate the control instruction for the robot model when the evaluation result does not meet the requirements, and evaluate the preview result of the robot model executing the control instruction until the evaluation result meets the requirements; the reasoning and action model is also used to feedback response completion indication information to the thinking model when the evaluation result meets the requirements.

[0089] For example, Figure 3 As shown in the figure, the working principle of the large language model agent is shown. Assume that the text information corresponding to the audio signal is: wave your hand. At this time, the thinking model (Thought) of the large language model agent judges the text information and determines whether to perform an action or answer a question. The judgment result here is: perform an action; then the reasoning and action model (ReAct) of the large language model agent can perform the following steps:

[0090] ReAct1 issues control instructions to the robot model: wave your hand.

[0091] Obs1 determines whether the waving action is completed / correct.

[0092] If it is not completed / incorrect, ReAct2 sends a control instruction to the robot model: wave your hand.

[0093] Obs2 determines whether the waving action is completed / correct.

[0094] If it is not completed / incorrect, ReAct3 sends a control instruction to the robot model: wave your hand.

[0095] Obs3 determines whether the waving action is completed / correct.

[0096] …

[0097] Until the waving action is completed / correct, the robot model gives feedback to Thought and the action is completed.

[0098] The above ReAct1, ReAct2, ReAct3, and Obs1, Obs2, Obs3 respectively represent the three reasoning and dynamic decision-making processes of the reasoning and action model. Among them, ReAct is an intelligent agent model that combines reasoning and action, which aims to enable the model to reason when performing tasks and take corresponding actions based on the reasoning results. It can handle complex problems through reasoning, generate reasonable intermediate reasoning steps, and help intelligent agents make wise decisions. When performing tasks, ReAct can dynamically adjust decision-making strategies according to the progress of tasks and changes in the environment to adapt to different situations. In addition, ReAct can perform self-optimization through reinforcement learning, gradually improve the effectiveness of decisions and actions, and break down complex tasks into multiple small tasks, and complete the entire goal by gradually solving these subtasks.

[0099] Optionally, by combining multiple information sources such as text, voice or images, ReAct can perceive the environment more comprehensively and make more appropriate decisions based on information from different modalities.

[0100] Similarly, the thinking model (Thought) of the large language model agent judges the text information and determines whether to perform an action or answer a question. The judgment result here is: answer the question; then the reasoning and action model (ReAct) of the large language model agent can perform the following steps:

[0101] ReAct1 answers related questions.

[0102] Obs1 determines whether the answer is complete / correct.

[0103] If incomplete / incorrect, ReAct2 will answer the relevant questions again.

[0104] Obs2 determines whether the answer is complete / correct.

[0105] If incomplete / incorrect, ReAct3 will answer the relevant questions again.

[0106] Obs3 determines whether the answer is complete / correct.

[0107] …

[0108] The answer is complete when it is complete / correct.

[0109] Step 204: preview the response method based on the constructed robot model, and evaluate the preview result of the response method through the large language model intelligent agent to obtain a corresponding evaluation result.

[0110] Exemplarily, when the response method includes answering a question, the reasoning and action model generates a control instruction based on the text information and sends the control instruction to the robot model, wherein the control instruction is used to instruct the robot model to broadcast the answer to the question; the reasoning and action model evaluates the answer to the question to determine whether the answer to the question is complete and / or correct.

[0111] Exemplarily, when the response method includes executing an action, the reasoning and action model generates a control instruction based on the text information and sends the control instruction to the robot model, wherein the control instruction is used to instruct the robot model to execute an action sequence; the reasoning and action model evaluates the execution result of the action sequence to determine whether the action corresponding to the action sequence is completed and / or correct.

[0112] In this embodiment, the large language model agent can continuously check whether a complete / correct answer to the question is obtained, or check whether a correct action sequence is obtained. If the answer to the question is incomplete / incorrect, or the action sequence is incorrect, it can be continuously cycled until a complete / correct answer to the question is generated, or a correct action sequence is generated. This makes the robot's response more accurate.

[0113] It should be noted that when it comes to the two response methods of answering questions and performing actions, the above two methods can be combined to achieve answering questions and performing actions at the same time, so that the robot's response results are more humanized and the interactivity is greatly enhanced.

[0114] Step 205, when the evaluation result meets the requirements, the robot is controlled to broadcast the answer to the question generated by the robot model during the preview, and / or execute the action sequence generated by the robot model during the preview.

[0115] In this embodiment, the large language model agent can accurately generate complete / correct answers to questions, or generate a sequence of correct actions, so that the robot can determine different response methods based on different speech semantics, making the response method more flexible and varied.

[0116] In the control instruction response method, when the preset trigger condition is met, the audio signal is collected; thus, when the preset trigger condition is met, the audio signal can be actively collected, avoiding long-term monitoring of the voice signal in the environment and reducing power consumption. The audio signal is semantically recognized and processed to obtain the corresponding text information; the text information is judged by the pre-established large language model agent to determine the response method, and the response method includes: executing an action and / or answering a question; thus, the response method can be determined according to the text corresponding to the voice information, making the response method more abundant and intelligent. The response method is previewed based on the constructed robot model, and the preview result of the response method is evaluated by the large language model agent to obtain the corresponding evaluation result; thus, the result of the response method can be previewed and evaluated in advance, making the control instruction finally executed by the robot more accurate. When the evaluation result meets the requirements, the robot is controlled to broadcast the answer to the question generated by the robot model during the preview, and / or execute the action sequence generated by the robot model during the preview. Thus, the response method can be adaptively determined according to the text content corresponding to the audio signal, making the response method more diversified, the feedback of the answer more intelligent, and improving the interactive experience.

[0117] In another exemplary embodiment, Figure 4 As shown, a control instruction response method is provided, which is applied to Figure 1 The robot in the example is used to illustrate, including the following steps 401 to 406. Among them:

[0118] Step 401: When a preset trigger condition is met, an audio signal is collected.

[0119] Step 402: Perform semantic recognition processing on the audio signal to obtain corresponding text information.

[0120] Step 403: The text information is judged by the pre-established large language model agent to determine the response method.

[0121] Step 404: preview the response method based on the constructed robot model, and evaluate the preview result of the response method through the large language model intelligent agent to obtain a corresponding evaluation result.

[0122] For the specific implementation process and technical effects of steps 401 to 404 in this embodiment, please refer to Figure 2 The descriptions of steps 201 to 204 in the illustrated method embodiment are not repeated here.

[0123] Step 405 , setting the answers to the questions generated by the robot model during the preview and the action sequences generated by the robot model during the preview to a plurality of timestamps.

[0124] In this embodiment, the number of timestamps to be added can be determined based on the text length of the answer to the question and the length of the action sequence. For example, the speech speed of the robot's broadcast can be determined based on experience, and the estimated broadcast duration of the answer to the question can be determined based on the speech speed of the broadcast. Similarly, the estimated duration of the robot to complete the action sequence can be determined based on experience. Then, based on the duration of the answer to the question and the duration of the action sequence, a timestamp is added, for example, a timestamp is added every 2 seconds.

[0125] Step 406, aligning the text content of the answer to the question and the action sequence through the correspondence between the timestamps, so that the robot synchronously executes the corresponding action sequence when broadcasting the answer to the question.

[0126] In this embodiment, after step 405, the answer to the question including N timestamps and the action sequence including N timestamps can be obtained, where N is a natural number greater than 1. Further, by aligning the same timestamps, the aligned answer to the question and the action sequence can be obtained.

[0127] In this embodiment, the robot can complete the synchronous execution of the answer to the question and the action sequence according to the aligned answer to the question, so that the robot can synchronously execute the corresponding action sequence in the process of answering the question.

[0128] Step 407, when the evaluation result meets the requirements, the robot is controlled to broadcast the answers to the questions generated by the robot model during the preview, and to execute the action sequence generated by the robot model during the preview.

[0129] For the specific implementation process and technical effects of step 407 in this embodiment, please refer to Figure 2 The description of step 205 in the illustrated method embodiment will not be repeated here.

[0130] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0131] Based on the same inventive concept, the embodiment of the present application also provides a control instruction response device for implementing the control instruction response method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more control instruction response device embodiments provided below can refer to the limitations of the control instruction response method above, and will not be repeated here.

[0132] In an exemplary embodiment, Figure 5 As shown, a control instruction response device is provided, including: a collection module 501, an identification module 502, a determination module 503, an evaluation module 504 and a control module 505, wherein:

[0133] The acquisition module 501 is used to acquire the audio signal when a preset trigger condition is met;

[0134] The recognition module 502 is used to perform semantic recognition processing on the audio signal to obtain corresponding text information;

[0135] The determination module 503 is used to identify the text information through a pre-established large language model agent and determine a response method, wherein the response method includes: performing an action and / or answering a question;

[0136] An evaluation module 504 is used to preview the response mode based on the constructed robot model, and evaluate the preview result of the response mode through the large language model agent to obtain a corresponding evaluation result;

[0137] The control module 505 is used to control the robot to broadcast the answer to the question generated by the robot model during the preview and / or execute the action sequence generated by the robot model during the preview when the evaluation result meets the requirements.

[0138] Exemplarily, the preset trigger condition includes at least one of the following:

[0139] An object is detected entering and / or a human face is detected within a preset detection range;

[0140] It is detected that the decibel value of the audio signal in the environment is greater than a preset value;

[0141] Detecting that the audio signal contains a target wake-up word;

[0142] The pre-established visual model collects continuous video frames containing the lip area of ​​the human face, and the image recognition results of the continuous video frames indicate the presence of a lip opening and closing movement.

[0143] Exemplarily, the recognition module 502 is specifically used to: segment the audio signal according to a preset duration using an end-to-end speech recognition model, and convert the segmented audio signal into a feature vector; decode the feature vector and output text information.

[0144] Exemplarily, the large language model agent includes: a thinking model, a reasoning and action model, wherein:

[0145] The thinking model is used to determine a response method according to the text information;

[0146] The reasoning and action model is used to generate a control instruction for the robot model according to the response mode, and evaluate the preview result of the robot model executing the control instruction to obtain a corresponding evaluation result;

[0147] The reasoning and action model is further used to regenerate control instructions for the robot model when the evaluation result does not meet the requirements, and evaluate the preview results of the robot model executing the control instructions until the evaluation result meets the requirements;

[0148] The reasoning and action model is also used to feed back response completion indication information to the thinking model when the evaluation result meets the requirements.

[0149] Exemplarily, the evaluation module 504 is specifically used to: generate a control instruction according to the text information by the reasoning and action model, and send the control instruction to the robot model, wherein the control instruction is used to instruct the robot model to broadcast the answer to the question; and evaluate the answer to the question by the reasoning and action model to determine whether the answer to the question is complete and / or correct.

[0150] Exemplarily, the evaluation module 504 is specifically used to: generate a control instruction according to the text information by the reasoning and action model, and send the control instruction to the robot model, wherein the control instruction is used to instruct the robot model to execute an action sequence; and evaluate the execution result of the action sequence by the reasoning and action model to determine whether the action corresponding to the action sequence is completed and / or correct.

[0151] In another exemplary embodiment, Figure 6 As shown, a control instruction response device is provided. Figure 5 Based on the device shown, it can also include:

[0152] A setting module 506, for setting the answers to the questions generated by the robot model during the preview and the action sequences generated by the robot model during the preview to a number of timestamps;

[0153] The alignment module 507 is used to align the text content of the answer to the question and the action sequence through the corresponding relationship between the timestamps, so that the robot can synchronously execute the corresponding action sequence when broadcasting the answer to the question.

[0154] Each module in the above control instruction response device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0155] In an exemplary embodiment, a processing system of a robot is provided, and the internal structure diagram of the processing system can be shown as follows: Figure 7 As shown. The processing system includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the processing system is used to provide computing and control capabilities. The memory of the processing system includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the processing system is used to exchange information between the processor and an external device. The communication interface of the processing system is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for generating robot actions is implemented. The display unit of the processing system is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the processing system can be a touch layer covering the display screen, or a button, trackball or touchpad set on the robot shell, or an external keyboard, touchpad or mouse.

[0156] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0157] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0158] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0159] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0160] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0161] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0162] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0163] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A control instruction response method, characterized in that: The method comprises: When the preset trigger conditions are met, the audio signal is collected; Performing semantic recognition processing on the audio signal to obtain corresponding text information; The text information is judged by a pre-established large language model agent to determine a response method, wherein the response method includes: performing an action and / or answering a question; Previewing the response method based on the constructed robot model, and evaluating the preview result of the response method through the large language model agent to obtain a corresponding evaluation result; The answers to the questions generated by the robot model during the preview and the action sequences generated by the robot model during the preview are set to a number of timestamps; wherein the number of timestamps is related to the text length of the answers to the questions and the length of the action sequences; By using the correspondence between the timestamps, the text content of the answer to the question and the action sequence are aligned, so that the robot synchronously executes the corresponding action sequence when announcing the answer to the question; When the evaluation result meets the requirements, the robot is controlled to broadcast the answer to the question generated by the robot model during the preview, and / or execute the action sequence generated by the robot model during the preview.

2. The method according to claim 1, characterized in that The preset trigger condition includes at least one of the following: An object is detected entering and / or a human face is detected within a preset detection range; It is detected that the decibel value of the audio signal in the environment is greater than a preset value; Detecting that the audio signal contains a target wake-up word; The pre-established visual model collects continuous video frames containing the lip area of ​​the human face, and the image recognition results of the continuous video frames indicate the presence of a lip opening and closing movement.

3. The method according to claim 1, characterized in that The performing semantic recognition processing on the audio signal to obtain corresponding text information includes: Using an end-to-end speech recognition model to segment the audio signal according to a preset duration, and converting the segmented audio signal into a feature vector; The feature vector is decoded and text information is output.

4. The method according to any one of claims 1 to 3, characterized in that: The large language model agent includes: a thinking model, a reasoning and action model, wherein: The thinking model is used to determine a response method according to the text information; The reasoning and action model is used to generate a control instruction for the robot model according to the response mode, and evaluate the preview result of the robot model executing the control instruction to obtain a corresponding evaluation result; The reasoning and action model is further used to regenerate control instructions for the robot model when the evaluation result does not meet the requirements, and evaluate the preview results of the robot model executing the control instructions until the evaluation result meets the requirements; The reasoning and action model is also used to feed back response completion indication information to the thinking model when the evaluation result meets the requirements.

5. The method according to claim 4, characterized in that When the response method includes answering a question, the large language model agent evaluates the preview result of the response method to obtain a corresponding evaluation result, including: The reasoning and action model generates a control instruction according to the text information, and sends the control instruction to the robot model, wherein the control instruction is used to instruct the robot model to broadcast the answer to the question; The answer to the question is evaluated by the reasoning and action model to determine whether the answer to the question is complete and / or correct.

6. The method according to claim 4, characterized in that When the response mode includes executing an action, the large language model agent evaluates the preview result of the response mode to obtain a corresponding evaluation result, including: The reasoning and action model generates a control instruction according to the text information, and sends the control instruction to the robot model, wherein the control instruction is used to instruct the robot model to execute an action sequence; The reasoning and action model evaluates the execution result of the action sequence to determine whether the action corresponding to the action sequence is completed and / or correct.

7. A control instruction response device, characterized in that: The device comprises: A collection module, used to collect audio signals when a preset trigger condition is met; A recognition module, used to perform semantic recognition processing on the audio signal to obtain corresponding text information; A determination module, used to identify the text information through a pre-established large language model agent and determine a response method, wherein the response method includes: performing an action and / or answering a question; An evaluation module, used for previewing the response mode based on the constructed robot model, and evaluating the preview result of the response mode through the large language model agent to obtain a corresponding evaluation result; A setting module, used to set the answers to the questions generated by the robot model during the preview and the action sequences generated by the robot model during the preview to a number of timestamps; wherein the number of timestamps is related to the text length of the answers to the questions and the length of the action sequences; An alignment module, used to align the text content of the answer to the question and the action sequence through the correspondence between the timestamps, so that the robot synchronously executes the corresponding action sequence when broadcasting the answer to the question; The control module is used to control the robot to broadcast the answer to the question generated by the robot model during the preview and / or execute the action sequence generated by the robot model during the preview when the evaluation result meets the requirements.

8. The device according to claim 7, characterized in that The preset trigger condition includes at least one of the following: An object is detected entering and / or a human face is detected within a preset detection range; It is detected that the decibel value of the audio signal in the environment is greater than a preset value; Detecting that the audio signal contains a target wake-up word; The pre-established visual model collects continuous video frames containing the lip area of ​​the human face, and the image recognition results of the continuous video frames indicate the presence of a lip opening and closing movement.

9. A robot comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images