Human-computer interaction method, system, electronic device and storage medium

Abstract

This invention provides a human-computer interaction method, system, electronic device, and storage medium, relating to the field of artificial intelligence technology. The method includes: inputting a user-inputted voice command and intent classification prompts into an intent recognition model to obtain the intent type of the voice command output by the intent recognition model; determining a matching database for intent types; and obtaining response data for the voice command based on the matching database to respond to the voice command. This invention, by obtaining the intent type of a voice command through intent classification prompts and an intent recognition model, can accurately uncover the deeper intent of the voice command in the current physical environment. Extracting response data from the matching database allows for the combination of matching databases for different intent types, resulting in more accurate response data and improving the efficiency of human-computer interaction.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a human-computer interaction method, system, electronic device, and storage medium. Background Technology

[0002] With the development of humanoid robot technology, its application scenarios are rapidly expanding from single, structured industrial environments to complex environments where humans coexist, such as homes and commercial services. In these dynamic and unstructured scenarios, natural, intelligent, and highly generalizable human-computer interaction capabilities have become core technological requirements. Traditional human-computer interaction systems mostly rely on preset rules, keyword matching, or simple classification models, requiring users to control the robot through precise and rigid commands.

[0003] However, this interaction method lacks naturalness and intelligence. Robots cannot understand the vague expressions or implicit intentions commonly found in human everyday language, resulting in low efficiency in human-computer interaction. Summary of the Invention

[0004] This invention provides a human-computer interaction method, system, electronic device, and storage medium to address the shortcomings of low efficiency in existing human-computer interaction technologies and improve the efficiency of human-computer interaction.

[0005] This invention provides a human-computer interaction method, comprising: Input the user's voice command and intent classification prompts into the intent recognition model to obtain the intent type of the voice command output by the intent recognition model; Determine the matching database for intent types; Response data for voice commands is obtained from a matching database to answer the voice commands.

[0006] According to the human-computer interaction method provided by the present invention, determining the matching database of intent types includes: If the intent type is determined to be question-and-answer type, then the matching database is determined to be the local knowledge base, which includes various knowledge about the physical environment in which the robot is located; Alternatively, if the intent type is determined to be the execution type, then the matching database is determined to be the robot action library, which includes all the actions that the robot can perform in the physical environment.

[0007] According to the human-computer interaction method provided by the present invention, obtaining response data of voice commands based on a matching database includes: Extract entity information of voice commands of execution type; Based on the extracted entity information, the robot retrieves each object from its environmental semantic map, obtains the retrieved objects, and extracts the environmental location data of the retrieved objects. The identifiers of the retrieved objects and / or the environmental location data of the retrieved objects are used as the matching context information for the voice commands of the execution type. Based on the voice commands of the execution type and the matching context information, response data is extracted from the robot action library.

[0008] According to the human-computer interaction method provided by the present invention, obtaining response data of voice commands based on a matching database includes: Retrieve local knowledge bases using question-and-answer type voice commands, and construct target prompts based on the retrieved knowledge information; Input the target prompts and question-and-answer type voice commands into the preset large model, and obtain the response data output by the preset large model.

[0009] According to the human-computer interaction method provided by the present invention, response data is extracted from a robot action library based on the voice command of the execution type and matching context information, including: Input the voice command that matches the context information and execution type into the preset large model, and obtain the response data output by the preset large model; The pre-set large model is based on a pre-set thought chain, matching context information, and various actions extracted from the robot action library. It breaks down the execution type of voice command into at least one ordered sub-task action and determines the response data based on each ordered sub-task action.

[0010] According to the human-computer interaction method provided by the present invention, after obtaining the response data output by a preset large model, the method further includes: The function instructions execute the actions of each ordered subtask, and update the robot's environmental position data in the environmental semantic map based on the execution results; Based on the updated environmental location data of the robot in the environmental semantic map, the execution of each subtask action is verified.

[0011] According to the human-computer interaction method provided by the present invention, the environmental semantic map is constructed based on the following method: Each object is labeled in the image data of the physical environment to obtain labeled image data; A physical environment map is constructed based on radar point cloud data of the physical environment; The labeled image data and physical environment map are time-aligned to obtain an environmental semantic map.

[0012] According to the human-computer interaction method provided by the present invention, the intent classification prompt words are constructed based on the following method: Intent classification prompts are constructed based on intent recognition keywords and prompt word templates.

[0013] The present invention also provides a human-computer interaction system, comprising: The intent recognition module is used to input the user's voice commands and intent classification prompts into the intent recognition model, and obtain the intent type of the voice commands output by the intent recognition model. The matching module is used to determine the matching database for intent types; The response module is used to retrieve response data for voice commands based on a matching database in order to respond to the voice commands.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the human-computer interaction methods described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the human-computer interaction methods described above.

[0016] This invention provides a human-computer interaction method, system, electronic device, and storage medium. It involves inputting user-input voice commands and intent classification prompts into an intent recognition model to obtain the intent type of the voice command output by the model; determining a matching database for intent types; and retrieving response data from the matching database to answer the voice command. This invention, by obtaining the intent type of a voice command through intent classification prompts and an intent recognition model, can accurately uncover the deeper intent of the voice command in the current physical environment. Extracting response data from the matching database allows for the combination of matching databases for different intent types, resulting in more accurate response data and improved efficiency in human-computer interaction. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the human-computer interaction method provided by the present invention.

[0019] Figure 2 This is one of the structural schematic diagrams of the human-computer interaction system provided by the present invention.

[0020] Figure 3 This is the second structural schematic diagram of the human-computer interaction system provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] The following is combined with Figures 1 to 4 The present invention describes a human-computer interaction method, system, electronic device, and storage medium.

[0024] Figure 1 This is a flowchart illustrating the human-computer interaction method provided by the present invention, as shown below. Figure 1 As shown, the human-computer interaction method includes steps S100 to S300, and the specific steps are as follows.

[0025] S100: Input the user's voice command and intent classification prompts into the intent recognition model to obtain the intent type of the voice command output by the intent recognition model.

[0026] The executing entity of this invention includes a robot. The robot includes a multi-information perception and fusion module, an intent recognition module based on a large model, an intelligent response module based on a preset large model and a local knowledge base, a task decomposition module based on a preset large model and a robot action library, and a feedback module.

[0027] Human-computer interaction includes voice dialogue between users and robots, as well as action interactions performed by robots based on task instructions input by users.

[0028] Voice commands are first picked up by the robot's microphone, and then the audio data transmitted from the microphone is converted into text information using an Automatic Speech Recognition (ASR) algorithm, resulting in the voice command. Voice commands include question-and-answer type voice commands and execution type voice commands. Intent types include question-and-answer type and execution type.

[0029] The intent recognition module based on a large model includes an intent recognition model trained and fine-tuned for a specific task. This model can be a pre-trained large language model used to identify the intent type of voice commands. The voice command and intent classification prompts are input into the intent recognition model. The model identifies the intent type of the voice command based on the intent classification prompts, determining whether the intent type is a question-and-answer type or an execution type.

[0030] For example, in a showroom demonstration scenario, a user might voice-input, "Please introduce your company's main business areas." The robot receives this audio data via its microphone and converts it into a text-based voice command using an ASR algorithm. The robot then inputs the voice command and intent classification prompts into the intent recognition module to determine the intent type, identifying it as a question-and-answer type.

[0031] S200: A matching database for determining intent types.

[0032] The matching database is a database that matches intent types. Matching databases for various intent types are pre-built.

[0033] After identifying the intent type, the intelligent response module or task decomposition module calls the matching database according to the intent type to obtain response data. Optionally, if there are two or more intent types, a matching database is determined for each intent type.

[0034] S300: Retrieves response data for voice commands based on a matching database to respond to voice commands.

[0035] The response data includes the answer to the voice command or the subtask action. The subtask action includes the action required to complete the subtask. When the voice command is a question-and-answer type, the response data includes the answer to that question-and-answer type voice command. The matching database includes a local knowledge base. Based on the question-and-answer type voice command, the answer to the question-and-answer type voice command is extracted from the local knowledge base to obtain the response data.

[0036] When the voice command is an execution-type voice command, the response data includes the ordered sub-task actions or function instructions of the sub-task actions for that execution-type voice command. The matching database includes a robot motion library. Based on the robot motion library, various actions that the robot can perform in the physical environment are extracted. The execution-type voice command is broken down into multiple sub-tasks. Based on each sub-task, the actions of each sub-task are extracted from the robot motion library to obtain the response data.

[0037] Optionally, when the semantic instruction includes both question-and-answer type voice instructions and execution type voice instructions, the response data for the question-and-answer type voice instructions is extracted from the local knowledge base. Simultaneously, the response data for the execution type voice instructions is extracted from the robot motion library.

[0038] Optionally, when a semantic instruction includes multiple question-and-answer type voice instructions, response data for each question-and-answer type voice instruction is extracted from the local knowledge base based on the voice instructions for each question-and-answer type.

[0039] Optionally, when a semantic instruction includes multiple voice instructions of different execution types, response data for each execution type of voice instruction is extracted from the robot's motion library.

[0040] Furthermore, during the execution of tasks, the robot interacts with the user in a voice or non-voice manner to report on the execution status of tasks corresponding to voice commands.

[0041] The human-computer interaction method provided in this invention involves inputting user-inputted voice commands and intent classification prompts into an intent recognition model to obtain the intent type of the voice command output by the intent recognition model; determining a matching database for intent types; and obtaining response data for the voice command based on the matching database to respond to the voice command. This invention, by obtaining the intent type of the voice command through intent classification prompts and an intent recognition model, can accurately uncover the deeper intent of the voice command in the current physical environment. Extracting response data based on the matching database allows for the combination of matching databases for different intent types to extract more accurate response data, thus improving the efficiency of human-computer interaction.

[0042] Based on the above embodiments, the intent classification prompts are constructed in the following manner: Intent classification prompts are constructed based on intent recognition keywords and prompt word templates.

[0043] Intent recognition keywords include keywords that can determine the intent type of a voice command. Intent recognition keywords include execution-type keywords and question-and-answer-type keywords. Execution-type keywords are used to identify execution-type voice commands. Question-and-answer-type keywords are used to identify question-and-answer-type voice commands. Optionally, execution-type keywords include action keywords and / or specific parameters of the action. Question-and-answer-type keywords include common question-and-answer types.

[0044] The prompt word template can be set according to the steps of identifying the intent type.

[0045] Optionally, the intent classification prompts are as follows.

[0046] You are an intent classifier. Please analyze the user's voice input to determine whether it is a robot control command or a general question; If it is a robot control command, return "Execution Type". If it is a general question, return "Question-Answer Type". Robot control commands typically include the following keywords (keywords for execution type): move, forward, backward, turn, stop, face, move to, grasp, release, take a picture, speak, and other action keywords, as well as specific parameters such as distance and angle; General questions include, but are not limited to: knowledge quizzes, information retrieval, explanation of concepts, and providing suggestions in chat conversations; When answering, please only return "Execution Type" or "Question and Answer Type", and do not include other text.

[0047] This invention constructs intent classification prompts, which can accurately guide the intent recognition model to identify intent types based on intent recognition keywords and prompt templates.

[0048] Based on the above embodiments, determining the matching database for intent types includes the following steps: If the intent type is determined to be question-and-answer type, then the matching database is determined to be the local knowledge base, which includes various knowledge about the physical environment in which the robot is located; Alternatively, if the intent type is determined to be the execution type, then the matching database is determined to be the robot action library, which includes all the actions that the robot can perform in the physical environment.

[0049] A mapping relationship between intent types and matching databases is pre-built. When the intent type is question-and-answer, the matching database is the local knowledge base. When the intent type is execution, the matching database is the robot action library.

[0050] The local knowledge base includes attribute data for each object in a custom environmental semantic map. For example, the robot's physical environment might be an exhibition hall. As a guide, the robot needs to introduce the exhibits to users. The local knowledge base would then include attribute data (introductory materials) for each exhibit (object) in the hall. This attribute data includes the exhibit's name, function, specifications, development history, and application areas. Furthermore, the local knowledge base can also include introductory materials for the company owning the exhibition hall, including its profile, development history, business areas, and core values. The introductory materials for each object in the environmental semantic map included in the local knowledge base can be in Word, PDF, or TXT formats.

[0051] The attribute data of each object in the environmental semantic map, the introduction materials of the enterprise, etc. (various knowledge of the physical environment in which the robot is located) are converted into word vectors, and a local knowledge base is built based on all the word vectors.

[0052] Furthermore, the environmental semantic map includes the identifiers of various objects in the physical environment and their environmental location data. For example, the physical environment in which the robot is located is an exhibition hall scene. The environmental semantic map includes the names of exhibit 1, exhibit 2, and exhibit 3, as well as the environmental location data of exhibit 1, exhibit 2, and exhibit 3. The local knowledge base includes detailed attribute data for exhibit 1, exhibit 2, and exhibit 3.

[0053] The robot motion library includes all the actions a robot can perform in the current physical environment. Optionally, the robot motion library includes navigation actions, grasping actions, placement actions, and transport actions. Furthermore, the robot motion library also includes functions for each action. Specifically, the robot motion library includes all the actions that a robot can perform, as detailed below.

[0054] move_to_object(object: object_name): Moves to the specified object; move_forward(meters: number): Moves the device forward by the specified number of meters; turn_left_angle(angle: number): Turns the angle to the left by a specified angle (e.g., 90°); turn_right_angle(angle: number): Turn right by a specified angle (e.g., 90°); face(object: object name): Faces the specified object; shake_head(): Head shaking action (no parameters); nod_head(): Nod action (no parameters).

[0055] Furthermore, the robot motion library also includes functions for various actions that the robot can perform. For example, the function to move forward 2 meters is {"type":"command","function":"move_forward","parameters":{"meters":2}}. The function to nod is {"type":"command","function":"shake_head","parameters":{}}.

[0056] This invention pre-stores various knowledge about the robot's physical environment in a local knowledge base. By searching the local knowledge base, answers to question-and-answer type voice commands can be quickly obtained, improving the efficiency of human-computer interaction. Furthermore, by pre-storeing various actions the robot can perform in the physical environment in a robot action library, searching the robot action library allows for the rapid identification and execution of various sub-task actions associated with different types of voice commands, further improving the efficiency of human-computer interaction.

[0057] Based on the above embodiments, the environmental semantic map is constructed in the following manner: Each object is labeled in the image data of the physical environment to obtain labeled image data; A physical environment map is constructed based on radar point cloud data of the physical environment; The labeled image data and physical environment map are time-aligned to obtain an environmental semantic map.

[0058] The robot's multi-information perception and fusion module includes various sensors, such as LiDAR, Inertial Measurement Unit (IMU), and 3D binocular camera.

[0059] The robot's physical environment is photographed using a 3D binocular camera, generating image data. Object recognition algorithms then identify the objects within these images. Each object is labeled within the physical environment image data, resulting in labeled image data. Optionally, the names (object identifiers) of each object are labeled within the image data, such as walls, pillars, and tables.

[0060] LiDAR is used to acquire radar point cloud data. A physical environment map is then constructed based on the radar point cloud data. The physical environment map is used to record and display the positions of various objects in the physical environment and the relative positions between objects.

[0061] The labeled image data and environmental semantic map are subjected to time stamp alignment, noise reduction, and other processing to obtain the environmental semantic map.

[0062] Because the physical environment in which the robot operates may be dynamic and objects in the physical environment, such as obstacles, may move, the information about the physical environment and obstacles will be aligned over time.

[0063] This invention constructs an environmental semantic map based on labeled image data and a physical environment map, enabling the environmental semantic map to intuitively display various objects in the physical environment and their environmental location data.

[0064] Furthermore, the robot's real-time position is added to the environmental semantic map. The robot's inertial measurement unit (IMU) measures its own motion posture and motion changes in real time. Based on this data, the robot's position in the environmental semantic map is updated in real time.

[0065] Based on the above embodiments, obtaining response data for voice commands based on a matching database includes the following steps: Extract entity information of voice commands of execution type; Based on the extracted entity information, the robot retrieves each object from its environmental semantic map, obtains the retrieved objects, and extracts the environmental location data of the retrieved objects. The identifiers of the retrieved objects and / or the environmental location data of the retrieved objects are used as the matching context information for the voice commands of the execution type. Based on the voice commands of the execution type and the matching context information, response data is extracted from the robot action library.

[0066] A robot's environmental semantic map consists of structured map data of the robot's physical environment that contains semantic information. The environmental semantic map can be a geometric or topological map containing object labels.

[0067] Matching contextual information includes environmental semantic information that matches the speech commands of the execution type in the environmental semantic map.

[0068] When the robot moves to a new physical environment or the physical environment changes, the environmental semantic map is updated in real time to accurately extract the matching context information of the voice commands of the execution type.

[0069] Existing human-computer interaction methods typically execute actions based on preset rules for specific hardware and scenarios. Once the physical environment changes, the performance of the human-computer interaction drops sharply, lacking cross-scenario generalization. This invention introduces an environmental semantic map, which is updated in real time when the robot's physical environment changes, thus ensuring the performance of the human-computer interaction.

[0070] The retrieved objects include those mentioned or implied by the executed voice command. For example, if the executed voice command is "Please pick up the water glass on the table," the retrieved objects include the table and the water glass.

[0071] Extracting matching context information for execution-type voice commands from the robot's environmental semantic map can specifically include the following steps.

[0072] First, feature extraction is performed on the execution type of voice commands to obtain the object names (or object identifiers), company names (including "your company," "showroom company," etc.), or other feature information mentioned in the semantic commands. For example, the execution type of voice command is "Please pick up the water glass on the table." The extracted object names include "table" and "water glass."

[0073] Then, based on the extracted object identifiers, company names, or other feature information, the identifiers of each object in the environmental semantic map are retrieved, and the identifiers of the retrieved objects are obtained. For example, the object identifiers extracted from the voice command include "table" and "water cup." Based on "table" and "water cup," the identifiers of each object in the environmental semantic map are retrieved, and the table and water cup are found in the environmental semantic map. The table and water cup are then used as the identifiers of the retrieved objects.

[0074] The environmental location data of retrieved objects is extracted from the environmental semantic map. This data includes the object's distance relative to the robot. Furthermore, when two or more objects are retrieved, the environmental location data also includes the relative positions between them. For example, if the retrieved objects include a table and a water glass, the environmental location data would include the distance and direction of the table from the robot, the distance and direction of the water glass from the robot, and the relative positions of the table and the water glass (e.g., the water glass is on the left, right, or center of the table).

[0075] Finally, the identifiers of the retrieved objects and / or the environmental location data of the retrieved objects are used as matching context information.

[0076] Optionally, the identifiers of each object are retrieved according to the execution instruction, and the identifiers of the retrieved objects are obtained. The environmental location data of the retrieved objects is extracted from the environmental semantic map. The identifiers of the retrieved objects and the environmental location data of the retrieved objects are used as matching context information. For example, the semantic instruction of the execution type is "Please pick up the water glass on the table". The identifiers of the retrieved objects include "table" and "water glass". The environmental location data of the retrieved objects includes "the table is 20 meters away from the robot, the table is located to the upper left of the robot, and the water glass is in the center of the table". Then the matching context information is "table and water glass, the table is 20 meters away from the robot, the table is located to the upper left of the robot, and the water glass is in the center of the table".

[0077] Optionally, objects in the environmental semantic map are retrieved based on the voice command of the execution type. The environmental location data of the retrieved objects is then extracted from the environmental semantic map. This environmental location data is used as matching context information.

[0078] Optionally, when the voice command is an execution-type voice command, the matching context information can serve as supplementary explanations for the execution parameters of each subtask action of the execution-type voice command. For example, the execution-type voice command is "Please pick up the water cup on the table." In this case, the execution-type voice command does not specify precisely how far to walk to the table or how much force to apply when picking up the water cup. In this case, the matching context information is "The table is 20 meters away from the robot, and the water cup is a disposable paper cup." This indicates that the robot needs to walk 20 meters to reach the table. The force applied when picking up the water cup should be relatively low because the paper cup is not secure.

[0079] This invention determines matching context information based on the retrieved object identifier and / or the retrieved object's environmental location data. This allows for further mining of the deeper environmental information of the executed voice command in the current physical environment, which is beneficial for improving the accuracy and comprehensiveness of subsequent response data determination.

[0080] Based on the above embodiments, obtaining response data for voice commands based on a matching database includes the following steps: Retrieve local knowledge bases using question-and-answer type voice commands, and construct target prompts based on the retrieved knowledge information; Input the target prompts and question-and-answer type voice commands into the preset large model, and obtain the response data output by the preset large model.

[0081] When the voice command is a question-and-answer type, the matching database is the local knowledge base. The preset large model is a pre-trained large model for the robot's intelligent response. Response data for the voice command is generated using the preset large model. The preset large model includes a pure language large model.

[0082] The system retrieves local knowledge bases based on question-and-answer type voice commands and constructs target prompts based on the retrieved knowledge information. For example, if the question-and-answer type voice command is "Please introduce the signal amplification device of model A", the retrieved knowledge information includes detailed information about the signal amplification device of model A, such as its functions, development history, application areas, usage methods, or videos.

[0083] The response data includes answers to question-and-answer type voice commands. The pre-built large model possesses natural language understanding and reasoning capabilities. Target prompts are constructed based on retrieved knowledge information. The target prompts and question-and-answer type voice commands are input into the pre-built large model. The target prompts guide the model to perform logical reasoning or organize the retrieved knowledge information based on the question-and-answer type voice command to obtain the answer.

[0084] After receiving the answer to a question-and-answer type voice command, the answer (response data) is converted into voice information using a text-to-speech (TTS) algorithm, and then the voice information is played through the robot's speakers.

[0085] This application retrieves local knowledge bases based on question-and-answer type voice commands, enabling comprehensive acquisition of various knowledge information associated with the question-and-answer type voice commands, which helps improve the comprehensiveness and accuracy of the answers (response data) of the subsequently generated question-and-answer type voice commands.

[0086] Furthermore, during the process of outputting answers to question-and-answer type voice commands, the robot collects the user's facial expressions and voice feedback in real time to verify the accuracy of the answers. If an inaccurate answer is detected, the robot re-identifies the intent type and matches it against the database. Response data for the voice command is then retrieved from the database to respond. If the response fails again, the voice command is recorded. Alternatively, the voice command is recorded when the number of failed responses reaches a threshold.

[0087] If the answer to a question-and-answer type voice command is determined to be accurate, the voice command and its answer are recorded. Subsequent users submitting the same question-and-answer type voice command will retrieve the answer directly from the historical record.

[0088] Based on the above embodiments, the response data is extracted from the robot action library based on the voice command of the execution type and the matching context information, including the following steps: Input the voice command that matches the context information and execution type into the preset large model, and obtain the response data output by the preset large model; The pre-set large model is based on a pre-set thought chain, matching context information, and various actions extracted from the robot action library. It breaks down the execution type of voice command into at least one ordered sub-task action and determines the response data based on each ordered sub-task action.

[0089] When the voice command is an execution-type voice command, the matching database is the robot motion library. In this case, the matching context information includes the identifier of the retrieved object and / or the environmental position data of the retrieved object. The preset large model also includes a multimodal large model.

[0090] Voice commands that match context information and execution type are input into a pre-defined large model. Simultaneously, the pre-defined large model can query and call the robot's motion library and extract various actions that the robot can perform in the physical environment. For example, the pre-defined large model can query the name of any action in the robot's motion library and the function of any action.

[0091] The pre-defined large model includes pre-defined thought chains and prompt templates. Using these templates, prompts are constructed based on the pre-defined thought chains. The prompts guide the pre-defined large model to break down execution-type voice commands into multiple sub-tasks according to a specified thought process. During the process of breaking down execution-type voice commands, the pre-defined large model can consult a reference robot action library and match contextual information.

[0092] Specifically, the prompts based on the pre-set thought chain are as follows.

[0093] You are a domain expert specializing in large language model thought chain reasoning and robot task decomposition, possessing precise semantic parsing, logical decomposition, and instruction standardization capabilities. Please strictly follow the following rules to perform thought chain-based step-by-step decomposition of user-inputted execution-type voice commands: 1. First, analyze the core actions of the execution type of voice command through a preset thought chain: identify the execution type of voice command and match key behavioral subjects and target objects such as displacement actions and operation actions in the context information; 2. Remove redundant semantics and retain only the core steps that the robot can execute; the robot's executable steps are referenced from the robot motion library; 3. Standardize the output steps according to the order of action execution, and maintain a consistent format: Step 1: Robot + Core Displacement / Pre-Motion Action; Step 2: Robot + Core Operations / Subsequent Actions; 4. The number of steps strictly matches the independent executable actions within the instruction, and the language is concise, unambiguous, and conforms to the robot's execution logic.

[0094] The pre-defined large model queries and matches contextual information and robot action library based on the prompts of the pre-defined thought chain to break down the execution type of voice commands into at least one ordered sub-task.

[0095] For example, the voice command for execution type is "Please pick up the water glass on the table". The resulting subtask actions, after being broken down, are at least one ordered sequence as follows: Step 1: The robot moves 300 meters to the left to the table; Step 2: The robot picks up the water glass from the center of the table.

[0096] Furthermore, the pre-defined large model queries the robot motion library to obtain the actions for each ordered subtask, based on at least one ordered subtask derived from the decomposition. The subtask actions include the actions required to complete the subtask. Response data is output based on the ordered subtask actions.

[0097] Furthermore, the response data is sent to the robot's end effector to execute the various sub-task actions. The robot's end effector queries the robot's motion library based on the ordered sub-task actions, retrieves the functions for each ordered sub-task action, generates the function instructions for each ordered sub-task action, and then executes each sub-task action.

[0098] This invention combines a preset thought chain, matching context information, and various actions extracted from the robot action library to break down execution-type voice commands into at least one ordered sub-task action. This ensures that the decomposition process is logically reasonable, that the decomposed sub-task actions are all executable by the robot, and that the decomposed sub-task actions also conform to the matching context information.

[0099] Based on the above embodiments, after obtaining the response data output by the preset large model, the following steps are also included: The function instructions execute the actions of each ordered subtask, and update the robot's environmental position data in the environmental semantic map based on the execution results; Based on the updated environmental location data of the robot in the environmental semantic map, the execution of each subtask action is verified.

[0100] The environmental semantic map also includes the robot's real-time environmental position data. The robot's end effector executes function instructions for each ordered subtask action. The robot's environmental position data in the environmental semantic map is updated based on the execution results.

[0101] Based on the updated environmental location data of the robot in the semantic map, the execution of each subtask action is verified. For example, if the updated environmental location data of the robot in the semantic map shows that the robot is next to the table, then the subtask action "the robot moves 300 meters to the left to the table" is verified as completed. Similarly, if the updated environmental location data shows that the robot is holding a water cup, then the subtask action "the robot picks up the water cup from the center of the table" is verified as completed.

[0102] Furthermore, the robot's pose and environmental position data in the semantic map are updated based on the execution results. The execution of each subtask action is then verified using the updated robot position and pose data in the semantic map.

[0103] Furthermore, the execution status of each subtask action is verified. If each subtask action is executed successfully, the voice command for that execution type and its ordered subtask actions are recorded, and the validity period of the voice command for that execution type is set. Within the validity period, if the user issues the same voice command for that execution type, the ordered subtask actions corresponding to that voice command are retrieved from the historical records, response data is output, and the commands are executed.

[0104] The human-computer interaction method of this invention can not only understand the user's display commands, but also identify the user's implicit intentions and real needs based on matching context information, achieving a more human-like interaction. This invention uses joint reasoning of environmental semantic maps and execution-type voice commands to eliminate information gaps between modules, which helps improve the accuracy and response speed of human-computer interaction decisions.

[0105] Optionally, the intent recognition model and the preset large model of the present invention can be merged into a single super-large model. The execution type of voice command and the environmental semantic map are input into this super-large model. Based on the execution type of voice command, the environmental semantic map, and the preset thought chain, the super-large model simultaneously queries the robot action library and outputs at least one ordered sub-task action.

[0106] The human-computer interaction system provided by the present invention is described below. The human-computer interaction system described below can be referred to in correspondence with the human-computer interaction method described above.

[0107] like Figure 2 As shown, a human-computer interaction system includes: The intent recognition module 201 is used to input the user's voice command and intent classification prompt words into the intent recognition model, and obtain the intent type of the voice command output by the intent recognition model; Matching module 202 is used to determine the matching database of intent type; The response module 203 is used to obtain response data for the voice command based on the matching database in order to respond to the voice command.

[0108] The human-computer interaction system can be a robot; this invention uses a robot as an example for illustration. Further, such as... Figure 3 As shown, the robot also includes a multi-information perception and fusion module. This module comprises a 3D binocular camera, a LiDAR sensor, and a microphone array. The 3D binocular camera is used to photograph the robot's physical environment and acquire image data. The LiDAR sensor is used to acquire radar point cloud data of the physical environment. The microphone array is used to acquire voice commands.

[0109] The multi-information perception and fusion module is also used to annotate various objects in image data to obtain annotated image data; construct a physical environment map based on radar point cloud data; and perform time alignment between the annotated image data and the physical environment map to obtain an environmental semantic map. The environmental semantic map and voice commands are then sent to the intent recognition module.

[0110] The intent recognition module identifies the intent type of the voice command based on intent classification prompts and the intent recognition model, and determines the matching database for that intent type. If the intent recognition module identifies the intent type as a question-and-answer type, it determines the matching database as the local knowledge base. If the intent recognition module identifies the intent type as an execution type, it determines the matching database as the robot action library.

[0111] The response module includes an intelligent answering module based on a preset large model and a local knowledge base, and a task decomposition module based on a preset large model and a robot action library.

[0112] Optionally, if the intent recognition module identifies the intent type as question-and-answer type, it sends the intent type and voice command to the intelligent response module. The intelligent response module searches the local knowledge base based on the question-and-answer type voice command, constructs a target prompt based on the retrieved knowledge information, and inputs the target prompt and the question-and-answer type voice command into a preset large model to obtain the response data output by the preset large model.

[0113] Optionally, if the intent recognition module identifies the intent type as execution type, it extracts the entity information of the voice command of execution type; based on the extracted entity information, it retrieves various objects in the robot's environmental semantic map, obtains the retrieved objects, and extracts the environmental location data of the retrieved objects; it uses the identifier of the retrieved objects and / or the environmental location data of the retrieved objects as the matching context information of the voice command. The matching context information and the voice command of execution type are sent to the task decomposition module. The task decomposition module inputs the matching context information and the voice command of execution type into a preset large model. The preset large model, based on a preset thought chain, the matching context information, and various actions extracted from the robot's action library, decomposes the voice command of execution type into at least one ordered sub-task action, and determines the response data based on each ordered sub-task action.

[0114] Furthermore, such as Figure 3 As shown, the robot also includes a feedback module. This module can provide both voice-based and non-voice-based feedback. During task execution, the robot interacts with the user via voice, such as replying, "I have finished answering." Alternatively, the robot can interact with the user non-voice-based, such as waving goodbye when the user says goodbye.

[0115] The human-computer interaction system provided in this invention involves inputting user-inputted voice commands and intent classification prompts into an intent recognition model to obtain the intent type of the voice command output by the intent recognition model; determining a matching database for intent types; and obtaining response data for the voice command based on the matching database to respond to the voice command. This invention, by obtaining the intent type of the voice command through intent classification prompts and an intent recognition model, can accurately uncover the deeper intent of the voice command in the current physical environment. Extracting response data from the matching database allows for the combination of matching databases for different intent types to extract more accurate response data, thus improving the efficiency of human-computer interaction.

[0116] The human-computer interaction system (robot) provided by this invention can quickly adapt to new environments. For example, when the robot is placed in a new physical environment, it can process new voice commands by updating the environmental semantic map in real time.

[0117] Figure 4An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a human-computer interaction method, which includes: inputting a user-inputted voice command and intent classification prompts into an intent recognition model; obtaining the intent type of the voice command output by the intent recognition model; determining a matching database for the intent type; and obtaining response data for the voice command based on the matching database to respond to the voice command.

[0118] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0119] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the human-computer interaction method provided by the above methods, the method comprising: inputting a user-inputted voice command and intent classification prompt words into an intent recognition model, obtaining the intent type of the voice command output by the intent recognition model; determining a matching database of intent types; and obtaining response data of the voice command based on the matching database to respond to the voice command.

[0120] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A human-computer interaction method, characterized in that, include: Input the user's voice command and intent classification prompts into the intent recognition model, and obtain the intent type of the voice command output by the intent recognition model; Determine the matching database for the intent type; The response data for the voice command is obtained based on the matching database to respond to the voice command.

2. The human-computer interaction method according to claim 1, characterized in that, The matching database for determining the intent type includes: If the intent type is determined to be a question-and-answer type, then the matching database is determined to be a local knowledge base, which includes various knowledge about the physical environment in which the robot is located; Alternatively, if the intent type is determined to be an execution type, then the matching database is determined to be a robot action library, which includes various actions that the robot can perform in the physical environment.

3. The human-computer interaction method according to claim 2, characterized in that, The step of obtaining the response data of the voice command based on the matching database includes: Extract entity information of voice commands of execution type; Based on the extracted entity information, the robot retrieves each object from its environmental semantic map, obtains the retrieved objects, and extracts the environmental location data of the retrieved objects. The identifier of the retrieved object and / or the environmental location data of the retrieved object are used as the matching context information for the voice command of the execution type; Based on the voice command of the execution type and the matching context information, the response data is extracted from the robot action library.

4. The human-computer interaction method according to claim 2, characterized in that, The step of obtaining the response data of the voice command based on the matching database includes: The local knowledge base is retrieved based on question-and-answer type voice commands, and target prompts are constructed based on the retrieved knowledge information; The target prompt and the question-and-answer type voice command are input into a preset large model, and the response data output by the preset large model is obtained.

5. The human-computer interaction method according to claim 3, characterized in that, The extraction of response data from the robot action library based on the voice command of the execution type and the matching context information includes: Input the matching context information and the voice command of the execution type into a preset large model, and obtain the response data output by the preset large model; The preset large model, based on a preset thought chain, the matching context information, and various actions extracted from the robot action library, breaks down the voice command of the execution type into at least one ordered sub-task action, and determines the response data based on each ordered sub-task action.

6. The human-computer interaction method according to claim 5, characterized in that, After obtaining the response data output by the preset large model, the method further includes: The function instructions execute the actions of each ordered subtask, and update the robot's environmental position data in the environmental semantic map based on the execution results; Based on the updated environmental location data of the robot in the environmental semantic map, the execution of each subtask action is verified.

7. The human-computer interaction method according to claim 3, characterized in that, The environmental semantic map is constructed based on the following method: Each object is labeled in the image data of the physical environment to obtain labeled image data; A physical environment map is constructed based on radar point cloud data of the physical environment; The labeled image data and the physical environment map are time-aligned to obtain the environmental semantic map.

8. The human-computer interaction method according to claim 1, characterized in that, The intent classification prompts are constructed based on the following method: The intent classification prompts are constructed based on intent recognition keywords and prompt templates.

9. A human-computer interaction system, characterized in that, include: The intent recognition module is used to input the user's voice command and intent classification prompt words into the intent recognition model, and obtain the intent type of the voice command output by the intent recognition model; A matching module is used to determine the matching database for the intent type; A response module is used to obtain response data for the voice command based on the matching database, in order to respond to the voice command.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the human-computer interaction method as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the human-computer interaction method as described in any one of claims 1 to 8.

Images