Human-computer interaction method and human-computer interaction system based on large model assistance

Abstract

The present application provides a human-computer interaction method and human-computer interaction system based on large model assistance, which relates to the technical field of human-computer interaction. The human-computer interaction method includes: constructing a target speech recognition model based on an initial instruction set; recognizing user voice instructions from the user based on the target speech recognition model to determine user text instructions based on the user voice instructions; based on the user text instructions, performing intent fusion recognition on the user text instructions through a large language model and a preset small language model to obtain the user instruction intention; performing task decomposition and planning on the user instruction intention to obtain a high-level task sequence; generating machine control instructions based on the high-level task sequence to perform human-computer interaction according to the machine control instructions. The present application can improve the accuracy of user instruction analysis and can improve the accuracy of human-computer interaction.

Description

Technical Field

[0001] The present application relates to the technical field of human-computer interaction, and in particular to a human-computer interaction method and a human-computer interaction system based on large model assistance. Background Art

[0002] Human-computer interaction (HCI) refers to the technology that enables efficient interaction between humans and computers through computer input or output devices. Traditional HCI testing (also known as human-computer hybrid testing) relies primarily on real-time manual operation by operators. This method is not only redundant and cumbersome, but also requires high operator skills and experience.

[0003] At present, human-computer interaction technology can obtain natural language text instructions issued by users through voice wake-up, collection and precise voice recognition technology, and then use natural language understanding technology to convert these instructions into machine-executable action control signals, thereby realizing human-computer interaction.

[0004] However, the traditional human-computer interaction process relies on complex operating interfaces and tedious command input, which not only increases the difficulty of user operation, but also limits the flexibility and accuracy of human-computer interaction, resulting in reduced testing efficiency of human-computer interaction.

[0005] Large-scale model-assisted human-computer interaction technology has attracted much attention due to its powerful reasoning capabilities and natural language understanding technology. Although large-scale model-assisted human-computer hybrid experiment technology has significant advantages in theory, a series of key technical challenges still need to be overcome in practical applications.

[0006] The contents of the background technology section are merely technologies known to the public and do not necessarily represent the existing technologies in this field. Summary of the Invention

[0007] The present application provides a large model-assisted human-computer interaction method and a human-computer interaction system to solve the problem of low human-computer interaction efficiency.

[0008] According to one aspect of the present application, the present application provides a human-computer interaction method based on the assistance of a large model, including: constructing a target speech recognition model based on an initial instruction set; recognizing user voice instructions from the user based on the target speech recognition model to determine user text instructions based on the user voice instructions; based on the user text instructions, performing intent fusion recognition on the user text instructions through a language large model and a preset language small model to obtain the user instruction intention; performing task decomposition and planning on the user instruction intention to obtain a high-level task sequence; generating machine control instructions based on the high-level task sequence to perform human-computer interaction according to the machine control instructions.

[0009] According to some embodiments of the present application, constructing a target speech recognition model based on an initial instruction set includes: receiving an interaction instruction set from the field of human-computer interaction to construct an initial instruction set; based on a large language model, performing semantic expansion processing on the initial instruction set to generate a synonymous instruction set; performing speech synthesis recording on the initial instruction set and the synonymous instruction set to obtain a training speech instruction data set; based on the training speech instruction data set, fine-tuning the preset speech recognition large model through a preset model fine-tuning technology to obtain a target speech recognition model.

[0010] According to some embodiments of the present application, based on user text instructions, the user text instructions are subjected to intent fusion recognition through a large language model and a preset language small model to obtain the user's user instruction intention, including: constructing a preset language small model based on a sample set containing a specific task; based on the user text instructions, obtaining first intent information according to the large language model; based on the user text instructions, obtaining second intent information according to the preset language small model; performing intent fusion recognition on the first intent information and the second intent information to obtain the user instruction intention.

[0011] According to some embodiments of the present application, constructing a preset language mini-model based on a sample set containing a specific task includes: constructing a sample set containing a specific task; and training a machine learning text detection mini-model based on the sample set to obtain a preset language mini-model.

[0012] According to some embodiments of the present application, obtaining the first intent information based on the user text instruction according to the language macro model includes: constructing a prompt template based on the user text instruction and preset interaction task information; generating guiding prompt words based on the prompt template; inputting the guiding prompt words and the user text instruction into the language macro model for semantic understanding to obtain the key elements in the user text instruction; and performing semantic alignment and intent recognition on the guiding prompt words and the key elements to obtain the first intent information.

[0013] According to some embodiments of the present application, obtaining the second intent information based on the user text instruction according to the preset language mini-model includes: processing the user text instruction to obtain intent data information; processing the intent data information based on the preset language mini-model to obtain the second intent information.

[0014] According to some embodiments of the present application, performing intent fusion identification on the first intent information and the second intent information to obtain the user command intention includes: determining whether the first intent information and the second intent information are consistent, and if so, outputting the user command intention; if not, based on the first intent information and the second intent information, supplementing and correcting the user intent through a multi-source information integration mechanism to obtain the user command intention.

[0015] According to some embodiments of the present application, task decomposition and planning are performed on the user instruction intention to obtain a high-level task sequence, including: decomposing the user instruction intention to obtain at least one candidate action information; predicting the relevant probability of the candidate action information for implementing high-level instructions through a large language model to obtain a predicted probability; determining candidate subtasks with guiding significance based on the predicted probability; estimating the probability of the execution success rate of the candidate subtasks based on the current interactive environment status data and historical data to obtain a feasibility score of the candidate subtask under current conditions; determining subtask planning information based on the predicted probability and the feasibility score; and determining a high-level task sequence based on the subtask planning information.

[0016] According to some embodiments of the present application, generating machine control instructions according to a high-level task sequence to perform human-computer interaction according to the machine control instructions includes: determining subtask execution parameters based on subtask planning information; establishing a mapping relationship between an application programming interface and subtask execution parameters to obtain machine control instructions; performing human-computer interaction based on the machine control instructions; the subtask execution parameters may at least include target parameters, input parameters, output parameters and execution constraints.

[0017] According to another aspect of the present application, the present application provides a human-computer interaction system assisted by a large model, including a speech recognition module, an intention understanding module, a task planning module and an instruction execution module. The speech recognition module constructs a target speech recognition model based on the initial instruction set, and recognizes user voice instructions from the user based on the target speech recognition model to determine user text instructions based on the user voice instructions; the intention understanding module performs intent fusion recognition on the user text instructions based on the user text instructions through the language large model and the preset language small model to obtain the user instruction intention; the task planning module performs task decomposition and planning on the user instruction intention to obtain a high-level task sequence; the instruction execution module generates machine control instructions according to the high-level task sequence to perform human-computer interaction according to the machine control instructions.

[0018] According to another aspect of the present application, an electronic device is provided. The electronic device includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the human-computer interaction method described above.

[0019] According to another aspect of the present application, a non-volatile computer-readable storage medium is provided, wherein a computer program is stored on the storage medium, and when the computer program is executed by a processor, the human-computer interaction method described above can be implemented.

[0020] According to another aspect of the present application, a computer program product is provided. The computer program product includes: a computer program stored on a computer-readable storage medium; the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the human-computer interaction method described above.

[0021] Beneficial technical effects:

[0022] This application constructs a target speech recognition model through an initial instruction set, so that the target speech recognition model can simulate various simulated speech scenarios in human-computer interaction experiments, so that this application can more accurately identify user voice instructions to determine user text instructions. This application uses a large language model and a preset small language model to perform intention fusion recognition on user text instructions, which can achieve dynamic adaptation of general semantic understanding and vertical field knowledge, so that this application can accurately obtain the user instruction intention. Afterwards, this application performs task decomposition and planning on the user instruction intention, determines the execution steps with the highest execution rate, and thus combines them to obtain a high-level task sequence. Finally, this application generates machine control instructions through high-level task sequences and performs human-computer interaction, so that this application can accurately perform human-computer interaction operations, thereby improving the efficiency of human-computer interaction testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1 A schematic diagram showing a flow chart of a human-computer interaction method according to an embodiment of the present application;

[0025] Figure 2 Another schematic diagram showing a flow chart of the human-computer interaction method according to an embodiment of the present application;

[0026] Figure 3 Another schematic diagram showing a flow chart of the human-computer interaction method according to an embodiment of the present application;

[0027] Figure 4 Another schematic diagram showing a flow chart of the human-computer interaction method according to an embodiment of the present application;

[0028] Figure 5 Another schematic diagram showing a flow chart of the human-computer interaction method according to an embodiment of the present application;

[0029] Figure 6 Another schematic diagram showing a flow chart of the human-computer interaction method according to an embodiment of the present application;

[0030] Figure 7 Another schematic diagram showing a flow chart of the human-computer interaction method according to an embodiment of the present application;

[0031] Figure 8 Another schematic diagram showing a flow chart of the human-computer interaction method according to an embodiment of the present application;

[0032] Figure 9 Another schematic diagram showing a flow chart of the human-computer interaction method according to an embodiment of the present application;

[0033] Figure 10 A schematic structural diagram of a human-computer interaction system according to an embodiment of the present application is shown.

[0034] Description of reference numerals:

[0035] 10. Human-computer interaction system;

[0036] 11. Speech recognition module; 12. Intention understanding module; 13. Task planning module; 14. Command execution module. DETAILED DESCRIPTION

[0037] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. Like reference numerals in the drawings represent like or similar parts, and thus repetitive description thereof will be omitted.

[0038] The described features, structures or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of these specific details, or other methods, components, materials, devices, etc. may be employed. In these cases, well-known structures, methods, devices, implementations, materials or operations will not be shown or described in detail.

[0039] Furthermore, the terms "include," "comprise," and "have," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0040] The terms "first", "second" and the like in the specification, claims and drawings of this application are used to distinguish different objects rather than to describe a specific order.

[0041] The following is a clear and complete description of the technical solution of this application in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making any creative efforts are within the scope of protection of this application.

[0042] According to one aspect of the present application, the present application provides a human-computer interaction method based on large model assistance, Figure 1 A flow chart of the human-computer interaction method according to an embodiment of the present application is shown. Figure 1 As shown, the human-computer interaction method may include steps S100-S500.

[0043] Exemplarily, the human-computer interaction method may be executed by a human-computer interaction system having computing capabilities.

[0044] According to an example embodiment, in step S100 , the human-computer interaction system constructs a target speech recognition model based on an initial instruction set.

[0045] For example, the human-computer interaction system receives an interaction instruction set from the human-computer interaction field and constructs an initial instruction set. Using a large language model, the system semantically expands the initial instruction set to generate a set of synonymous instructions with diverse expressions. The system then performs speech synthesis on the initial and synonymous instruction sets to generate a training speech instruction dataset. Based on the training speech instruction dataset and using a pre-defined model fine-tuning technique, the system fine-tunes the pre-defined speech recognition model to generate the target speech recognition model.

[0046] For example, the interactive instruction set may be an instruction set composed of operation terms, such as "start arc welding", etc. The human-computer interaction system constructs an initial instruction set according to different operation terms.

[0047] For example, the human-computer interaction system performs semantic expansion processing on "start arc welding" through a large language model to generate a set of synonymous instructions with diverse expressions, such as "start welding operation", "activate electrode welding", etc.

[0048] The human-computer interaction system can synthesize and record industrial noise speech data and interactive commands such as "start arc welding," "start welding operation," and "activate electrode welding" to generate a training speech command dataset. For example, a training speech command dataset can be generated based on a set of operational terms that include industrial noise.

[0049] The human-computer interaction system inputs the training speech command dataset into a pre-set speech recognition model and fine-tunes it using LoRA (Low-Rank Adaptation). This reduces computational complexity and memory requirements, resulting in a highly efficient target speech recognition model.

[0050] According to an example embodiment, in step S200 , the human-computer interaction system recognizes a user voice instruction from a user based on a target voice recognition model to determine a user text instruction according to the user voice instruction.

[0051] For example, user voice commands may include but are not limited to operation commands (such as starting a welding program), multimodal composite commands (such as grabbing this part), or complex task commands (such as sounding an alarm if the pressure exceeds 50 MPa).

[0052] The human-computer interaction system converts the user's voice commands into text through the target speech recognition model to determine the corresponding user text commands.

[0053] Exemplarily, the human-computer interaction system receives a multimodal compound instruction, such as "Go to area A to get flange parts". The human-computer interaction system performs speech-to-text processing on the multimodal compound instruction through the target speech recognition model, and fine-tunes the multimodal compound instruction through the target speech recognition model to obtain a user text instruction, such as "Go to area A to grab the flange".

[0054] According to an example embodiment, in step S300, the human-computer interaction system performs intent fusion recognition on the user text instruction based on the user text instruction through a large language model and a preset small language model to obtain the user instruction intent.

[0055] For example, large language models can process text through natural language processing technology.

[0056] For example, the human-computer interaction system constructs a prompt template based on user text instructions and pre-set interactive task information. It then generates guiding prompt words from this prompt template. The human-computer interaction system inputs the guiding prompt words and user text instructions into a large language model for semantic understanding, identifying the key elements of the user text instructions. The human-computer interaction system then semantically aligns the guiding prompt words and key elements and identifies intent, generating the first intent information.

[0057] The human-computer interaction system constructs a sample set containing specific tasks, and trains a machine learning text detection model based on the sample set to obtain a preset language model.

[0058] The human-computer interaction system processes the user text instruction to obtain the intent data information. The human-computer interaction system processes the intent data information based on the preset language model to obtain the second intent information.

[0059] The human-computer interaction system performs intent fusion recognition on the first intention information and the second intention information to obtain the user instruction intention.

[0060] According to an example embodiment, in step S400 , the human-computer interaction system performs task decomposition and planning on the user's instruction intention to obtain a high-level task sequence.

[0061] For example, the human-computer interaction system decomposes the user's command intent to obtain at least one candidate action information. Using a large language model, the human-computer interaction system predicts the probability of the candidate action information implementing the high-level command, deriving the predicted probability of the candidate action information. Based on the predicted probability, the human-computer interaction system identifies candidate subtasks with guiding significance. Based on the current interaction environment state data and historical data, the human-computer interaction system estimates the probability of success of the candidate subtasks and obtains a feasibility score for the candidate subtask under the current conditions. Based on the predicted probability and feasibility score, the human-computer interaction system determines subtask planning information. Based on the subtask planning information, the human-computer interaction system determines a high-level task sequence.

[0062] For example, a high-level task sequence can be a multi-level task structure driven by a goal, which is a systematic execution path to achieve the goal through step-by-step decomposition and dynamic adjustment.

[0063] According to an example embodiment, in step S500 , the human-computer interaction system generates a machine control instruction according to a high-level task sequence, so as to perform human-computer interaction according to the machine control instruction.

[0064] For example, the human-computer interaction system determines subtask execution parameters based on the subtask planning information. The subtask execution parameters may include at least target parameters, input parameters, output parameters, and execution constraints. The human-computer interaction system then establishes a mapping relationship between the application programming interface and the subtask execution parameters to obtain machine control instructions. The human-computer interaction system then executes the human-computer interaction based on the machine control instructions.

[0065] Through the above embodiments, the present application can construct a target speech recognition model through an initial instruction set, so that the target speech recognition model can simulate various simulated speech scenarios in human-computer interaction experiments, so that the present application can more accurately identify user voice instructions to determine user text instructions. The present application uses a large language model and a preset small language model to perform intention fusion recognition on user text instructions, which can achieve dynamic adaptation of general semantic understanding and vertical field knowledge, so that the present application can accurately obtain the user instruction intention. Afterwards, the present application performs task decomposition and planning on the user instruction intention, determines the execution steps with the highest execution rate, and thus combines them to obtain a high-level task sequence. Finally, the present application generates machine control instructions through high-level task sequences and performs human-computer interaction, so that the present application can accurately perform human-computer interaction operations, thereby improving the efficiency of human-computer interaction testing.

[0066] Figure 2 Another schematic diagram showing a flow chart of the human-computer interaction method according to an embodiment of the present application; Figure 3 Another schematic diagram showing a flow chart of the human-computer interaction method according to an embodiment of the present application; Figure 4 Another schematic diagram showing a flow chart of the human-computer interaction method according to an embodiment of the present application; Figure 5 Another schematic diagram showing a flow chart of the human-computer interaction method according to an embodiment of the present application; Figure 6 Another schematic diagram showing a flow chart of the human-computer interaction method according to an embodiment of the present application; Figure 7 Another schematic diagram showing a flow chart of the human-computer interaction method according to an embodiment of the present application; Figure 8 Another schematic diagram showing a flow chart of the human-computer interaction method according to an embodiment of the present application; Figure 9 Another flowchart of the human-computer interaction method according to an embodiment of the present application is shown.

[0067] Alternatively, as Figure 2 As shown, step S100 may further include steps S110-S140.

[0068] In step S110 , the human-computer interaction system receives an interaction instruction set from the human-computer interaction field to construct an initial instruction set.

[0069] For example, the interaction instruction set from the human-computer interaction field may include but is not limited to device operation instructions or parameter adjustment instructions, etc. The human-computer interaction system collects different device operation instructions or parameter adjustment instructions to construct an initial instruction set.

[0070] For example, the interactive instruction set may be a device operation instruction. As an example, the device operation instruction may be “go to area A to grab the flange.” As an example, the device operation instruction may be “start a flange grabbing procedure.”

[0071] The interactive instruction set may be a parameter control instruction. As an embodiment, the parameter control instruction may be “adjust the movement speed of the robotic arm to 0.3 m / s”.

[0072] In step S120, the human-computer interaction system performs semantic expansion processing on the initial instruction set based on the language macro model to generate a synonymous instruction set.

[0073] For example, semantic expansion processing may include, but is not limited to, synonym generation, professional term conversion, or dialect adaptation.

[0074] For example, when the semantic expansion processing is synonymous generation, the initial instruction is "go to area A to get the flange part". After synonymous generation, the human-computer interaction system can obtain synonymous instructions such as "slide to area A to get the flange part" or "transmit to area A to get the flange part".

[0075] When the semantic expansion processing is converted into professional terms, the initial instruction is "go to area A to get the flange parts". After the professional terminology conversion, the human-computer interaction system can obtain "activate the picking protocol in area A".

[0076] When the semantic expansion processing is dialect adaptation, the initial instruction is "Go to area A to get flange parts". After dialect adaptation, the human-computer interaction system can get "Go to area A to get flange parts" or "Move to area A to get flange parts" and so on.

[0077] In step S130 , the human-computer interaction system performs speech synthesis recording on the initial instruction set and the synonymous instruction set to obtain a training speech instruction data set.

[0078] For example, speech synthesis recording processing may include, but is not limited to, multi-timbre generation, environmental adaptation processing, or accent emphasis processing. The human-computer interaction system generates a training speech instruction dataset through the synthesized initial instruction set and synonymous instruction set.

[0079] For example, the initial instruction is "emergency stop the robot arm in area A", and the synonymous instructions are "immediately shut down the robot in section A" and "activate the emergency stop protocol in area A".

[0080] When the speech synthesis recording is processed as multi-timbre generation, the human-computer interaction system uses WaveNet to generate multi-timbre, which can generate 8 kinds of timbres, including male and female voices and different age groups.

[0081] In the case where the speech synthesis recording process is an environment adaptation process, as an embodiment, the human-computer interaction system simulates a factory environment by superimposing 85dB white noise. As an embodiment, the human-computer interaction system simulates a factory environment by adding mechanical impact sound.

[0082] When the speech synthesis recording processing is the stress emphasis processing, the human-computer interaction system increases the volume of the command keyword "stop" by 6dB.

[0083] In step S140, the human-computer interaction system fine-tunes the preset speech recognition model based on the training speech instruction data set through a preset model fine-tuning technology to obtain a target speech recognition model.

[0084] Exemplarily, the preset speech recognition large model can be a Whisper model, and the preset model fine-tuning technology can be LoRA technology. The human-computer interaction system fine-tunes the preset speech recognition large model through LoRA technology. The human-computer interaction system introduces a low-rank matrix within each level of the model, which can effectively reduce the number of parameters and resource requirements of the model while maintaining the performance of the model, so that the model can recognize more accurately.

[0085] Through the above embodiments, the present application can ensure a high degree of match between training data and industrial scenarios by receiving real instructions from human-computer interaction scenarios. Afterwards, the present application can improve the model's recognition accuracy for colloquial expressions and dialect variants through semantic expansion. The present application can further improve the model's recognition ability by performing multi-environment simulation recording on the expanded instruction set. The present application can improve the response speed of the target speech recognition model to voice instructions by fine-tuning the model, thereby improving the testing efficiency of human-computer interaction.

[0086] Alternatively, as Figure 3 As shown, step S300 may further include steps S310-S340.

[0087] In step S310, the human-computer interaction system constructs a preset language model based on a sample set containing a specific task.

[0088] For example, the human-computer interaction system constructs a sample set containing specific tasks, and then the human-computer interaction system trains a machine learning text detection model based on the sample set to obtain a preset language model.

[0089] Task-specific sample sets can include data composition and adversarial training samples. Text detection models can have built-in factory-specific vocabulary, anti-noise processing modules, and mixed preference optimization.

[0090] For example, the data might be voice instructions based on factory cargo handling. These instructions might include noise environments (e.g., mechanical noise) and mixed dialect samples (e.g., Mandarin mixed with Cantonese and Minnan dialects). Adversarial training samples might be voice data with synthetic mixed noise, such as mixing robotic arm mechanical noise with background human voice interference, to generate adversarial samples with a dynamic signal-to-noise ratio.

[0091] The factory-specific vocabulary may include "robotic arm code" and "PLC control instructions." The noise reduction processing module may include noise suppression, which filters out mechanical roar characteristics. The human-computer interaction system uses the factory-specific vocabulary and the noise reduction processing module to identify command keywords in voice commands.

[0092] Hybrid preference optimization can use a negative supervision correction algorithm to reduce misrecognition caused by dialect pronunciation, such as the confusion of acoustic features between "emergency stop" and "machine stop".

[0093] In step S320, the human-computer interaction system obtains first intention information based on the user text instruction and the language model.

[0094] For example, the human-computer interaction system constructs a prompt template based on the user's text instructions and pre-set interaction task information. Based on the prompt template, the human-computer interaction system generates guiding prompt words. The human-computer interaction system inputs the guiding prompt words and the user's text instructions into a large language model for semantic understanding, thereby obtaining key elements of the user's text instructions. The human-computer interaction system then performs semantic alignment and intent recognition on the guiding prompt words and key elements to obtain first intent information.

[0095] Preset interactive task information may include, but is not limited to, device control instructions or security alert processing. Guiding prompt word generation can be populated with dynamic parameters. Semantic understanding can include identifying and processing polysemy and ambiguity in vocabulary and understanding the meaning of text. Key element extraction can include entity recognition and relationship extraction.

[0096] Semantic alignment and intent recognition may include but are not limited to multi-dimensional verification and intent decision tree. Multi-dimensional verification may include but are not limited to integrity check and conflict detection.

[0097] For example, a user text command might be to go to area A and grab a flange. A device control command might be to identify robotic arm operation instructions and map them to a PLC (Programmable Logic Controller) control protocol. Safety alarm processing might detect abnormal commands, such as "emergency stop grabbing" or "high temperature alarm," and trigger an emergency response mechanism.

[0098] Dynamic parameter filling can include basic templates or scene adaptation. For example, the basic template can be "Please indicate the device number to be operated (such as Area A / No. B) and specific instructions (start / speed regulation / emergency stop). For example, scene adaptation can be to generate an enhanced prompt "Please confirm the device number and current operating status of the B-line robot arm" if the user input contains noise (such as "Why is the No. B robot arm not moving?").

[0099] Entity recognition can include device number ("Area A"), operation type ("pause"), and target parameter ("0 m / s"). Relationship extraction can be to establish a triple of "Area A - Speed_Adjust - 0 m / s" and map it to PLC control instructions.

[0100] Integrity check can be to verify whether the combination of "device number + operation type" is included. If missing, a prompt "Please specify the specific device area" is triggered. Conflict detection can be when the instruction parameter exceeds the limit (such as "adjust the speed to 5 m / s" exceeds the device maximum limit of 3.5 m / s), it is marked as a "red risk instruction".

[0101] In step S330, the human - machine interaction system obtains the second intention information based on the user text instruction according to the preset language small model.

[0102] For example, the human - machine interaction system processes the user text instruction to obtain intention data information. The human - machine interaction system processes the intention data information based on the preset language small model to obtain the second intention information.

[0103] Intention data processing can include but is not limited to pre - processing in a noisy environment and extraction of intention elements, etc. Pre - processing in a noisy environment can include but is not limited to dialect normalization and noise suppression, etc.

[0104] Exemplarily, dialect normalization can be to convert the Cantonese instruction "Why doesn't point B move?" to the standard instruction "The robotic arm at point B has an abnormal shutdown". Noise suppression can be to use a dual - path attention mechanism to filter out the mechanical roaring sound of the robotic arm.

[0105] The human - machine interaction system extracts intention elements from "The robotic arm at point B has an abnormal shutdown" to obtain "Check the device number and current operating status of the robotic arm on line B". The human - machine interaction system performs semantic integrity screening on the instruction "Check the device number and current operating status of the robotic arm on line B" and executes a full - dimension verification.

[0106] In step S340, the human - machine interaction system performs intention fusion recognition on the first intention information and the second intention information to obtain the user instruction intention.

[0107] For example, the human - machine interaction system determines whether the first intention information and the second intention information are consistent. If so, the human - machine interaction system outputs the user instruction intention; if not, the human - machine interaction system supplements and corrects the user intention through a multi - source information integration mechanism based on the first intention information and the second intention information to obtain the user instruction intention.

[0108] The multi - source information integration mechanism can be real - time data fusion.

[0109] For example, the first intention information is "Please confirm the B-line robotic arm equipment number and current operating status", and the second intention information is "Check the B-line robotic arm equipment number and current operating status". The first intention information and the second intention information are consistent, and the human-computer interaction system outputs the user command intention.

[0110] The first intent is "Please restart the B-line robot arm device number and current operating status," while the second intent is "Check the B-line robot arm device number and current operating status." Since the first and second intents are inconsistent, the human-computer interaction system uses real-time data fusion to read the robot arm status code. The system intercepts the "restart" command in the first intent and extracts the "check" command in the second intent. The system then fuses the real-time data from the devices to generate an enhanced command, ultimately outputting "Check the B-line robot arm device number and current operating status."

[0111] Through the above embodiments, the present application can parse the deep semantics of user instructions through the massive pre-training data of the language large model, and can support the context-related reasoning of complex instructions. The present application can build a preset language small model based on a specific task sample set to achieve domain focus of the small model and reduce the risk of semantic divergence of the large model in professional scenarios. The present application can obtain accurate user instruction intentions through intent fusion recognition, thereby improving the accuracy of instruction analysis and further improving the testing efficiency of human-computer interaction.

[0112] Alternatively, as Figure 4 As shown, step S310 may further include steps S311-S312.

[0113] In step S311 , the human-computer interaction system constructs a sample set containing a specific task.

[0114] For example, a task-specific sample set may include both data composition and adversarial training samples.

[0115] For example, the data might be voice instructions based on factory cargo handling. These instructions might include noise environments (e.g., mechanical noise) and mixed dialect samples (e.g., Mandarin mixed with Cantonese and Minnan dialects). Adversarial training samples might be voice data with synthetic mixed noise, such as mixing robotic arm mechanical noise with background human voice interference, to generate adversarial samples with a dynamic signal-to-noise ratio.

[0116] In step S312, the human-computer interaction system processes the intention data information based on the preset language model to obtain second intention information.

[0117] For example, the preset language model can be a text detection model, and the text detection model can have a built-in factory-specific vocabulary, an anti-noise processing module, and mixed preference optimization.

[0118] For example, the factory-specific vocabulary may include "robotic arm code" and "PLC control instructions." The noise reduction processing module may include noise suppression, which filters out mechanical roar characteristics. The human-computer interaction system uses the factory-specific vocabulary and the noise reduction processing module to identify command keywords in voice commands.

[0119] Hybrid preference optimization can use a negative supervision correction algorithm to reduce misrecognition caused by dialect pronunciation, such as the confusion of acoustic features between "emergency stop" and "machine stop".

[0120] Through the above examples, this application can construct a sample set containing a mixture of mechanical noise and dialect, so that the small model can cover typical factory interference scenarios during the training phase. By pre-setting the small language model, this application can further improve the accuracy of command analysis, thereby improving the efficiency of human-computer interaction testing.

[0121] Alternatively, as Figure 5 As shown, step S320 may further include steps S321-S324.

[0122] In step S321, the human-computer interaction system constructs a prompt template based on the user text instruction and preset interaction task information.

[0123] For example, the preset interactive task information may include, but is not limited to, device control instructions or security alarm processing.

[0124] For example, a user text command might be to go to area A and grab a flange. A device control command might be to identify robotic arm operation instructions and map them to a PLC (Programmable Logic Controller) control protocol. Safety alarm processing might detect abnormal commands, such as "emergency stop grabbing" or "high temperature alarm," and trigger an emergency response mechanism.

[0125] In step S322, the human-computer interaction system generates guiding prompt words based on the prompt template.

[0126] For example, the generation of guiding prompt words can be filled with dynamic parameters.

[0127] Exemplarily, dynamic parameter filling may include basic templates or scene adaptation. For example, the basic template may be to indicate the device number to be operated (such as area A / number B) and specific instructions (start / speed regulation / emergency stop). For example, scene adaptation may be to generate an enhanced prompt "Please confirm the device number and current operating status of the B-line robot arm" if the user input contains noise (such as "Why is the No. B robot arm not moving?").

[0128] In step S323, the human-computer interaction system inputs the guiding prompt words and the user text instructions into the language model for semantic understanding to obtain the key elements in the user text instructions.

[0129] For example, semantic understanding may include identifying and processing the polysemy and ambiguity of vocabulary and understanding the meaning of text. Key element extraction may include entity recognition and relationship extraction.

[0130] Exemplarily, entity recognition may include a device number ("Area A"), an operation type ("pause"), and a target parameter ("0 m / s"). Relationship extraction may be to establish a triple "Area A - Speed_Adjust - 0 m / s" and map it to a PLC control instruction.

[0131] In step S324, the human - machine interaction system performs semantic alignment and intent recognition on the guiding prompt words and key elements to obtain first intent information.

[0132] For example, semantic alignment and intent recognition may include but are not limited to multi - dimensional verification and intent decision trees. Multi - dimensional verification may include but is not limited to integrity checking and conflict detection, etc.

[0133] Exemplarily, integrity checking may be to verify whether the combination of "device number + operation type" is included. If it is missing, a follow - up question is triggered: "Please specify the specific device area." Conflict detection may be when the instruction parameters exceed the limit (such as "adjust the speed to 5 m / s" exceeding the device's maximum limit of 3.5 m / s), which is marked as a "red - risk instruction".

[0134] Through the above - mentioned embodiments, the present application can construct a prompt template by presetting interaction task information, enabling the language large - model to focus its attention on domain - key elements. The present application can also trigger the domain - knowledge activation mechanism of the large - model through guiding prompt words, improving the semantic - understanding accuracy of the language large - model. The present application can obtain more accurate first intent information through semantic alignment and intent recognition, thereby improving the test efficiency of human - machine interaction.

[0135] Optionally, as Figure 6 shown, step S330 may further include steps S331 - S332.

[0136] In step S331, the human - machine interaction system processes the user's text instruction to obtain intent data information.

[0137] For example, intent data processing may include but is not limited to pre - processing in a noisy environment and extracting intent elements, etc. Pre - processing in a noisy environment may include but is not limited to dialect normalization and noise suppression, etc.

[0138] Exemplarily, dialect normalization may be to convert the Cantonese instruction "Why doesn't point B move?" to the standard instruction "The robotic arm at point B has an abnormal shutdown." Noise suppression may be to use a dual - path attention mechanism to filter out the mechanical roaring sound of the robotic arm.

[0139] In step S332, the human-computer interaction system processes the intention data information based on the preset language model to obtain second intention information.

[0140] For example, the human-computer interaction system extracts the intent element for the command "Abnormal shutdown of robot arm B" and obtains the command "Check the equipment number and current operating status of robot arm line B." The human-computer interaction system then performs semantic integrity screening on the command "Check the equipment number and current operating status of robot arm line B" and performs a full-dimensional verification.

[0141] Through the above embodiments, the present application can process user text instructions through a preset language model, and perform similarity matching between user intention data and semantic vectors, so as to obtain more accurate second intention information, thereby improving the testing efficiency of human-computer interaction.

[0142] Alternatively, as Figure 7 As shown, step S340 may further include steps S341-S332.

[0143] In step S341 , the human-computer interaction system determines whether the first intention information and the second intention information are consistent. If so, the human-computer interaction system outputs the user instruction intention.

[0144] For example, the first intention information is "Please confirm the B-line robotic arm equipment number and current operating status", and the second intention information is "Check the B-line robotic arm equipment number and current operating status". The first intention information and the second intention information are consistent, and the human-computer interaction system outputs the user command intention.

[0145] In step S342, the human-computer interaction system determines whether the first intention information and the second intention information are consistent. If not, based on the first intention information and the second intention information, the human-computer interaction system supplements and corrects the user intention through a multi-source information integration mechanism to obtain the user's instruction intention.

[0146] For example, the multi-source information integration mechanism can be real-time data fusion.

[0147] For example, the first intent is "Please restart the B-line robot arm device number and current operating status," and the second intent is "Check the B-line robot arm device number and current operating status." The first and second intents are inconsistent. The human-computer interaction system reads the robot arm status code through real-time data fusion. The human-computer interaction system intercepts the "restart" command in the first intent and extracts the "check" command in the second intent. The human-computer interaction system integrates the real-time data of the devices to generate an enhanced command, and finally outputs "Check the B-line robot arm device number and current operating status."

[0148] Through the above embodiments, the present application can improve the accuracy of intent analysis in complex scenarios through multi-model collaboration and multi-source information integration mechanism.

[0149] Alternatively, as Figure 8 As shown, step S400 may further include steps S410-S460.

[0150] In step S410 , the human-computer interaction system decomposes the user instruction intention to obtain at least one candidate action information.

[0151] For example, the human-computer interaction system can convert voice commands into structured commands, such as extracting the object, action, starting point, and end point in the user's command intent. The human-computer interaction system can also predict the path of the action, starting point, and end point to obtain candidate action information.

[0152] For example, the user's instruction intention may be "move the display from area A to area B", object: display; starting point: area A; end point: area B; operation type: transport.

[0153] The human-computer interaction system generates a list of candidate actions: the robotic arm grabs the display; the AGV (Automated Guided Vehicle) is dispatched to area A; the path is dynamically planned; obstacles are avoided in real time; and the display placement posture is calibrated.

[0154] In step S420, the human-computer interaction system predicts the probability of the candidate action information implementing the high-level instruction through the language model to obtain the predicted probability.

[0155] For example, the human-computer interaction system uses a large language model to drive decision-making technology, evaluating the relevance of each action to core instructions. The human-computer interaction system integrates a human-computer collaboration rule base to eliminate low-compliance actions.

[0156] For example, robotic arm grasping as the operation of direct operation object has a correlation of 0.95; AGV scheduling as a necessary condition for transportation has a correlation of 0.93; the execution of path planning will affect transportation efficiency, with a correlation of 0.88; real-time obstacle avoidance as a safety requirement has a correlation of 0.82; manual handling backup plan is a low-compliance action, with a correlation of 0.12.

[0157] The human-computer interaction system is combined with the human-computer collaboration rule library to directly eliminate the low-correlation "direct manual handling".

[0158] In step S430 , the human-computer interaction system determines candidate subtasks with guiding significance based on the predicted probability.

[0159] Exemplarily, the human-computer interaction system retains candidate actions with a relevance probability ≥ 0.8, such as robotic arm grasping, AGV scheduling, path planning, or obstacle avoidance.

[0160] In step S440 , the human-computer interaction system estimates the probability of the execution success rate of the candidate subtask based on the current interaction environment state data and historical data to obtain a feasibility score of the candidate subtask under the current conditions.

[0161] For example, the interactive environmental status data can be the working status of the equipment, and the historical data can be the operating status of the equipment. The human-computer interaction system confirms the probability of the success rate of the execution of the candidate subtask through the environmental status data and historical data, and comprehensively obtains the feasibility score of the candidate subtask under the current conditions.

[0162] For example, the human-machine interaction system uses environmental status data to confirm the availability of the robotic arm in Area A (idle status score 0.95), the sufficient battery life of the AGV (score 0.92), and the unobstructed placement space in Area B (score 0.89). The human-machine interaction system then uses the process database to determine if the probability of static damage to the "monitor handling" task in history is greater than 10%, adding an anti-static packaging subtask (feasibility score +0.15).

[0163] In step S450 , the human-computer interaction system determines subtask planning information based on the predicted probability and feasibility score.

[0164] For example, the human-computer interaction system can combine the predicted probability and feasibility score to perform weighted scoring to obtain a comprehensive score for each candidate subtask, arrange the priority of the candidate subtasks according to the comprehensive score, and determine the subtask planning information.

[0165] For example, the correlation accounts for 60%, the feasibility score accounts for 40%, the robotic arm grasping score is 0.95×0.6=0.57, 0.95×0.4=0.38, and the comprehensive score is 0.95; the calculation method is shown above and will not be repeated here. The results are given directly: the AGV scheduling comprehensive score is 0.93; the dynamic path planning comprehensive score is 0.89; the real-time obstacle avoidance comprehensive score is 0.83.

[0166] The human-computer interaction system executes robotic arm grasping - AGV scheduling - path planning - obstacle avoidance according to priority.

[0167] In step S460 , the human-computer interaction system determines a high-level task sequence based on the subtask planning information.

[0168] For example, based on temporal logic, a high-level task sequence is determined.

[0169] For example, the human-machine interaction system can generate an optimal path by scanning the site. The system then dispatches the AGV to area A and controls the robotic arm to grab the display. The system controls the AGV along the planned path. The system's real-time obstacle avoidance module activates. The system controls the robotic arm to calibrate its placement and safely release the display to area B.

[0170] Through the above-mentioned embodiments, this application can accurately screen core subtasks through intent parsing and relevance prediction based on a large language model. This application can improve the success rate of task execution by integrating a dynamic evaluation model with real-time environmental perception and historical execution data. This application can significantly improve the reliability and execution efficiency of task planning by making weighted decisions based on relevance probability and feasibility scores.

[0171] Alternatively, as Figure 9 As shown, step S500 may further include steps S510-S530.

[0172] In step S510, the human-computer interaction system determines subtask execution parameters based on the subtask planning information. The subtask execution parameters may at least include target parameters, input parameters, output parameters, and execution constraints.

[0173] For example, target parameters may include but are not limited to navigation tasks or process tasks, input parameters may be sensor data, output parameters may be action feedback standards, and execution constraints may be physical resource limitations.

[0174] For example, if the target parameter is a navigation task, the human-computer interaction system can define the target coordinates as the shelf coordinates in area B. If the target parameter is a process task, the human-computer interaction system can set the clamping force threshold of the robot end to 25N±2N.

[0175] The input parameters can be configured to configure the lidar scanning frequency to 20Hz, the output parameters can be AGV speed fluctuation <0.2m / s, path deviation <10cm, and the execution constraint condition can be limiting the CPU usage of the path planning algorithm to ≤35%.

[0176] In step S520 , the human-computer interaction system establishes a mapping relationship between the application programming interface and the subtask execution parameters to obtain a machine control instruction.

[0177] Exemplarily, the human-computer interaction system calls an execution program through an API (Application Programming Interface), executes subtasks, and obtains machine control instructions.

[0178] In step S530 , the human-computer interaction system performs human-computer interaction based on the machine control instruction.

[0179] For example, the human-computer interaction system can complete the clamping-moving-releasing action sequence according to API instructions.

[0180] Through the above embodiments, this application can accelerate instruction generation by building semantically driven API mapping rules combined with resource awareness mechanisms. During the execution phase, this application implements real-time verification of output parameters and dynamic adjustment of constraints. This application achieves precise and efficient machine control through parametric modeling and dynamic interface mapping mechanisms.

[0181] Figure 10 A schematic structural diagram of a human-computer interaction system according to an embodiment of the present application is shown.

[0182] According to another aspect of the present application, the present application provides a human-computer interaction system based on large model assistance. Figure 10 As shown, the human-computer interaction system 10 includes a speech recognition module 11 , an intention understanding module 12 , a task planning module 13 and an instruction execution module 14 .

[0183] According to an exemplary embodiment, the speech recognition module 11 constructs a target speech recognition model based on the initial instruction set, and recognizes a user speech instruction from the user based on the target speech recognition model to determine a user text instruction according to the user speech instruction.

[0184] For example, the speech recognition module 11 receives an interactive instruction set from the human-computer interaction field and constructs an initial instruction set. Using a large language model, the speech recognition module 11 semantically expands the initial instruction set to generate a set of synonymous instructions with diverse expressions. The speech recognition module 11 performs speech synthesis recording on the initial instruction set and the synonymous instruction set to obtain a training speech instruction dataset. Based on the training speech instruction dataset and using a pre-defined model fine-tuning technique, the speech recognition module 11 fine-tunes the pre-set speech recognition large model to obtain a target speech recognition model.

[0185] For example, the interactive instruction set may be an instruction set composed of operation terms, such as "start arc welding", etc. The speech recognition module 11 constructs an initial instruction set according to different operation terms.

[0186] Exemplarily, the speech recognition module 11 performs semantic expansion processing on "start arc welding" through the language large model to generate a synonymous instruction set with diverse expressions, such as "start welding operation", "activate electrode welding", etc.

[0187] The speech recognition module 11 can synthesize and record the speech data of industrial noise and interactive commands such as "start arc welding," "start welding operation," and "activate electrode welding" to obtain a training speech command dataset. For example, the training speech command dataset can be obtained based on a set of operation terms that include industrial noise.

[0188] The speech recognition module 11 inputs the training speech command dataset into a pre-set speech recognition model and fine-tunes the pre-set speech recognition model using LoRA (Low-Rank Adaptation). This reduces computational complexity and memory requirements, resulting in a highly efficient target speech recognition model.

[0189] Optionally, the speech recognition module 11 receives an interaction instruction set from the field of human-computer interaction to construct an initial instruction set.

[0190] For example, the interaction instruction set from the field of human-computer interaction may include but is not limited to device operation instructions or parameter adjustment instructions, etc. The speech recognition module 11 collects different device operation instructions or parameter adjustment instructions to construct an initial instruction set.

[0191] For example, the interactive instruction set may be a device operation instruction. As an example, the device operation instruction may be “go to area A to grab the flange.” As an example, the device operation instruction may be “start a flange grabbing procedure.”

[0192] The interactive instruction set may be a parameter control instruction. As an embodiment, the parameter control instruction may be “adjust the movement speed of the robotic arm to 0.3 m / s”.

[0193] Optionally, the speech recognition module 11 performs semantic expansion processing on the initial instruction set based on the language macro model to generate a synonymous instruction set.

[0194] For example, semantic expansion processing may include, but is not limited to, synonym generation, professional term conversion, or dialect adaptation.

[0195] For example, when the semantic expansion processing is synonymous generation, the initial instruction is "go to area A to get the flange parts". After synonymous generation, the speech recognition module 11 can obtain synonymous instructions such as "slide to area A to get the flange parts" or "transmit to area A to get the flange parts".

[0196] When the semantic expansion processing is converted into professional terms, the initial instruction is "go to area A to pick up flange parts". After the professional term conversion, the voice recognition module 11 can obtain "activate the picking protocol of area A".

[0197] When the semantic expansion processing is dialect adaptation, the initial instruction is "go to area A to get flange parts". After dialect adaptation, the speech recognition module 11 can obtain "go to area A to get flange parts" or "move to area A to get flange parts" and so on.

[0198] Optionally, the speech recognition module 11 performs speech synthesis recording on the initial instruction set and the synonymous instruction set to obtain a training speech instruction data set.

[0199] For example, the speech synthesis recording process may include but is not limited to multi-timbre generation, environment adaptation processing, or accent emphasis processing. The speech recognition module 11 generates a training speech instruction data set by synthesizing the initial instruction set and the synonymous instruction set.

[0200] For example, the initial instruction is "emergency stop the robot arm in area A", and the synonymous instructions are "immediately shut down the robot in section A" and "activate the emergency stop protocol in area A".

[0201] When the speech synthesis recording process is multi-timbre generation, the speech recognition module 11 performs multi-timbre generation through WaveNet, and can generate 8 timbres, including male and female voices and different age groups.

[0202] When the speech synthesis recording process is an environment adaptation process, as an embodiment, the speech recognition module 11 simulates a factory environment by superimposing 85dB white noise. As an embodiment, the speech recognition module 11 simulates a factory environment by adding mechanical impact sound.

[0203] When the speech synthesis recording process is the accent emphasis process, the speech recognition module 11 increases the volume of the instruction keyword "stop" by 6dB.

[0204] Optionally, the speech recognition module 11 fine-tunes the preset speech recognition large model based on the training speech instruction data set through a preset model fine-tuning technology to obtain a target speech recognition model.

[0205] Exemplarily, the preset speech recognition large model can be a Whisper model, and the preset model fine-tuning technology can be LoRA technology. The speech recognition module 11 fine-tunes the preset speech recognition large model through LoRA technology. The speech recognition module 11 introduces a low-rank matrix within each level of the model, which can effectively reduce the number of parameters and resource requirements of the model while maintaining the performance of the model, so that the model can recognize more accurately.

[0206] Through the above embodiments, the present application can ensure a high degree of match between training data and industrial scenarios by receiving real instructions from human-computer interaction scenarios. Afterwards, the present application can improve the model's recognition accuracy for colloquial expressions and dialect variants through semantic expansion. The present application can further improve the model's recognition ability by performing multi-environment simulation recording on the expanded instruction set. The present application can improve the response speed of the target speech recognition model to voice instructions by fine-tuning the model, thereby improving the testing efficiency of human-computer interaction.

[0207] Optionally, the voice recognition module 11 identifies a user voice instruction from the user to determine a user text instruction according to the user voice instruction.

[0208] For example, user voice commands may include but are not limited to operation commands (such as starting a welding program), multimodal composite commands (such as grabbing this part), or complex task commands (such as sounding an alarm if the pressure exceeds 50 MPa).

[0209] The speech recognition module 11 performs speech-to-text processing on the user's speech instruction through the target speech recognition model to determine the corresponding user text instruction.

[0210] Exemplarily, the speech recognition module 11 receives a multimodal compound instruction, such as "Go to area A to get flange parts". The speech recognition module 11 performs speech-to-text processing on the multimodal compound instruction through the target speech recognition model, and fine-tunes the multimodal compound instruction through the target speech recognition model to obtain a user text instruction, such as "Go to area A to grab the flange".

[0211] According to an exemplary embodiment, the intention understanding module 12 performs intention fusion recognition on the user text instruction based on the user text instruction through a large language model and a preset small language model to obtain the user instruction intention.

[0212] For example, large language models can process text through natural language processing technology.

[0213] For example, the intent understanding module 12 constructs a prompt template based on the user's text instruction and pre-set interactive task information, and then generates guiding prompt words from the prompt template. The intent understanding module 12 inputs the guiding prompt words and the user's text instruction into the language model for semantic understanding, obtaining the key elements of the user's text instruction. The intent understanding module 12 then performs semantic alignment and intent recognition on the guiding prompt words and key elements to obtain the first intent information.

[0214] The intention understanding module 12 constructs a sample set containing specific tasks and trains a machine learning text detection model based on the sample set to obtain a preset language model.

[0215] The intention understanding module 12 processes the user text instruction to obtain the intention data information. The intention understanding module 12 processes the intention data information based on the preset language model to obtain the second intention information.

[0216] The intention understanding module 12 performs intention fusion recognition on the first intention information and the second intention information to obtain the user instruction intention.

[0217] Optionally, the intention understanding module 12 constructs a preset language small model based on a sample set containing a specific task.

[0218] For example, the intent understanding module 12 constructs a sample set containing a specific task, and then the intent understanding module 12 sample set is based on the text detection model trained by machine learning to obtain a preset language model.

[0219] Task-specific sample sets can include data composition and adversarial training samples. Text detection models can have built-in factory-specific vocabulary, anti-noise processing modules, and mixed preference optimization.

[0220] For example, the data might be voice instructions based on factory cargo handling. These instructions might include noise environments (e.g., mechanical noise) and mixed dialect samples (e.g., Mandarin mixed with Cantonese and Minnan dialects). Adversarial training samples might be voice data with synthetic mixed noise, such as mixing robotic arm mechanical noise with background human voice interference, to generate adversarial samples with a dynamic signal-to-noise ratio.

[0221] The factory-specific vocabulary may include "robot arm code" and "PLC control instructions." The noise reduction processing module may include noise suppression, which filters out mechanical roar characteristics. The intent understanding module 12 uses the factory-specific vocabulary and the noise reduction processing module to identify command keywords in the voice command.

[0222] Hybrid preference optimization can use a negative supervision correction algorithm to reduce misrecognition caused by dialect pronunciation, such as the confusion of acoustic features between "emergency stop" and "machine stop".

[0223] Optionally, the intention understanding module 12 obtains the first intention information based on the user text instruction and the language model.

[0224] For example, the intent understanding module 12 constructs a prompt template based on the user text instruction and the preset interactive task information. The intent understanding module 12 generates guiding prompt words based on the prompt template. The intent understanding module 12 inputs the guiding prompt words and the user text instruction into the language macro model for semantic understanding, thereby obtaining key elements in the user text instruction. The intent understanding module 12 performs semantic alignment and intent recognition on the guiding prompt words and key elements to obtain first intent information.

[0225] Preset interactive task information may include, but is not limited to, device control instructions or security alert processing. Guiding prompt word generation can be populated with dynamic parameters. Semantic understanding can include identifying and processing polysemy and ambiguity in vocabulary and understanding the meaning of text. Key element extraction can include entity recognition and relationship extraction.

[0226] Semantic alignment and intent recognition may include but are not limited to multi-dimensional verification and intent decision tree. Multi-dimensional verification may include but are not limited to integrity check and conflict detection.

[0227] For example, a user text command might be to go to area A and grab a flange. A device control command might be to identify robotic arm operation instructions and map them to a PLC (Programmable Logic Controller) control protocol. Safety alarm processing might detect abnormal commands, such as "emergency stop grabbing" or "high temperature alarm," and trigger an emergency response mechanism.

[0228] Dynamic parameter filling can include basic templates or scene adaptation. For example, the basic template can be "Please indicate the device number to be operated (such as Area A / No. B) and specific instructions (start / speed regulation / emergency stop). For example, scene adaptation can be to generate an enhanced prompt "Please confirm the device number and current operating status of the B-line robot arm" if the user input contains noise (such as "Why is the No. B robot arm not moving?").

[0229] Entity recognition can include the device number ("Area A"), operation type ("Pause"), and target parameter ("0m / s"). Relationship extraction can create a triplet of "Area A - Speed_Adjust - 0m / s" and map it to a PLC control instruction.

[0230] Integrity checks can verify the presence of the "device number + operation type" combination. If missing, a follow-up question will be prompted: "Please specify the specific device area." Conflict detection can flag a command as a "red risk command" when a parameter exceeds a limit (e.g., "speed adjustment to 5m / s" exceeds the device's maximum limit of 3.5m / s).

[0231] Optionally, the intention understanding module 12 obtains the second intention information based on the user text instruction and a preset language model.

[0232] For example, the intention understanding module 12 processes the user text instruction to obtain the intention data information. The intention understanding module 12 processes the intention data information based on the preset language model to obtain the second intention information.

[0233] Intent data processing may include, but is not limited to, noise environment preprocessing and intent element extraction. Noise environment preprocessing may include, but is not limited to, dialect normalization and noise suppression.

[0234] Exemplarily, dialect normalization can be converting the Cantonese instruction "Why doesn't the B number move?" to the standard instruction "The robotic arm of B number has abnormal shutdown". Noise suppression can be filtering out the mechanical rumbling sound of the robotic arm using a dual-path attention mechanism.

[0235] The intent understanding module 12 extracts intent elements from "The robotic arm of B number has abnormal shutdown" and obtains "Check the device number and current operating status of the robotic arm on B line". The intent understanding module 12 performs semantic integrity screening on the instruction "Check the device number and current operating status of the robotic arm on B line" and executes a full-dimensional verification.

[0236] Optionally, the intent understanding module 12 performs intent fusion recognition on the first intent information and the second intent information to obtain the user instruction intent.

[0237] For example, the intent understanding module 12 determines whether the first intent information and the second intent information are consistent. If so, the intent understanding module 12 outputs the user instruction intent; if not, the intent understanding module 12 supplements and corrects the user intent through a multi-source information integration mechanism based on the first intent information and the second intent information to obtain the user instruction intent.

[0238] The multi-source information integration mechanism can be real-time data fusion.

[0239] Exemplarily, the first intent information is "Please confirm the device number and current operating status of the robotic arm on B line", the second intent information is "Check the device number and current operating status of the robotic arm on B line", the first intent information and the second intent information are consistent, and the intent understanding module 12 outputs the user instruction intent.

[0240] The first intent information is "Please restart the device number and current operating status of the robotic arm on B line", the second intent information is "Check the device number and current operating status of the robotic arm on B line", the first intent information and the second intent information are inconsistent. The intent understanding module 12 reads the robotic arm status code through real-time data fusion. The intent understanding module 12 intercepts the "restart" instruction in the first intent information and extracts the "check" instruction in the second intent information. The intent understanding module 12 fuses the device real-time data to generate an enhanced instruction and finally outputs "Check the device number and current operating status of the robotic arm on B line".

[0241] Through the above embodiments, the present application can parse the deep semantics of user instructions through the massive pre-training data of the language model, and can support the context-related reasoning of complex instructions. The present application can build a preset language small model based on a specific task sample set to achieve the domain focus of the small model and reduce the semantic divergence risk of the large model in professional scenarios. The present application can obtain accurate user instruction intent through intent fusion recognition, thereby improving the accuracy of instruction analysis and further improving the test efficiency of human-computer interaction.

[0242] Optionally, the intention understanding module 12 constructs a sample set containing a specific task.

[0243] For example, a task-specific sample set may include both data composition and adversarial training samples.

[0244] For example, the data might be voice instructions based on factory cargo handling. These instructions might include noise environments (e.g., mechanical noise) and mixed dialect samples (e.g., Mandarin mixed with Cantonese and Minnan dialects). Adversarial training samples might be voice data with synthetic mixed noise, such as mixing robotic arm mechanical noise with background human voice interference, to generate adversarial samples with a dynamic signal-to-noise ratio.

[0245] Optionally, the intention understanding module 12 processes the intention data information based on a preset language model to obtain second intention information.

[0246] For example, the preset language model can be a text detection model, and the text detection model can have a built-in factory-specific vocabulary, an anti-noise processing module, and mixed preference optimization.

[0247] For example, the factory-specific vocabulary may include "robotic arm code" and "PLC control instructions." The noise reduction processing module may include noise suppression, which filters out mechanical roar characteristics. The intent understanding module 12 uses the factory-specific vocabulary and the noise reduction processing module to identify command keywords in the voice command.

[0248] Hybrid preference optimization can use a negative supervision correction algorithm to reduce misrecognition caused by dialect pronunciation, such as the confusion of acoustic features between "emergency stop" and "machine stop".

[0249] Through the above examples, this application can construct a sample set containing a mixture of mechanical noise and dialect, so that the small model can cover typical factory interference scenarios during the training phase. By pre-setting the small language model, this application can further improve the accuracy of command analysis, thereby improving the efficiency of human-computer interaction testing.

[0250] Optionally, the intention understanding module 12 constructs a prompt template based on the user text instruction and preset interactive task information.

[0251] For example, the preset interactive task information may include, but is not limited to, device control instructions or security alarm processing.

[0252] For example, a user text command might be to go to area A and grab a flange. A device control command might be to identify robotic arm operation instructions and map them to a PLC (Programmable Logic Controller) control protocol. Safety alarm processing might detect abnormal commands, such as "emergency stop grabbing" or "high temperature alarm," and trigger an emergency response mechanism.

[0253] Optionally, the intention understanding module 12 generates guiding prompt words based on the prompt template.

[0254] For example, the generation of guiding prompt words can be filled with dynamic parameters.

[0255] Exemplarily, dynamic parameter filling may include basic templates or scene adaptation. For example, the basic template may be to indicate the device number to be operated (such as area A / number B) and specific instructions (start / speed regulation / emergency stop). For example, scene adaptation may be to generate an enhanced prompt "Please confirm the device number and current operating status of the B-line robot arm" if the user input contains noise (such as "Why is the No. B robot arm not moving?").

[0256] Optionally, the intention understanding module 12 inputs the guiding prompt words and the user text instructions into the language model for semantic understanding to obtain key elements in the user text instructions.

[0257] For example, semantic understanding can include identifying and handling polysemy and ambiguity of words and understanding the meaning of text. Key element extraction can include entity recognition and relationship extraction.

[0258] For example, entity recognition can include the device number ("A zone"), the operation type ("pause"), and the target parameter ("0 m / s"). Relationship extraction can be to create a triple "A zone - Speed_Adjust - 0 m / s" and map it to a PLC control instruction.

[0259] Optionally, the intention understanding module 12 performs semantic alignment and intention recognition on the guiding prompt words and key elements to obtain the first intention information.

[0260] For example, semantic alignment and intent recognition may include but are not limited to multi-dimensional verification and intent decision tree, and multi-dimensional verification may include but are not limited to integrity check and conflict detection.

[0261] For example, an integrity check could verify the presence of the "device number + operation type" combination. If missing, a follow-up question would be prompted: "Please specify the specific device area." Conflict detection could flag a command as a "red risk command" if the command parameter exceeds the limit (e.g., "speed adjustment to 5m / s" exceeds the device's maximum limit of 3.5m / s).

[0262] Through the above embodiments, the present application can construct a prompt template by presetting the interactive task information, so that the language model can focus on the key elements of the domain. The present application can also trigger the domain knowledge activation mechanism of the large model through guiding prompt words to improve the accuracy of the semantic understanding of the language model. The present application can obtain more accurate first intent information through semantic alignment and intent recognition, thereby improving the testing efficiency of human-computer interaction.

[0263] Optionally, the intent understanding module 12 processes the user text instruction to obtain intent data information.

[0264] For example, intent data processing may include but is not limited to preprocessing in a noisy environment and extracting intent elements, etc. Preprocessing in a noisy environment may include but is not limited to dialect normalization and noise suppression, etc.

[0265] Exemplarily, dialect normalization may be converting the Cantonese instruction "Why doesn't point B move?" to the standard instruction "The robotic arm of line B has an abnormal shutdown". Noise suppression may be filtering out the mechanical roaring sound of the robotic arm using a dual-path attention mechanism.

[0266] Optionally, the intent understanding module 12 processes the intent data information based on a preset language small model to obtain the second intent information.

[0267] Exemplarily, the intent understanding module 12 extracts intent elements from "The robotic arm of line B has an abnormal shutdown" to obtain "Check the equipment number and current operating status of the robotic arm on line B". The intent understanding module 12 performs semantic integrity screening on the instruction "Check the equipment number and current operating status of the robotic arm on line B" and executes a full-dimensional verification.

[0268] Through the above embodiments, the present application can process the user text instruction through a preset language small model and perform similarity matching between the user intent data and the semantic vector, so as to obtain more accurate second intent information, thereby improving the test efficiency of human-computer interaction.

[0269] Optionally, the intent understanding module 12 determines whether the first intent information and the second intent information are consistent. If so, the intent understanding module 12 outputs the user instruction intent.

[0270] Exemplarily, the first intent information is "Please confirm the equipment number and current operating status of the robotic arm on line B", the second intent information is "Check the equipment number and current operating status of the robotic arm on line B", the first intent information and the second intent information are consistent, and the intent understanding module 12 outputs the user instruction intent.

[0271] Optionally, the intent understanding module 12 determines whether the first intent information and the second intent information are consistent. If not, based on the first intent information and the second intent information, the intent understanding module 12 supplements and corrects the user intent through a multi-source information integration mechanism to obtain the user instruction intent.

[0272] For example, the multi-source information integration mechanism may be real-time data fusion.

[0273] For example, the first intent message is "Please restart the B-line robot arm device number and current operating status," and the second intent message is "Check the B-line robot arm device number and current operating status." The first and second intent messages are inconsistent, and the intent understanding module 12 reads the robot arm status code through real-time data fusion. The intent understanding module 12 intercepts the "restart" instruction in the first intent message and extracts the "check" instruction in the second intent message. The intent understanding module 12 fuses the real-time data of the device to generate an enhanced instruction, and finally outputs "Check the B-line robot arm device number and current operating status."

[0274] Through the above embodiments, the present application can improve the accuracy of intent analysis in complex scenarios through multi-model collaboration and multi-source information integration mechanism.

[0275] According to an exemplary embodiment, the task planning module 13 performs task decomposition and planning on the user's instruction intention to obtain a high-level task sequence.

[0276] For example, the task planning module 13 decomposes the user's instruction intention and obtains at least one candidate action information. The task planning module 13 predicts the probability of the candidate action information for implementing high-level instructions through the language large model, and obtains the predicted probability of the candidate action information. The task planning module 13 determines the candidate subtasks with guiding significance based on the predicted probability. The task planning module 13 estimates the probability of the execution success rate of the candidate subtasks based on the current interactive environment status data and historical data, and obtains the feasibility score of the candidate subtask under the current conditions. The task planning module 13 determines the subtask planning information based on the predicted probability and feasibility score. The task planning module 13 determines the high-level task sequence based on the subtask planning information.

[0277] For example, a high-level task sequence can be a multi-level task structure driven by a goal, which is a systematic execution path to achieve the goal through step-by-step decomposition and dynamic adjustment.

[0278] Optionally, the task planning module 13 decomposes the user instruction intention to obtain at least one candidate action information.

[0279] For example, the task planning module 13 can convert voice instructions into structured instructions, such as extracting the object, action, starting point, and end point in the user's instruction intention. The task planning module 13 can predict the path of the action, starting point, and end point to obtain candidate action information.

[0280] For example, the user's instruction intention may be "move the display from area A to area B", object: display; starting point: area A; end point: area B; operation type: transport.

[0281] The task planning module 13 generates a list of candidate actions: the robotic arm grabs the display; the AGV (Automated Guided Vehicle) is dispatched to area A; the path is dynamically planned; obstacles are avoided in real time; and the display placement posture is calibrated.

[0282] Optionally, the task planning module 13 predicts the relevant probability of the candidate action information to implement the high-level instruction through the language large model to obtain the predicted probability.

[0283] For example, the task planning module 13 uses a large language model to drive decision-making technology, evaluating the relevance of each action to the core instructions. The task planning module 13 combines the human-machine collaboration rule library to eliminate low-compliance actions.

[0284] For example, robotic arm grasping as the operation of direct operation object has a correlation of 0.95; AGV scheduling as a necessary condition for transportation has a correlation of 0.93; the execution of path planning will affect transportation efficiency, with a correlation of 0.88; real-time obstacle avoidance as a safety requirement has a correlation of 0.82; manual handling backup plan is a low-compliance action, with a correlation of 0.12.

[0285] The task planning module 13 is combined with the human-machine collaboration rule library to directly exclude the "direct human handling" with low correlation.

[0286] Optionally, the task planning module 13 determines candidate subtasks with guiding significance based on the predicted probability.

[0287] Exemplarily, the task planning module 13 retains candidate actions with a relevance probability ≥ 0.8, such as robotic arm grasping, AGV scheduling, path planning, or obstacle avoidance.

[0288] Optionally, the task planning module 13 estimates the probability of the execution success rate of the candidate subtask based on the current interactive environment state data and historical data to obtain a feasibility score of the candidate subtask under the current conditions.

[0289] For example, the interactive environmental status data can be the working status of the equipment, and the historical data can be the operating status of the equipment. The task planning module 13 confirms the probability of the execution success rate of the candidate subtask through the environmental status data and historical data, and comprehensively obtains the feasibility score of the candidate subtask under the current conditions.

[0290] For example, task planning module 13 uses environmental status data to confirm the availability of the robotic arm in area A (idle status score 0.95), the sufficient battery life of the AGV (score 0.92), and the unobstructed placement space in area B (score 0.89). Task planning module 13 then uses the process database to determine if the probability of static damage to the "monitor handling" task in history is greater than 10%, and then adds an anti-static packaging subtask (feasibility score +0.15).

[0291] Optionally, the task planning module 13 determines subtask planning information based on the predicted probability and feasibility score.

[0292] For example, the task planning module 13 may combine the predicted probability and the feasibility score to perform weighted scoring to obtain a comprehensive score for each candidate subtask, arrange the priority of the candidate subtasks according to the comprehensive score, and determine the subtask planning information.

[0293] For example, the correlation accounts for 60%, the feasibility score accounts for 40%, the robotic arm grasping score is 0.95×0.6=0.57, 0.95×0.4=0.38, and the comprehensive score is 0.95; the calculation method is shown above and will not be repeated here. The results are given directly: the AGV scheduling comprehensive score is 0.93; the dynamic path planning comprehensive score is 0.89; the real-time obstacle avoidance comprehensive score is 0.83.

[0294] The task planning module 13 executes the robot arm grasping - AGV scheduling - path planning - obstacle avoidance according to the priority.

[0295] Optionally, the task planning module 13 determines a high-level task sequence based on the subtask planning information.

[0296] For example, based on temporal logic, a high-level task sequence is determined.

[0297] For example, the mission planning module 13 can generate an optimal path by scanning the site. It then dispatches the AGV to area A and controls the robotic arm to grab the display. The mission planning module 13 controls the AGV along the planned path. The mission planning module 13 activates the real-time obstacle avoidance module. The mission planning module 13 controls the robotic arm to calibrate its placement and safely release the display to area B.

[0298] Through the above-mentioned embodiments, this application can accurately screen core subtasks through intent parsing and relevance prediction based on a large language model. This application can improve the success rate of task execution by integrating a dynamic evaluation model with real-time environmental perception and historical execution data. This application can significantly improve the reliability and execution efficiency of task planning by making weighted decisions based on relevance probability and feasibility scores.

[0299] According to an example embodiment, the instruction execution module 14 generates machine control instructions according to a high-level task sequence to perform human-machine interaction according to the machine control instructions.

[0300] For example, instruction execution module 14 determines subtask execution parameters based on the subtask planning information. The subtask execution parameters may include at least target parameters, input parameters, output parameters, and execution constraints. Instruction execution module 14 establishes a mapping relationship between the application programming interface and the subtask execution parameters to obtain a machine control instruction. Instruction execution module 14 then performs human-computer interaction based on the machine control instruction.

[0301] Through the above embodiments, the present application can construct a target speech recognition model through an initial instruction set, so that the target speech recognition model can simulate various simulated speech scenarios in human-computer interaction experiments, so that the present application can more accurately identify user voice instructions to determine user text instructions. The present application uses a large language model and a preset small language model to perform intention fusion recognition on user text instructions, which can achieve dynamic adaptation of general semantic understanding and vertical field knowledge, so that the present application can accurately obtain the user instruction intention. Afterwards, the present application performs task decomposition and planning on the user instruction intention, determines the execution steps with the highest execution rate, and thus combines them to obtain a high-level task sequence. Finally, the present application generates machine control instructions through high-level task sequences and performs human-computer interaction, so that the present application can accurately perform human-computer interaction operations, thereby improving the efficiency of human-computer interaction testing.

[0302] Optionally, the instruction execution module 14 determines subtask execution parameters based on the subtask planning information. The subtask execution parameters may at least include target parameters, input parameters, output parameters, and execution constraints.

[0303] For example, target parameters may include but are not limited to navigation tasks or process tasks, input parameters may be sensor data, output parameters may be action feedback standards, and execution constraints may be physical resource limitations.

[0304] For example, when the target parameter is a navigation task, the instruction execution module 14 may define the target coordinates as the shelf coordinates of area B. When the target parameter is a process task, the instruction execution module 14 may set the clamping force threshold of the robot end to 25N±2N.

[0305] The input parameters can be configured to configure the lidar scanning frequency to 20Hz, the output parameters can be AGV speed fluctuation <0.2m / s, path deviation <10cm, and the execution constraint condition can be limiting the CPU usage of the path planning algorithm to ≤35%.

[0306] Optionally, the instruction execution module 14 establishes a mapping relationship between the application programming interface and the subtask execution parameters to obtain the machine control instruction.

[0307] Exemplarily, the instruction execution module 14 calls an execution program through an API (Application Programming Interface), executes a subtask, and obtains a machine control instruction.

[0308] Optionally, the instruction execution module 14 performs human-computer interaction based on the machine control instruction.

[0309] For example, the instruction execution module 14 may complete a clamping-moving-releasing action sequence according to the API instruction.

[0310] Through the above embodiments, this application can accelerate instruction generation by building semantically driven API mapping rules combined with resource awareness mechanisms. During the execution phase, this application implements real-time verification of output parameters and dynamic adjustment of constraints. This application achieves precise and efficient machine control through parametric modeling and dynamic interface mapping mechanisms.

[0311] According to another aspect of the present application, an electronic device is provided. The electronic device includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the human-computer interaction method described above.

[0312] According to another aspect of the present application, a non-volatile computer-readable storage medium is provided, wherein a computer program is stored on the storage medium, and when the computer program is executed by a processor, the human-computer interaction method described above can be implemented.

[0313] According to another aspect of the present application, a computer program product is provided. The computer program product includes: a computer program stored on a computer-readable storage medium; the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the human-computer interaction method described above.

[0314] Finally, it should be noted that the above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Although the present application is described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions of the aforementioned embodiments or replace some of the technical features therein with equivalents. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A human-computer interaction method based on large model assistance, characterized in that: include: Build a target speech recognition model based on the initial instruction set; recognizing a user voice instruction from a user based on the target speech recognition model, so as to determine a user text instruction according to the user voice instruction; Based on the user text instruction, the user text instruction is subjected to intent fusion recognition through a large language model and a preset small language model to obtain the user instruction intent; Performing task decomposition and planning on the user instruction intention to obtain a high-level task sequence; generating machine control instructions according to the high-level task sequence to perform human-machine interaction according to the machine control instructions; The constructing of the target speech recognition model based on the initial instruction set includes: Receive an interaction instruction set from the human-computer interaction field to construct an initial instruction set; Based on the language macro model, semantically expanding the initial instruction set to generate a synonymous instruction set; Performing speech synthesis recording on the initial instruction set and the synonymous instruction set to obtain a training speech instruction data set; Based on the training voice command data set, the preset voice recognition large model is fine-tuned by a preset model fine-tuning technology to obtain the target voice recognition model.

2. The human-computer interaction method according to claim 1, characterized in that: The step of performing intent fusion recognition on the user text instruction based on the user text instruction by using a large language model and a preset small language model to obtain the user instruction intent includes: Building the preset language small model based on a sample set containing a specific task; Based on the user text instruction, obtain first intent information according to the language model; Based on the user text instruction, obtain second intent information according to the preset language model; Perform intent fusion identification on the first intent information and the second intent information to obtain the user instruction intention.

3. The human-computer interaction method according to claim 2, characterized in that: The step of constructing the preset language model based on the sample set containing the specific task includes: Constructing the sample set containing the specific task; A machine-learned text detection model is trained based on the sample set to obtain the preset language model.

4. The human-computer interaction method according to claim 2, characterized in that: The obtaining of first intent information based on the user text instruction and the language model includes: Building a prompt template based on the user text instruction and preset interactive task information; generating guiding prompt words based on the prompt template; Inputting the guiding prompt words and the user text instruction into the language macro model for semantic understanding to obtain key elements in the user text instruction; Semantic alignment and intent recognition are performed on the guiding prompt words and the key elements to obtain the first intent information.

5. The human-computer interaction method according to claim 2, characterized in that: The obtaining of the second intent information based on the user text instruction and the preset language model includes: Processing the user text instruction to obtain intent data information; The intent data information is processed based on the preset language model to obtain the second intent information.

6. The human-computer interaction method according to claim 2, characterized in that: The performing intent fusion identification on the first intent information and the second intent information to obtain the user instruction intention includes: Determine whether the first intent information and the second intent information are consistent, If so, output the user instruction intention; If not, based on the first intention information and the second intention information, the user intention is supplemented and corrected through a multi-source information integration mechanism to obtain the user instruction intention.

7. The human-computer interaction method according to claim 1, characterized in that: The task decomposition and planning of the user instruction intention to obtain a high-level task sequence includes: Decomposing the user instruction intention to obtain at least one candidate action information; Predicting the probability of the candidate action information implementing the high-level instruction through the language macro model to obtain a predicted probability; Determining candidate subtasks with guiding significance based on the predicted probability; Based on the current interactive environment state data and historical data, the probability of success rate of execution of the candidate subtask is estimated to obtain the feasibility score of the candidate subtask under the current conditions; Determining subtask planning information based on the predicted probability and the feasibility score; The high-level task sequence is determined based on the subtask planning information.

8. The human-computer interaction method according to claim 7, characterized in that: Generating a machine control instruction according to the high-level task sequence to perform human-computer interaction according to the machine control instruction includes: Determining subtask execution parameters based on the subtask planning information; Establishing a mapping relationship between an application programming interface and the subtask execution parameters to obtain the machine control instruction; performing human-machine interaction based on the machine control instructions; The subtask execution parameters may include at least target parameters, input parameters, output parameters and execution constraints.

9. A large model-assisted human-computer interaction system, characterized in that: The human-computer interaction system executes the human-computer interaction method according to any one of claims 1 to 8, and the human-computer interaction system includes: a speech recognition module, which constructs a target speech recognition model based on the initial instruction set, and recognizes a user speech instruction from the user based on the target speech recognition model, so as to determine a user text instruction according to the user speech instruction; An intention understanding module, based on the user text instruction, performs intention fusion recognition on the user text instruction through a large language model and a preset small language model to obtain the user instruction intention; A task planning module decomposes and plans the user's instruction intention to obtain a high-level task sequence; An instruction execution module generates a machine control instruction according to the high-level task sequence, so as to execute human-computer interaction according to the machine control instruction.

Images