Man-machine interaction method and system for processing by combining keys with natural language

Abstract

The invention discloses a man-machine interaction method and system for processing by combining keys with natural languages. The invention relates to the technical field of natural language processing and keys. And detecting a function key, a shortcut key or a custom key pressed by a user, and identifying a function or operation corresponding to the key. Judging whether the identified function needs to be further processed in combination with a natural language processing module or not; and for the directly executable function (such as playing or stopping), directly executing the corresponding operation, and skipping to the step S4. If the function needs natural language processing, a voice recognition module is started, voice input of a user is captured, and a voice signal is converted into text information; a user can quickly trigger the preset function or operation of the system through simple key operation, and tedious menu navigation or a complex input process is not needed. Meanwhile, the introduction of voice input enables the user to interact with the system more naturally, and the method is especially suitable for scenes where rapid input is needed or manual input is inconvenient to use.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing and key technology, in particular to a solution combining key operation and natural language processing, and in particular to a human-computer interaction method and system combining key operation with natural language processing. Background Art

[0002] At present, natural language processing (NLP) technology has become one of the indispensable core technologies in the field of smart devices and human-computer interaction (HCI), and its application scope covers many aspects from smart homes, smart customer service to virtual assistants. Nevertheless, the traditional natural language processing paradigm mainly focuses on two major modes: voice input and text input. Although these methods have promoted the convenience of human-computer interaction to a certain extent, they still face several inherent limitations.

[0003] For example, voice input is limited by factors such as environmental noise, user accent and privacy protection; while text input is affected by issues such as user input efficiency, spelling errors and ambiguous expressions that affect the interactive experience.

[0004] In view of the problem, the present invention proposes a human-computer interaction method and system that combines keystrokes with natural language processing. Summary of the invention

[0005] In view of this, the present invention hopes to provide a human-computer interaction method and system combining key operation with natural language processing to solve or alleviate the technical problems existing in the prior art, that is, how to combine key operation and natural language processing to form a comprehensive human-computer interaction solution, and at least provide a beneficial option for this; the technical solution of the present invention is implemented as follows:

[0006] First, the human-computer interaction method combining keystrokes with natural language processing:

[0007] 1. Overview:

[0008] The present invention aims to build an efficient, flexible and user-friendly human-computer interaction mode, and realize seamless communication between users and intelligent systems by cleverly combining key operations and natural language processing technology. In the scheme, the key design covers function keys, shortcut keys and custom keys, aiming to enable users to quickly trigger preset functions or operations and improve interaction efficiency. At the same time, the introduction of the natural language processing module enables the system to accurately understand the user's voice input, identify its intentions, and generate corresponding responses. This combination not only retains the directness of traditional key interactions, but also gives the system a more powerful semantic understanding ability, thereby meeting the diverse needs of users in different scenarios. In addition, the solution also focuses on personalization and intelligence, allowing users to customize key functions and continuously optimize the interactive experience through preset rules and algorithms.

[0009] (II) Technical solution:

[0010] To achieve the above technical objectives, when receiving the user's activation command (such as pressing a preset start button, voice wake-up word or touching an icon on the screen), the following operation steps are initiated:

[0011] 2.1 Step S1, key input recognition:

[0012] When the function key, shortcut key or custom key pressed by the user is detected, the function or operation corresponding to the key is identified.

[0013] 2.1.1 Step S100, key detection:

[0014] Continuously monitor the status of the keyboard or specific input device, waiting for user key events; when it is detected that a key is pressed, capture the event and record the key code or identifier.

[0015] 2.1.2 Step S101, key recognition:

[0016] According to the captured key code or identifier, the corresponding function or operation is searched in the preset key mapping table. The key mapping table is a data structure that associates key codes with specific functions or operations. It can be hard-coded or dynamically configured.

[0017] 2.1.3 Step S102, function or operation determination:

[0018] After finding the function or operation corresponding to the key, if the key corresponds to a user interface interaction (such as opening a menu or dialog box), the corresponding interface update is triggered and then step S2 is entered.

[0019] 2.2 Step S2, function execution judgment:

[0020] For directly executable functions (such as play or stop), the corresponding operation is directly executed and the process jumps to step S4.

[0021] 2.2.1 Step S200, directly executable function determination:

[0022] Determine whether the identified function is directly executable, such as play, stop, or volume adjustment. If yes, directly execute the corresponding operation and jump to step S4; if no (such as search, send message, set reminder), start the voice recognition module and enter S201.

[0023] 2.2.2 Step S201, capturing user voice input:

[0024] Capture user voice input through an audio input device such as a microphone.

[0025] 2.2.3 Step S202: converting the voice signal into text information:

[0026] The captured voice signals are converted into text information using speech recognition technology.

[0027] 2.3 Step S3, semantic understanding and response generation:

[0028] The natural language processing module analyzes user intentions through semantic parsing and understanding, and generates corresponding response strategies according to preset rules or preset algorithms, including performing specific operations, providing information feedback and / or requesting user confirmation.

[0029] 2.3.1 Step S300, preprocessing and word segmentation:

[0030] The natural language processing module pre-processes the received text information, including removing irrelevant characters and punctuation marks; and performs word segmentation to divide the text into meaningful words or phrases.

[0031] 2.3.2 Step S301, semantic analysis:

[0032] The natural language processing module uses semantic analysis technology to parse the text after word segmentation to understand its inner meaning and contextual relationship;

[0033] Identify key information in the text, including actions, objects, and conditions, to build a complete framework of user intent.

[0034] 2.3.3 Step S302, intention recognition:

[0035] Based on the results of semantic analysis, the user's specific intention is identified through pattern matching or machine learning algorithms, that is, the operation the user wants to perform or the information he wants to obtain.

[0036] 2.3.4 Step S303, rule matching and algorithm application:

[0037] Based on the identified user intent, match the preset rules or algorithms to determine how to respond to the user request; including operation execution logic, information feedback template and / or user confirmation request.

[0038] 2.3.5 Step S304, response strategy generation:

[0039] Generate a specific response strategy based on the matching results, including:

[0040] Perform specific system operations, including querying databases and / or sending messages;

[0041] Execution information feedback, including displaying results and / or playing prompt sounds;

[0042] A request confirmation message is sent to the user, asking whether to continue the operation and / or confirm the input information.

[0043] 2.3.6 Step S305, response execution preparation:

[0044] Convert the generated response strategy into executable instructions or data format.

[0045] 2.4 Step S4, response execution and output:

[0046] Execute corresponding operations according to the generated response strategy, including updating the interface display, playing audio and / or sending messages; and present the response results to the user (through voice output, text display and / or graphical interface, etc.).

[0047] 2.4.1 Step S400, perform the following operations:

[0048] According to the response strategy, perform corresponding system operations; including:

[0049] Update interface display: refresh the data list and / or display prompt information;

[0050] Play audio: such as prompt sound and / or background music;

[0051] Send Message: Send notifications and / or requests to other users or systems.

[0052] 2.4.2 Step S401, response result preparation:

[0053] Prepare corresponding response result data according to the executed operation; including:

[0054] The text content of the voice output is converted into voice and played to the user;

[0055] The text display content is used to display to the user on the interface;

[0056] Updated data for graphical interfaces, including charts and / or images.

[0057] 2.4.3 Step S402, select output channel:

[0058] include:

[0059] Voice output, played to the user through an audio output device (such as a speaker);

[0060] Text display, displayed to users in text form on the interface;

[0061] Graphical interface, which displays data and information through charts and / or images visual elements.

[0062] 2.4.4 Step S403, execution output:

[0063] Display the prepared response result data according to the selected output method.

[0064] 2.5 Step S5, loop or end judgment:

[0065] If the task is completed and no further interaction is required, the system enters a standby state and waits for the next activation instruction; if continued interaction is required, the system returns to step S1 and prepares to receive the next input from the user.

[0066] (III) Mechanism for solving technical problems:

[0067] 3.1 Core Mechanism:

[0068] By designing function keys, shortcut keys and custom keys, an intuitive and fast interaction method is provided for users. Users can quickly trigger the system's preset functions or operations through simple key operations. At the same time, the introduction of voice recognition technology allows users to interact with the system through voice input and input verbal commands; it is especially suitable for scenarios with complex commands or requiring fast input.

[0069] The user's voice input is first converted into text information by voice recognition technology. Subsequently, the natural language processing module performs semantic analysis and understanding of the text information to identify the user's intentions and needs. Based on the user's intentions, the system combines preset rules and algorithms to generate corresponding response strategies, such as updating the interface display, playing audio, sending messages, etc.

[0070] The response results are presented to the user through multiple modes of feedback, including voice output, text display, graphical interface, etc. This multi-modal feedback method enhances the user experience and enables users to understand the system status and operation results more intuitively.

[0071] Allowing users to customize key functions improves the flexibility and personalization of the system. Users can adjust the key layout and function settings according to their own usage habits and needs.

[0072] 3.2 Principle:

[0073] Based on the physical button design at the hardware level, the button function is recognized and triggered through circuit connection and signal processing. When the user presses the button, the system detects the corresponding electrical signal change and executes the preset function or operation. In the natural language processing module, the text information is semantically parsed and understood through algorithms and models to identify the user's intentions and needs. The diversity and flexibility of human-computer interaction are achieved by using multiple perception and output channels (such as vision, sound, touch, etc.). Users can interact with the system through different input methods (such as buttons, voice, gestures, etc.), and the system conveys information and results to users through multiple modal feedback methods. This multimodal interaction method is more in line with human's natural interaction habits, improving the efficiency of interaction and user experience.

[0074] Second, the human-computer interaction system that combines buttons with natural language processing:

[0075] The system is used to implement the human-computer interaction method described above, and includes:

[0076] (1) The key input module is responsible for detecting the function keys, shortcut keys or custom keys pressed by the user: passing the key events to the system. It interacts directly with the user and provides an intuitive input method.

[0077] (2) The speech recognition module is responsible for capturing the user's voice signal and converting it into text information: it receives the start signal from the key input module or the system, starts the speech recognition process, and passes the recognition result to the natural language processing module.

[0078] (3) A natural language processing module that performs semantic analysis and understanding on the text information transmitted by the speech recognition module: Based on the user's intentions and needs, it generates a response strategy in combination with preset rules and algorithms, and passes the response strategy to the response execution module.

[0079] (4) The response execution module is responsible for executing corresponding operations according to the response strategy generated by the natural language processing module: such as updating the interface display, playing audio, sending messages, etc. It directly controls the hardware and software resources of the system to realize specific functions or services.

[0080] (5) A multimodal feedback module that provides users with feedback on operation results and system status through multiple modes (voice, text and / or graphical interface, etc.): It receives information from the response execution module and converts it into a feedback form that the user can understand.

[0081] (6) Personalization module that allows users to customize button functions: interacts with users, receives users' personalized setting requests, and modifies system configurations to meet user needs.

[0082] Compared with the prior art, the present invention has the following beneficial effects:

[0083] 1. Improve interaction efficiency: Users can quickly trigger system preset functions or operations through simple key operations, without cumbersome menu navigation or complicated input processes. At the same time, the introduction of voice input enables users to interact with the system more naturally, especially for scenarios that require fast input or are inconvenient to use manual input.

[0084] 2. Enhanced user experience: The present invention focuses on personalization and intelligence, allowing users to customize key functions, thereby improving the flexibility and personalization of the system. Users can adjust the key layout and function settings according to their own usage habits and needs, making the interaction process more in line with personal preferences. In addition, the adoption of multimodal feedback methods (such as voice output, text display, graphical interface, etc.) enhances the user experience, allowing users to more intuitively understand the system status and operation results.

[0085] 3. Improve the intelligence level of the system: The introduction of natural language processing technology enables the system to accurately understand the user's voice input and identify their intentions and needs. This greatly improves the intelligence level of the system, allowing the system to respond to user instructions more accurately and provide more precise services. At the same time, the addition of intelligent learning capabilities enables the system to continuously learn the user's operating habits and preferences, and further optimize the interactive experience and response strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0087] Figure 1 It is a schematic diagram of the method of the present invention;

[0088] Figure 2 It is a schematic diagram of the execution flow of the method of the present invention;

[0089] Figure 3 It is a schematic diagram of the system composition of the present invention. DETAILED DESCRIPTION

[0090] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below;

[0091] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0092] Explanation of relevant terms:

[0093] (1) Function keys, shortcut keys or custom keys: These are keys that users can press to trigger specific functions or operations;

[0094] (2) Function or operation: refers to the specific task or behavior that the system will perform after the key is pressed;

[0095] (3) Preset rules and preset algorithms: These are the predetermined criteria and calculation methods used by the system to interpret user intent, generate response strategies, and perform operations;

[0096] (4) Specific operation: refers to the specific action performed by the system based on user input or preset rules, such as playing music, displaying information, etc.;

[0097] (5) Providing information feedback: the process by which the system displays operation results or related information to the user;

[0098] (6) Request user confirmation: In some cases, the system requires the user to confirm information or operations to ensure accuracy;

[0099] (7) Response result: The final result or feedback displayed to the user after the system performs the operation.

[0100] Embodiment 1: Figures 1-2 As shown, this embodiment discloses an application method of a human-computer interaction method combining keystrokes with natural language processing in an online car-hailing platform. The following execution steps are included:

[0101] In this embodiment, regarding step S1: key input recognition:

[0102] Specifically, S100: key monitoring: the mobile phone APP continuously monitors the status of virtual keys and physical keys (such as volume keys) on the screen. When the user presses the "call a car" button, the APP captures the event and records the key code.

[0103] Specifically, S101: key recognition:

[0104] The APP searches for the corresponding function, "Call a car", in the preset button mapping table according to the button code. Specifically, S102: Function or operation determination:

[0105] After the "hailing a taxi" function is identified, the corresponding interface update is triggered, the taxi-hailing interface is displayed, and step S2 is entered.

[0106] In this embodiment, regarding step S2: function execution determination:

[0107] Specifically, S200: directly executable function determination: determining whether the "hailing a car" function is a directly executable type. Since the "hailing a car" function requires further input (such as a destination), it is not a directly executable function.

[0108] Specifically, S201: Capturing user voice input: activating the voice recognition module, and prompting the user to "Please state your destination." For example, the user says "Go to Beijing West Railway Station."

[0109] Specifically, S202: voice signal is converted into text information: the voice recognition technology converts the user's voice into text: "Go to Beijing West Railway Station".

[0110] In this embodiment, regarding step S3: semantic understanding and response generation:

[0111] Specifically, S300: preprocessing and word segmentation: preprocess the text "go to Beijing West Railway Station", remove irrelevant characters, and segment the words into "go" and "Beijing West Railway Station".

[0112] Specifically, S301: semantic analysis: analyzing the text after word segmentation to understand its inner meaning as "the user wants to go to Beijing West Railway Station".

[0113] Specifically, S302: Intent recognition: through pattern matching, recognize that the user's specific intention is "to set the taxi destination to Beijing West Railway Station".

[0114] Specifically, S303: rule matching and algorithm application: matching preset rules, and determining that the method of responding to the user's request is "setting the taxi call information and displaying a confirmation interface".

[0115] Specifically, S304: generating a response strategy includes:

[0116] Generate a response strategy: Update the ride-hailing interface, display the destination as "Beijing West Railway Station", and request user confirmation.

[0117] Specifically, S305: response execution preparation: converting the response strategy into executable instructions to prepare for updating interface data.

[0118] In this embodiment, regarding step S4: responding to execution and outputting specific, S400: executing operation: updating the taxi-calling interface, displaying the destination "Beijing West Railway Station" and a confirmation button.

[0119] Specifically, S401: Response result preparation: prepare response result data, including text content displayed on the interface and the style of the confirmation button.

[0120] S402: Select output channel: select text display as the output channel to display the updated interface to the user.

[0121] S403: Execution output: The prepared interface data is displayed to the user, and the user sees a confirmation interface with the destination being "Beijing West Railway Station".

[0122] In this embodiment, regarding step S5: loop or end judgment: after the user confirms that the information is correct, click the "Confirm" button, and the APP enters the next step of the ride-hailing process. If the user needs to modify the destination or perform other operations, return to step S1 and prepare to receive the user's next input.

[0123] It can be understood that the solution provided in this embodiment improves the flexibility and convenience of interaction: users can quickly trigger functions by pressing buttons and input detailed information by voice, which reduces the tediousness of manual input. Voice input makes it easy to interact with the platform while driving or when hands are busy, which improves safety. Natural language processing can accurately understand user intentions, quickly generate response strategies, and reduce user waiting time.

[0124] Key operation captures user key events, triggers preset functions or operations, and achieves quick response. Natural language processing converts user voice into text through speech recognition, and then analyzes user intentions through semantic parsing and understanding to generate corresponding response strategies. The combination of the two not only retains the intuitiveness and speed of key operation, but also utilizes the flexibility and intelligence of natural language processing, forming a comprehensive human-computer interaction solution.

[0125] Embodiment 2: This embodiment further provides a Python pseudo-code execution process framework of the solution described in Embodiment 1:

[0126]

[0127]

[0128]

[0129] In the above procedure:

[0130] Key monitoring and identification (S1): In actual applications, the mobile phone APP will continuously monitor the key events on the screen. When the user presses a key, the APP will capture the event and search for the corresponding function in the preset key mapping table according to the key code.

[0131] In this example, assume that the user presses the "Call a Car" button, and the key mapping table associates the "Call a Car" button with the "call_car" function.

[0132] Function execution judgment (S2): After identifying the "call_car" function, we determine whether it is a directly executable type. Since calling a car requires further input (such as a destination), we believe it is not directly executable.

[0133] Capturing user voice input and converting voice signals into text information (S201, S202): In actual applications, we will start the voice recognition module and prompt the user to say the destination. The user's voice will be captured and converted into text information.

[0134] Semantic understanding and response generation (S3): Preprocess, segment and parse the captured text information. Identify the user's specific intention through pattern matching or machine learning algorithms.

[0135] That is, the user's intention is identified as "setting the taxi destination to Beijing West Railway Station" through word segmentation and judgment. A response strategy is generated according to the identified intention, including the operation to be performed and related parameters (such as the destination).

[0136] Response execution and output (S4): Execute corresponding operations according to the generated response strategy, that is, update the taxi-calling interface, display the destination as "Beijing West Railway Station" and a confirmation button.

[0137] Loop or end judgment (S5): Wait for the user to confirm that the information is correct and click the "Confirm" button. If the user confirms, the APP will enter the next step of the taxi-calling process; if the user needs to modify the destination or perform other operations, it will return to step S1 to prepare for receiving the user's next input.

[0138] All the above embodiments only express the implementation methods of the relevant practical applications of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

[0139] For those skilled in the art, it can be further appreciated that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0140] At the same time, those skilled in the art can understand that all or part of the processes in all the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

Claims

1. A human-computer interaction method combining keystrokes with natural language processing, characterized in that: After receiving the activation instruction from the user, the following steps are executed: S1, after detecting the function key, shortcut key or custom key pressed by the user, the function or operation corresponding to the key is identified; S2, for directly executable functions, directly execute the corresponding operations and jump to S4; S3, analyzing user intent through semantic parsing and understanding, and generating corresponding response strategies according to preset rules or preset algorithms, including performing specific operations, providing information feedback, and / or requesting user confirmation; S4, performing corresponding operations according to the generated response strategy, including updating the interface display, playing audio and / or sending a message; And present the response result to the user.

2. The human-computer interaction method according to claim 1, characterized in that: The execution process of S1 includes: S100, continuously monitoring the state of the keyboard or a specific input device, waiting for a user key event; when it is detected that a key is pressed, capturing the event and recording the key code or identifier; S101, searching for a corresponding function or operation in a preset key mapping table according to the captured key code or identifier.

3. The human-computer interaction method according to claim 2, characterized in that: In S1, when the function or operation corresponding to the key is found, if the key corresponds to a user interface interaction, the corresponding interface update is triggered and then step S2 is entered.

4. The human-computer interaction method according to claim 1, characterized in that: The execution process of S2 includes: S200, determining whether the identified function is of a directly executable type; if so, directly executing the corresponding operation and jumping to step S4; if not, starting the voice recognition module and entering S201; S201, capturing a user's voice input through an audio input device; S202, converting the captured voice signal into text information using voice recognition technology.

5. The human-computer interaction method according to claim 1 or 4, characterized in that: The execution process of S3 includes: S300, preprocessing the received text information, including removing irrelevant characters and punctuation marks; performing word segmentation processing to divide the text into words or phrases; S301, using semantic analysis technology to parse the text after word segmentation, understand its inner meaning and contextual relationship; identify key information in the text, including actions, objects and conditions; S302, based on the result of semantic analysis, identifying the user's intention through pattern matching or machine learning algorithm; S303, matching preset rules or algorithms to determine how to respond to the user request; including operation execution logic, information feedback template and / or user confirmation request.

6. The human-computer interaction method according to claim 5, characterized in that: After completing S303, one or more of the following generation strategies are executed: Perform system operations, including querying databases and / or sending messages; Execution information feedback, including displaying results and / or playing prompt sounds; A request confirmation message is sent to the user, asking whether to continue the operation and / or confirm the input information.

7. The human-computer interaction method according to claim 6, characterized in that: The execution process of S4 includes: S400, executing corresponding system operations according to the response strategy; including: Update interface display: refresh the data list and / or display prompt information; Play audio: such as prompt sound and / or background music; Send message: send notifications and / or requests to other users or systems; S401, preparing corresponding response result data according to the executed operation; including: The text content of the voice output is converted into voice and played to the user; The text display content is used to display to the user on the interface; Updated data of graphical interfaces, including charts and / or images; S402, select output channels, including: Voice output, played to the user through the audio output device; Text display, displayed to users in text form on the interface; Graphical interface, which displays data and information through charts and / or images visual elements.

8. The human-computer interaction method according to claim 1, characterized in that: It also includes S5: if the task is completed and no further interaction is required, enter the standby state and wait for the next activation instruction; if continued interaction is required, return to S1 and prepare to receive the next input from the user.

9. A system for implementing the human-computer interaction method according to any one of claims 1 to 8, characterized in that: The system comprises: A key input module responsible for detecting the function keys, shortcut keys or custom keys pressed by the user; The speech recognition module is responsible for capturing the user's voice signal and converting it into text information; A natural language processing module that performs semantic analysis and understanding on the text information sent by the speech recognition module; A response execution module responsible for executing corresponding operations according to the response strategy generated by the natural language processing module; A multimodal feedback module that provides multimodal feedback of operation results and system status to users.

10. The system according to claim 9, characterized in that: It also includes a personalization module that allows users to customize the button functions.

Images