Mobile banking multi-mode interaction method and system based on language user interface

Abstract

The embodiment of the invention provides a mobile banking multi-mode interaction method and system based on a language user interface, and relates to the field of financial science and technology, the method comprises the following steps: obtaining input information of multiple modes input by a user, and fusing the input information of multiple modes to obtain a unified natural language instruction; inputting the unified natural language instruction into a large language model for intention classification and parameter extraction to obtain an operation intention corresponding to the unified natural language instruction and a corresponding service parameter; and executing the operation intention according to the unified natural language instruction and the service parameters. The problem that in the prior art, multi-mode input information in mobile phone banking application is mutually separated, and user intentions cannot be analyzed and understood in a collaborative mode is solved.

Description

Technical Field

[0001] This invention relates to the field of financial technology, and in particular to a multimodal interaction method and system for mobile banking based on a language user interface. Background Technology

[0002] Current mobile banking applications primarily rely on graphical user interfaces (GUIs), requiring users to manually click, swipe menus, or input text in fixed locations to complete operations. Some banking applications integrate voice assistant functionality.

[0003] In the process of developing this invention, the applicant discovered at least the following problems in the prior art: In existing mobile banking applications, multimodal input information is fragmented and cannot be collaboratively analyzed to understand user intent. Summary of the Invention

[0004] This invention provides a multimodal interaction method and system for mobile banking based on a language user interface, which at least solves the problem that multimodal input information in existing mobile banking applications is fragmented and cannot be collaboratively analyzed and understood to understand user intent.

[0005] To achieve the above objectives, in one aspect, embodiments of the present invention provide a multimodal interaction method for mobile banking based on a language user interface, comprising: The system acquires input information from multiple modalities from the user, fuses the input information from these multiple modalities, and obtains a unified natural language instruction. The unified natural language instruction is input into a large language model for intent classification and parameter extraction to obtain the operation intent and corresponding business parameters of the unified natural language instruction. The operation intent is executed according to the unified natural language instructions and the business parameters.

[0006] On the other hand, embodiments of the present invention provide a mobile banking multimodal interaction system based on a language user interface, comprising: The multimodal input to unified natural language instruction unit is used to acquire input information from multiple modalities of user input, fuse the input information from multiple modalities, and obtain unified natural language instructions. The operation intent and business parameter determination unit is used to input the unified natural language instruction into the large language model for intent classification and parameter extraction, so as to obtain the operation intent and corresponding business parameters of the unified natural language instruction. An operation intent execution unit is used to execute the operation intent according to the unified natural language instructions and the business parameters.

[0007] The above technical solution has the following beneficial effects: Under traditional technology, users need to find the relevant function item among the many function items in the multi-layer interface of the mobile banking interface. After entering the function item, they also need to fill in the relevant business parameters one by one according to different formatted interfaces. Traditional technology requires users to adapt to or explore how to use the various functions of mobile banking. In essence, it requires users to adapt to different mobile banking usage methods. The usability and experience are very poor for groups with low comprehension and operation ability, such as the elderly. In this invention, users are not required to first find the corresponding function interface or operation entry point and then input business parameters one by one according to the formatted requirements. In this invention, users interact with mobile banking in a human-like manner, directly providing the mobile banking with multiple modal input information that humans can understand, such as text, photos, images, and / or voice. The mobile banking can automatically integrate the multiple modal input information input by the user to process them collaboratively, and automatically identify the user's operation intention and corresponding business parameters, and execute the operation intention according to the business parameters. Essentially, the mobile banking proactively adapts to the user. Users only need to interact with the mobile banking as if they were a human to complete the desired operation intention. This significantly improves the user experience and enhances the usability of mobile banking for user groups with different abilities. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a flowchart of a mobile banking multimodal interaction method based on a language user interface, which is one embodiment of the present invention. Figure 2 This is a schematic diagram of the architecture of a language-based user interface-based mobile banking multimodal interaction system, which is one embodiment of the present invention. Figure 3 This is a flowchart illustrating the processing of unified natural language instructions using a large language model, as described in one embodiment of the present invention. Figure 4 This is another flowchart of a mobile banking multimodal interaction method based on a language user interface, which is one embodiment of the present invention; Figure 5 This is another flowchart of a multimodal interaction method for mobile banking based on a language user interface, which is one embodiment of the present invention; Figure 6 This is a schematic diagram of the card management platform architecture according to one embodiment of the present invention; Figure 7 This is a schematic diagram of the client-side JavaScript virtual machine structure according to one embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of a canvas component according to one embodiment of the present invention; Figure 9 This is a flowchart of a card JS calling native functions, which is one embodiment of the present invention. Detailed Implementation

[0010] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0011] On the one hand, such as Figure 1 As shown, this embodiment of the invention provides a multimodal interaction method for mobile banking based on a language user interface, including: Step S10: Obtain input information from multiple modalities of user input, fuse the input information from multiple modalities to obtain a unified natural language command; Step S11: Input the unified natural language instruction into the large language model for intent classification and parameter extraction to obtain the operation intent and corresponding business parameters corresponding to the unified natural language instruction. Step S12: Execute the operation intent according to the unified natural language instruction and the business parameters.

[0012] In some embodiments, user input of multiple modalities is completed through a unified user interaction information input interface provided by the mobile banking app. This unified user interaction information input interface allows for multi-modal information input, similar to a chat interface, enabling simple operations to input text, voice, photos, and / or images. Various operational intentions can be completed by inputting multi-modal information through this unified user interaction information input interface, eliminating the need for separate function entry points for different intentions. Users interact with the mobile banking app in a human-like manner, providing multiple modal input information, such as text, images, photos, or voice. Users can provide complementary information through input from different modalities, and there may be overlap between different modalities. For example, some specific content in the user's text input may need to be obtained from the input image information. Similarly, users can express their intentions through voice, specifying information such as account names or ID numbers mentioned in the voice input through images or text. By fusing multiple modal inputs, a unified natural language instruction (UNLI) is obtained. The UNLI describes the user's input from multiple modalities using the same modality. The UNLI is a user-understandable expression and can be either text or voice-based, preferably text-based. The large language model can understand human language and respond. When the UNLI is input, it analyzes the corresponding operational intent and business parameters. After analyzing the operational intent and parameters, the large model can call a pre-set business service corresponding to that operational intent and execute the operation according to the business parameters. In some embodiments, multiple modalities of input information (speech signals, image pixel streams, text input streams) can be collected through the hardware (microphone, camera, touchscreen) of the mobile terminal to complete the basic process of "input signal capture to format conversion (such as converting speech to PCM format, image to JPG / PNG) and then temporary storage," providing data for subsequent unified processing. The large language model in this embodiment adopts an existing open-source (e.g., based on the Transformer architecture) large language model, selected based on accuracy and parameter size. For example, the smallest parameter model in an existing open-source large language model is tested using sample data; if the accuracy reaches 90%, the model with that parameter size is considered usable. If the accuracy of the currently selected model with a parameter size is lower than 90%, a model with a larger parameter size is tested, and this model is selected when the accuracy reaches 90%. For the selected model, fine-tuning can be performed using pre-prepared labeled sample data to obtain the large language model used in this embodiment.

[0013] The embodiments of this invention have the following technical effects: In traditional technology, users need to find the relevant function item among numerous function items in the multi-layered interface of a mobile banking app. After entering the function item, they also need to fill in the relevant business parameters one by one according to different formatted interfaces. Traditional technology requires users to adapt to or explore how to use various functions of mobile banking, essentially requiring users to adapt to different mobile banking usage methods. This results in poor usability and experience for groups with lower comprehension and operational abilities, such as the elderly. In the embodiments of this invention, users interact with mobile banking in a human-like manner, directly providing the mobile banking with human-understandable text, photos, images, and / or voice information. Furthermore, users do not need to explicitly indicate the corresponding operation intent and included business parameters when inputting relevant information. The mobile banking of the embodiments of this invention can automatically fuse the multi-modal input information entered by the user, identify the user's operation intent and corresponding business parameters, and execute the operation intent according to the business parameters. Essentially, the embodiments of this invention allow mobile banking to proactively adapt to the user; users only need to interact with mobile banking like a human to complete the desired operation intent. This significantly improves the user experience and enhances the usability of mobile banking for user groups with different abilities.

[0014] Furthermore, the input information of the multiple modalities includes voice information, image information and / or first text information, as well as their respective input times; The input information from the multiple modalities is fused to obtain a unified natural language instruction, including: The speech information is converted into corresponding second text information, and the confidence level of the corresponding second text is calculated based on the sound quality of the speech information. Extract business-related third-party text information from the image information, and calculate the corresponding third-party text confidence level based on the image information's clarity; The first text information is time-stamped using the input timestamp corresponding to the first text information, the second text information is time-stamped using the input timestamp corresponding to the voice information, and the third text information is time-stamped using the input timestamp corresponding to the image information. Generate a speech information confidence assessment text for the second text information based on the second text confidence score, and use the speech information confidence assessment text to label the second text information; Generate image information confidence evaluation text for the third text information based on the third text confidence score, and use the image information confidence evaluation text to label the third text information; The unified natural language instruction is obtained by concatenating the first text information marked with the input time corresponding to the first text information, the second text information marked with the confidence evaluation text of the voice information and the input time corresponding to the voice information, and / or the third text information marked with the confidence evaluation text of the image information and the input time corresponding to the image information.

[0015] In some embodiments, for voice information, preferably, an ASR (Automatic Speech Recognition Engine) that supports dialect recognition and noise optimization (preferably, recognition rate ≥95% when signal-to-noise ratio ≥15dB) is used to convert the original voice signal into corresponding text. The ASR engine can also provide a corresponding confidence score for the converted text. For example, the confidence score is a key indicator for measuring the reliability of the ASR recognition result, usually between 0 and 1. The higher the score, the more confident the ASR engine is in the correctness of the transcription result. Preferably, structured information (such as card number and amount) from images like receipts or bank cards is extracted using OCR (Optical Character Recognition) technology. This information is then combined with object detection to locate key elements (such as card area and amount field), ultimately converting it into descriptive text (such as "bank card number 6217XXXXXXXX1234"). OCR technology also provides the confidence level of the extracted data. OCR confidence levels are typically presented as a score of 0 to 1 or a percentage of 0% to 100%. A higher score indicates that the OCR recognition result for that character, word, or text block is more reliable. For example, if a user takes a picture (photo) at 10:30:14 or selects a picture from their album or other storage, and this picture (photo) contains bank account information, and then inputs "transfer 100 to this card" via voice at 10:30:15, the input time of the corresponding information will be recorded synchronously. The voice information (voice format "Transfer 100 to this card") is converted into corresponding second text information (corresponding text format "Transfer 100 to this card"), and the corresponding second text confidence score is calculated based on the voice quality. Multiple second text confidence levels are set, and each second text confidence level corresponds to a specific voice information confidence evaluation text. For example, second text confidence score 0.95 → "The voice can be clearly recognized," second text confidence score 0.78 → "The voice can be recognized, but there may be errors," and second text confidence score 0.60 → "The voice is blurry, and the recognition result needs confirmation." For instance, a user takes a photo at 10:30:14 containing information such as: Industrial and Commercial Bank of China, card number 622200xxxxx67890, and username.Extract business-related third-party text information (e.g., ICBC, card number 622200xxxxx67890, and username) from the image information, and calculate the corresponding third-party text confidence score based on the clarity of the image information. Multiple third-party text confidence levels are set, and each level corresponds to a specific image information confidence assessment text. For example, a third-party text confidence score of 0.95 indicates "The image clearly identifies the card number," a third-party text confidence score of 0.78 indicates "The image identifies the card number, but there may be errors," and a third-party text confidence score of 0.60 indicates "The image is blurry, and the recognition result needs confirmation." The first-party text information itself is text; only the input time needs to be marked. For example, if a user inputs "Use my main account to transfer funds externally" as the first-party text information at 10:30:10, the first-party text information marked with the input time would be "Input 'Use my main account to transfer funds externally' at 10:30:10." The second text information is time-stamped using the input timestamp corresponding to the voice information, and the third text information is time-stamped using the input timestamp corresponding to the image information; a voice information confidence evaluation text for the second text information is generated based on the confidence level of the second text information, and the second text information is labeled using the voice information confidence evaluation text; an image information confidence evaluation text for the third text information is generated based on the confidence level of the third text information, and the third text information is labeled using the image information confidence evaluation text; for example, if the confidence level of the second text is 0.95, the following text is obtained: "At 10:30:15, a voice request was made to transfer 100 to this card (the voice can be clearly recognized), and at 10:30:14, an image of an ICBC bank card was taken, card number 622200xxxxx67890, username (the card number can be clearly recognized in the image)." The first text information, which is marked with the input time corresponding to the first text information, the second text information, which is marked with the confidence assessment text of the voice information and the input time corresponding to the voice information, and / or the third text information, which is marked with the confidence assessment text of the image information and the input time corresponding to the image information, are concatenated to obtain the unified natural language instruction. For example, the instruction is "At 10:30:10, the user entered 'Use my main account to transfer money' in text mode. At 10:30:15, the user made a voice request to transfer 100 to this card (the voice can be clearly recognized). At 10:30:14, the user took a picture of the Industrial and Commercial Bank of China card, card number 622200xxxxx67890, and username (the card number can be clearly recognized in the image)".

[0016] The embodiments of this invention have the following technical effects: Input information from different modalities is unified into a text format, retaining the input time of each modal input signal and the confidence level of some modalities, thus forming a unified natural language command. This command is used as input to a large language model, which is pre-trained using training data with semantic associations, information complementarity, and conflict resolution capabilities. This enables the large model to output operational intentions and business parameters. Semantic association analysis identifies cross-modal referential relationships, such as binding "this card" to the card number recognized by the image. Information complementarity is performed to supplement missing parameters with complete information. Furthermore, when there is inconsistency among multi-source data, conflict resolution is achieved by selecting data from a high-confidence source. The embodiments of this invention allow users to input multi-modal data from various sources that require coordination to complete their operational intentions in a convenient manner. After fusion, a unified natural language command is obtained and processed by the large language model.

[0017] Furthermore, such as Figure 3 As shown, the unified natural language instruction is input into a large language model for intent classification and parameter extraction to obtain the operation intent and corresponding business parameters corresponding to the unified natural language instruction, including: Construct target prompt words, which include: the unified natural language instruction, domain knowledge related to the unified natural language instruction retrieved from a preset financial domain knowledge base, and system prompt words obtained from a preset system prompt word library. The system prompt words are at least used to indicate the role of the large language model. The target prompt word is input into the large language model for semantic analysis to obtain the response text corresponding to the unified natural language instruction. The response text includes the preliminary operation intention, preliminary business parameters and corresponding answer confidence. Determine whether the confidence level of the response is greater than or equal to a preset confidence threshold; If the confidence level of the answer is greater than or equal to the preset confidence threshold, then the preliminary operation intention and the preliminary business parameters are respectively used as the operation intention and the corresponding business parameters of the unified natural language instruction; If the confidence level of the answer is less than a preset confidence threshold, a multi-round clarification dialogue is triggered to confirm the key information in the unified natural language instruction with the user, until the large language model outputs the preliminary operation intention and preliminary business parameters with the confidence level of the answer greater than the preset confidence threshold, and the obtained preliminary operation intention and preliminary business parameters are respectively used as the operation intention and corresponding business parameters of the unified natural language instruction.

[0018] In some embodiments, a pre-defined financial domain knowledge base includes financial terminology, business rules, and few-sample examples. Domain knowledge related to unified natural language instructions can be retrieved from the financial domain knowledge base using methods such as keyword matching, semantic understanding, or a hybrid strategy of keyword filtering and semantic understanding ranking. A pre-defined prompt word library contains system prompt words indicating the pre-defined role of the large language model. Target prompt words include relevant domain knowledge, system prompt words, and unified natural language instructions; inputting the target prompt words into the large language model allows it to output content that is more relevant to the professional scenarios and depth of the financial domain, improving the accuracy and relevance of the large language model's responses to the input questions. The large language model in this embodiment is fine-tuned using a large amount of sample data during training, enabling it to output corresponding operational intentions, corresponding business parameters, and response confidence levels for input unified natural language instructions. Based on the response confidence level and multiple rounds of clarification dialogue to confirm key information, the accuracy of the response is ensured, reducing the probability of erroneous operations.

[0019] The embodiments of the present invention have the following technical effects: by adding domain knowledge and system prompts to the unified natural language instructions, the large language model can use knowledge in the financial field to understand the unified natural language instructions in a targeted manner, and confirm key information based on the confidence level of the answer and multiple rounds of clarification dialogue, thereby ensuring the accuracy of the response content, reducing the probability of misoperation, and improving the accuracy of the operation intention to greater than or equal to 95%, thus avoiding financial risks caused by the operation.

[0020] Furthermore, such as Figure 5 As shown, the target prompt words also include: user profile information and historical session information; The target prompt word is input into the large language model for semantic analysis to obtain the response text corresponding to the unified natural language instruction, including: Record the unified natural language instructions in the prompt words input to the large language model each time, as well as the response text corresponding to the unified natural language instructions output by the large language model, as historical conversations; The construction of the target prompt word includes: generating user profile information based on user-related information and historical operations; taking the most recent preset number of sessions in the historical sessions as recent sessions, compressing the information of historical sessions before the most recent preset number of sessions to extract key semantic information, taking the key semantic information as early compressed sessions, and taking the recent sessions and the early compressed sessions as the historical session information; and adding the user profile information and the historical session information to the target prompt word.

[0021] In some embodiments, by recording historical sessions and using the most recent preset number of sessions as recent sessions, compressing the information of historical sessions before the most recent preset number of sessions to extract key semantic information, using the key semantic information as early compressed sessions, and using the recent sessions and the early compressed sessions as the historical session information, a short-term memory of the dialogue between the user and the large language model is constructed. For example, the content of the most recent 5-10 rounds of dialogue in the current session, including user input, system response, identified entities, business process status, etc., is used as historical session information to support the contextual coherence of the current dialogue. When the historical session content is large and exceeds the token limit of the large language model input, the information of historical sessions before the most recent preset number of sessions can be compressed to extract key semantic information, and the key semantic information can be used as early compressed sessions. The early compressed sessions contain key information from the historical sessions, achieving the effect of providing comprehensive contextual information for each round of dialogue while occupying fewer tokens, thereby reserving more available tokens for newly input unified natural language instructions. By storing persistent user information, including but not limited to historical interaction records, frequently used account information, preference settings, and / or high-frequency operations, for personalized services and intelligent recommendations across conversations, user profiles are generated based on relevant user information and historical operations, forming a long-term memory of the user. Vector database technology can be used to semantically index historical dialogues, combined with the contextual understanding capabilities of large models, to achieve referential resolution (e.g., mapping "it" or "that account" to specific entities), omission completion (automatically supplementing parameters not explicitly stated by the user), and cross-turn information association (associating relevant information from multiple rounds of dialogue). It supports dialogue interruption recovery (e.g., after a user switches topics, they can quickly return to the previous dialogue state using the "return to the previous transfer" command), and also supports multi-device synchronization (a dialogue started on a mobile device can be seamlessly continued on a tablet or other device).

[0022] Further, executing the operation intent according to the unified natural language instruction and the business parameters includes: calling a preset business service corresponding to the operation intent according to the unified natural language instruction and the business parameters, and collecting the return data of the preset business service; wherein, different operation intents correspond to different preset business services; The returned data is parsed and displayed to the user according to the business result feedback method defined within the preset business service; wherein, the business result feedback method includes text, voice or video.

[0023] In some embodiments, each operation intent corresponds to a pre-defined business service. These pre-defined business services include, but are not limited to, microservices capable of fulfilling the corresponding operation intent, a set of API interfaces providing the service, or modules providing the service. Pre-defined business services can be generated manually or automatically by the system in advance. Operation intents may include, but are not limited to, transferring funds, checking balances, purchasing financial products, applying for loans, and / or repaying credit cards. Each operation intent corresponds to its own business parameters; for example, the business parameters for a transfer include, but are not limited to, amount, account number, recipient, time, and remarks. When the large language model generates the operation intent and business parameters, and confirms that the confidence level of the response meets the threshold requirement, it automatically calls the pre-defined business service corresponding to the operation intent using unified natural language instructions and business parameters as input parameters to execute the operation intent. It collects the return data from the pre-defined business service and displays the returned data to the user according to the business result feedback method.

[0024] The embodiments of this invention have the following technical effects: By combining the output of a large language model with business services that execute operational intentions, the corresponding business services are invoked according to the operational intentions, the operational intentions are executed, and the returned data is fed back to the user in a business result feedback manner. Users only need to interact with the mobile banking app in a human-like manner through the interactive interface to complete the desired operational intentions and obtain feedback results, thereby improving the user experience and lowering the usage threshold.

[0025] Furthermore, based on the unified natural language instruction and the business parameters, a preset business service corresponding to the operation intent is invoked, and the return data of the preset business service is collected, including: For the necessary business parameters required during the execution of the preset business service corresponding to the operation intent but not in the business parameters, the constraint rule description of the necessary business parameters is obtained by querying the first preset bank interface. The constraint rule description and the semantic text of querying the necessary business parameters are used as additional prompt words. The unified natural language instruction and the additional prompt words are input to the large language model, and the necessary business parameters are obtained from the response text of the large language model. In some embodiments, some of the business parameters required to complete certain operational intentions can be obtained directly from user input information, while others need to be obtained in conjunction with data from the banking system. In such cases, the corresponding business service needs to be invoked first based on the operational intention and business parameters obtained from the user input information. During the execution of the business service, a first preset bank interface is queried to obtain the constraint rule description of the necessary business parameters. Using the constraint rule description and the semantic text of the query for the necessary business parameters as additional prompt words, the unified natural language instruction and the additional prompt words are input to the large language model. The necessary business parameters are then obtained from the response text of the large language model. For example, if the user's multimodal input information includes "query recent transfer expenditure records," the unified natural language instruction will also include this information. The operational intention output by the large language model is to query transfer records, and the business parameters include expenditures. Typically, transfer records include both expenditures and transfers. A preset business service is provided for processing queries of transfer records, which can handle queries of expenditures and transfers based on the business parameters. In this business service, it will determine that it is a query for expenditure. Then, it needs to determine the time range of the expenditure query (a required business parameter). This business parameter is currently missing. The business service will query the first preset bank interface to find that the bank's default time range for querying expenditure is the most recent 3 days. It will combine "the default time range for querying expenditure is the most recent 3 days" with the unified natural language command as a prompt word and input it into the big language model. The big language model can analyze that the user has not explicitly specified the specific time range for the query, so it will return the bank's default most recent 3 days as the time range for this expenditure query.

[0026] During the execution of the preset business service corresponding to the operation intent, if the parameter value of a certain parameter in the business parameters does not match the set of parameter values ​​of the certain parameter queried from the second preset bank interface, the parameter value of the certain parameter in the business parameters and the set of parameter values ​​of the certain parameter are sent to the large language model, so that the large language model can determine the N parameter values ​​that best match the parameter value of the certain parameter in the business parameters from the set of parameter values ​​of the certain parameter, where N is an integer greater than or equal to 0.

[0027] In some embodiments, for example, when the user's multimodal input includes "transfer 100 yuan to Li Ming," the corresponding operation intent obtained from the large language model is a transfer, and the target account included in the business parameters is Li Ming. During the execution of the corresponding business service, there are multiple target accounts in the set of previously transferred accounts queried from the second preset bank interface, one of which has the target account name "Li Ming," but none of them have the target account "Li Ming." At this time, the target account "Li Ming" in the business parameters corresponding to the operation intent and the set of target accounts queried from the bank system are both input to the large language model, and the large language model is instructed to determine the most matching parameter value through prompt words, so that the large language model returns the target account "Li Ming." The semantic understanding of the large language model is used to correct the information input by the user. The target account returned by the large language model will also be displayed to the user through other card interfaces so that the user can confirm or select the most matching target account recorded in the bank system to complete the transfer operation.

[0028] Furthermore, based on the unified natural language instruction and the business parameters, a preset business service corresponding to the operation intent is invoked, and the return data of the preset business service is collected, including: During the execution of the preset business service corresponding to the operation intent, when the execution reaches the pre-arranged interaction node in the preset business service, the pre-bound card code is obtained from the interaction node; The card package corresponding to the card code is retrieved from the card server, the card template is obtained from the card package, the card template is rendered to obtain the card interface, and the card interface is displayed to the user. In response to the user's input operation on the card interface, the card business logic code is obtained from the card package, the card business logic code is executed to process the input operation, the card processing result is obtained, and the card processing result is fed back to the interaction node of the preset business service; The interactive nodes include execution nodes that are pre-programmed in the preset business services and are required to display information to the user or to respond to user input.

[0029] In some embodiments, different cards are designed for information that needs to be displayed to the user or for scenarios where the user needs to input information on the mobile interface. Each card has an input or display component with relevant information. These components are bound to corresponding input data or display content via configuration files. Each card has its own pre-defined processing logic code. When a card is displayed or the user interacts with it, the corresponding card's processing logic code is executed to verify the validity of the data in the component or to respond to user input. Cards are centrally managed in a card server, and each card corresponds to a unique card code. During business service orchestration, relevant cards are orchestrated at the step nodes that require card usage so that the relevant cards can be called within the business service flow.

[0030] On the other hand, such as Figure 2 As shown, this embodiment of the invention provides a mobile banking multimodal interaction system based on a language user interface, including: a multimodal input to unified natural language instruction unit 200, used to acquire input information from multiple modalities input by the user, and fuse the input information from the multiple modalities to obtain a unified natural language instruction; an operation intent and business parameter determination unit 201, used to input the unified natural language instruction to a large language model for intent classification and parameter extraction, to obtain the operation intent and corresponding business parameters corresponding to the unified natural language instruction; and an operation intent execution unit 202, used to execute the operation intent according to the unified natural language instruction and the business parameters.

[0031] Further, the multimodal input information includes voice information, image information, and / or first text information, as well as their respective corresponding input times; the multimodal input to unified natural language instruction unit 200 includes: a voice-to-text module, used to convert the voice information into corresponding second text information, and calculate the corresponding second text confidence based on the voice quality of the voice information; an image information extraction module, used to extract business-related third text information from the image information, and calculate the corresponding third text confidence based on the image clarity; and a time stamping module, used to stamp the first text information with the input time corresponding to the first text information, stamp the second text information with the input time corresponding to the voice information, and stamp the second text information with the input time corresponding to the image information. The third text information; a confidence rating module, used to generate a speech information confidence assessment text for the second text information based on the second text confidence, and to use the speech information confidence assessment text to tag the second text information; to generate an image information confidence assessment text for the third text information based on the third text confidence, and to use the image information confidence assessment text to tag the third text information; a unified natural language instruction generation module, used to concatenate the first text information marked with the input time corresponding to the first text information, the second text information marked with the speech information confidence assessment text and the input time corresponding to the speech information, and / or the third text information marked with the image information confidence assessment text and the input time corresponding to the image information, to obtain the unified natural language instruction.

[0032] Further, the operation intent and business parameter determination unit 201 includes: a prompt word construction module, used to construct target prompt words, the target prompt words including: the unified natural language instruction, domain knowledge related to the unified natural language instruction retrieved from a preset financial domain knowledge base, and system prompt words obtained from a preset system prompt word library, the system prompt words being used to at least indicate the role of the large language model; a prompt word parsing module, used to input the target prompt words into the large language model for semantic analysis to obtain the response text corresponding to the unified natural language instruction, the response text including preliminary operation intent, preliminary business parameters, and corresponding response confidence; and a confidence judgment module, used to... The system determines whether the confidence level of the answer is greater than or equal to a preset confidence threshold. If the confidence level is greater than or equal to the preset confidence threshold, the preliminary operation intention and the preliminary business parameters are respectively used as the operation intention and corresponding business parameters of the unified natural language instruction. If the confidence level of the answer is less than the preset confidence threshold, a multi-round clarification dialogue is triggered to confirm the key information in the unified natural language instruction with the user until the large language model outputs the preliminary operation intention and preliminary business parameters with an answer confidence level greater than the preset confidence threshold. The obtained preliminary operation intention and preliminary business parameters are then used as the operation intention and corresponding business parameters of the unified natural language instruction.

[0033] Furthermore, the target prompt word also includes: user profile information and historical session information; the prompt word parsing module is used to record the unified natural language instructions in the prompt words input to the large language model each time, as well as the response text corresponding to the unified natural language instructions output by the large language model, as historical sessions; the prompt word construction module is configured to: generate user profile information based on relevant user information and historical operations; take the most recent preset number of sessions in the historical sessions as recent sessions, compress the information of historical sessions before the most recent preset number of sessions to extract key semantic information, take the key semantic information as early compressed sessions, take the recent sessions and the early compressed sessions as the historical session information; and add the user profile information and the historical session information to the target prompt word.

[0034] Furthermore, the operation intent execution unit 202 includes: an operation intent execution module, used to invoke a preset business service corresponding to the operation intent according to the unified natural language instruction and the business parameters, and collect the return data of the preset business service; wherein different operation intents correspond to different preset business services; and an execution result feedback module, used to parse the return data and display the return data to the user according to the business result feedback method defined in the preset business service; wherein the business result feedback method includes text, voice, or video.

[0035] Furthermore, the operation intent execution module is used to query a first preset bank interface to obtain the constraint rule description of the necessary business parameters required during the execution of the preset business service corresponding to the operation intent but not in the business parameters, use the constraint rule description and the semantic text of the query for the necessary business parameters as additional prompt words, input the unified natural language instruction and the additional prompt words to the large language model, and obtain the necessary business parameters from the response text of the large language model; and when, during the execution of the preset business service corresponding to the operation intent, the parameter value of a certain parameter in the business parameters does not match the parameter set of the certain parameter queried from the second preset bank interface, the module sends the parameter value of the certain parameter in the business parameters and the parameter set of the certain parameter to the large language model, so that the large language model determines N parameter values ​​from the parameter set of the certain parameter that best match the parameter value of the certain parameter in the business parameters, where N is an integer greater than or equal to 0.

[0036] Furthermore, the operation intent execution module is configured to: during the execution of a preset business service corresponding to the operation intent, when an interaction node in the preset business service is reached, obtain a pre-bound card code from the interaction node; retrieve a card package corresponding to the card code from the card server, obtain a card template from the card package, render the card template to obtain a card interface, and display the card interface to the user; in response to the user's input operation on the card interface, obtain card business logic code from the card package, execute the card business logic code to process the input operation, obtain a card processing result, and feed back the card processing result to the interaction node of the preset business service; the interaction node includes an execution node in the preset business service that needs to display information to the user or an execution node that needs to respond to user input.

[0037] This invention provides a mobile banking multimodal interaction system based on a language user interface, which corresponds one-to-one with the aforementioned embodiment of a mobile banking multimodal interaction method based on a language user interface. The system embodiment can be understood based on the aforementioned embodiment of a mobile banking multimodal interaction method based on a language user interface, and will not be repeated here.

[0038] The technical solutions of the present invention will be described in detail below with reference to specific application examples. For technical details not described in the implementation process, please refer to the relevant descriptions above.

[0039] This invention belongs to the interdisciplinary field of computer technology, artificial intelligence and financial technology. Specifically, it relates to an intelligent interaction method and system on a mobile terminal that uses natural language to uniformly understand and process multiple modal inputs such as voice, images and text, so as to improve the operational efficiency and user experience of mobile banking.

[0040] The inventors discovered the following problems with the existing technology: Modal fragmentation: Voice, image (e.g., photo recognition), and text input are independent, belonging to different functional entry points, making it impossible for the system to collaboratively understand and process mixed-modal instructions. For example, a user cannot simultaneously say "use this card to repay" and take a photo of their bank card. Discontinuous interaction: Existing voice assistants are mostly single-turn dialogues, unable to understand context-based continuous instructions. If voice recognition fails, the system cannot proactively supplement or correct it through other modalities (e.g., prompting the user to input text or retake the photo), leading to interruptions in the interaction process. Limitations in intent understanding: Current technologies have limited ability to understand natural language, especially struggling to handle complex instructions containing multimodal information (e.g., "deposit the amount on the check I just photographed into the account I transferred money to last month"), and cannot semantically unify the association between image information and voice / text instructions. The essence of these problems lies in the lack of a unified, language-centric interaction layer in current technologies to integrate multimodal input and deeply understand user intent. As a mainstream UI solution for mobile devices, cards, with their characteristics of "information blockization + modularization + interactivity", can not only solve the pain points of small mobile screens and complex content, but also provide a natural platform for the presentation of multimodal interaction results and the orchestration of AI dynamic services. Therefore, this embodiment of the invention uses "dynamic cards" as the core carrier for the output of interaction results, and combines them with a unified language interaction layer to form a complete solution.

[0041] The purpose of this invention is to overcome the aforementioned deficiencies of the prior art and provide a multimodal interaction method and system for AI mobile banking based on a Language User Interface (LUI). This method unifies various input methods such as voice, image, and text into machine-understandable "language UI commands," constructing a continuous interaction model of "Conversation as a Service" (CaaS). This technically solves the problems of modal fragmentation, interaction interruption, and difficulty in understanding complex intents, improving interaction efficiency, accuracy, and user experience. It also provides a "dynamic card" technical solution and a supporting dynamic card development platform, enabling one-end development and multi-end deployment. This facilitates AI banking developers in quickly presenting information returned by the intelligent agent in card format, greatly improving the overall development efficiency of AI banking projects. The core of this invention lies in constructing a multimodal command unification layer, mapping various inputs to a unified semantic representation, and having a dialogue management engine perform collaborative processing and state tracking. Figure 4The embodiments of the present invention mainly include the following steps and components: 1. Multimodal Input: Responsible for collecting raw input data (voice input, image input, text input) through the hardware (microphone, camera, touch screen) of the mobile terminal. It does not perform data processing, but completes the basic process of "from input signal capture to format conversion (such as converting voice to PCM format, and image to JPG / PNG) and then to temporary storage", providing raw data for subsequent unified processing. 2. Fusion of multimodal input information to obtain unified natural language commands: Based on multimodal input data, the "multimodal-to-text to command fusion" is completed, including: 2.1 Speech-to-text conversion from speech input: Using an ASR engine that supports dialect recognition (covering Mandarin, Cantonese, and Sichuanese) and noise optimization (recognition rate ≥95% when signal-to-noise ratio ≥15dB), the original speech signal ("Transfer 500 to this card") is converted into the corresponding text ("Transfer 500 to this card"); 2.2 Image-to-text conversion from image input: Using OCR technology to extract structured information (such as card number and amount) from images such as receipts or bank cards, combined with object detection to locate key elements (such as card area and amount field), it is finally converted into descriptive text (such as "Bank card number 6217XXXXXXXX1234"); 2.3 Command Fusion: The system aligns the timestamps of the corresponding text formats for speech, images, and text (marking their respective timestamps) to ensure the association of multimodal inputs for the same operation, and adds confidence assessment text tags (so that the large language model can allocate weights according to the recognition accuracy of each modality when analyzing unified natural language commands). These are then concatenated to generate unified natural language commands. 3. Large Language Model Analysis of Operation Intent and Business Parameters: A large language model (LLM) is used for operation intent classification and business parameter extraction. Compared to traditional pre-trained models such as BERT, it has stronger semantic understanding and generalization capabilities. Unified natural language commands are input into the large language model, and semantic association analysis is performed (e.g., binding "this card" to the card number recognized in the image). Through information complementarity (e.g., supplementing default speech parameters with corresponding information in the image), conflict resolution (when multi-source data is inconsistent, the data corresponding to the higher confidence level is used as the standard based on the confidence assessment text), and carefully designed prompt engineering, the model is guided to recognize the user's operation intent (e.g., transfer, query, wealth management, loan, etc.). It also extracts key business parameters (such as amount, account number, payee, time, etc.). The large model can understand complex, conversational expressions and handle ambiguous instructions and scenarios with multiple intentions. For financial terminology and business rules, it improves recognition accuracy through domain knowledge base enhancement and few-shot learning.When the confidence level of the model output is lower than a set threshold, the system automatically triggers multiple rounds of clarification dialogue to confirm key information with the user, ensuring that the accuracy of intent understanding reaches over 95% and avoiding financial risks caused by misoperation. 4. Determine the corresponding business service based on the operation intent, call the business service to execute the operation intent, and support multimodal output feedback on the output side, such as outputting feedback in the form of voice (TTS broadcast), text and images (dynamic cards), and video (operation guidance), and can automatically switch according to the scenario (e.g., voice output is prioritized when user movement is detected). 5. This embodiment of the invention also includes dialogue state tracking, i.e., context management, which is responsible for maintaining the short-term memory (historical conversation information) and long-term memory (user profile information) of the dialogue; Short-term memory (historical conversation information): stores the content of the most recent 5-10 rounds of dialogue in the current conversation, including user input, system response, identified entities, business process status, etc., to support the contextual coherence of the current dialogue; Long-term memory: stores persistent data such as the user's historical interaction records, frequently used account information, preference settings, and high-frequency operations, for personalized services and intelligent recommendations across conversations. This module employs vector database technology to semantically index historical dialogues, combined with the contextual understanding capabilities of a large model, to achieve referential resolution (e.g., mapping "it" or "that account" to specific entities), omission completion (automatically supplementing parameters not explicitly stated by the user), and cross-turn information association (linking relevant information across multiple turns of dialogue). It supports dialogue interruption recovery (e.g., after switching topics, users can quickly return to the previous dialogue state using the "return to the previous transfer" command), and also supports multi-device synchronization (dialogues started on a mobile device can be seamlessly continued on a tablet or other device).

[0042] 6. Business Orchestration and Development: Orchestrate corresponding preset business services for each operational intent. 6.1 Business Process Modeling and Standardization: Streamline and model the target business process, breaking down business operations into independently executable functional nodes. Each node includes input parameters, output results, execution conditions, triggering events, and contextual dependencies. The process is defined using a structured format (Domain-Specific Language (DSL)) to form a business process model that can be recognized by the system for subsequent workflow generation and execution. 6.2 Workflow Development and Assembly: After the business process model is established, the system provides a workflow development and assembly mechanism based on a visual canvas, enabling low-code and modular construction of business processes. Functional Node Component Definition: The system pre-provides multiple reusable functional node components (such as large models, conditional judgments, tool calls, message outputs, and loop processing). Each node defines input and output parameters and execution rules through metadata, possessing scalability and reusability; Slot Extraction: Based on semantic recognition and slot extraction technology using a large language model, natural language understanding is used to automatically identify and structure the entities, parameters, and contextual intents in business statements. Leveraging the semantic reasoning and contextual modeling capabilities of large-scale models, deep semantic parsing is performed on input text. This enables accurate entity recognition and slot mapping for diverse business statements without requiring a large number of manually labeled samples. Compared to traditional rule matching or sequence labeling models, it boasts higher semantic generalization ability and robustness, and can flexibly adapt to new semantic types and parameter structures in different business domains, achieving entity recognition, semantic ambiguity resolution, and dynamic slot filling under zero-sample or few-sample conditions. Through a context-related reasoning mechanism, it automatically identifies implicit logical relationships and dependencies between statements, improving the accuracy and consistency of extraction results and providing a unified data interface for process orchestration, task execution, and intelligent decision-making. Business service invocation: Supports multiple external interfaces and service invocation methods: Standardized API calls: Functional nodes in the process access internal business systems, third-party platforms, or model inference services through standardized APIs. The interface address, request parameters, and return result mapping rules can be configured. The system dynamically fills in the parameters according to the context and executes the call at runtime, realizing cross-system data interaction and task collaboration. Model Context Protocol (MCP) calls: The MCP protocol is introduced to enable large language models to directly call registered atomic services, utility functions or external APIs in a unified context. The core mechanisms include: a. Unified context format: defining a JSON format of "{model_id: model identifier, session_id: session ID, context: {identified parameters, business status}, service_list: list of callable services}" to ensure that the model and the server share the same context; b. Service call triggering rules: the model outputs a structured instruction of "{action: "call_service", service_id: "transfer", params: {amount: 500, target_account: "6217XXXX7890"}, callback_id: "cb123"}", which is automatically matched with the "transfer" service registered by the server after parsing by the MCP protocol; c. Result feedback mechanism: after the service is executed, the result is encapsulated into "{callback_id: "cb123", status: "success", data: {trade_no: "123456"}}" and returned to the model through MCP, ensuring that the call process is traceable and controllable.

[0043] Visual Canvas Assembly: Users can build business process diagrams by dragging and dropping node components, connecting logic lines, and configuring parameter properties through the system's visual canvas interface. The system automatically identifies the dependencies between nodes, generates corresponding workflow description files, and saves them in structured data formats such as JSON / YAML.

[0044] Workflow registration and publishing: After the user confirms the workflow design, it can be published with one click; the system will register the workflow description with the workflow execution engine and generate a unique workflow instance ID for subsequent execution and scheduling.

[0045] It supports dynamic business planning and autonomous process generation based on atomic services for large language models, providing three configurable decision-making modes: Reasoning-Action Loop Mode: The model operates in a closed loop of "context analysis → thought generation → action decision → result feedback → new round of reasoning". For example, if a user says "process my credit card bill", the model first considers "checking the bill amount → confirming the repayment method → ​​finally executing the repayment", then triggers "bill inquiry service → repayment method recommendation service → repayment execution service" in sequence, suitable for customer service and dynamic task collaboration scenarios; Service Direct Call Mode: The model can directly call registered atomic services (such as transfer interfaces and balance inquiry interfaces) and trigger execution through standardized parameters (such as "amount = 500 yuan, receiving account = 6217XXXX7890"). Atomic services need to complete information entry (including interface address, parameter format, and permission verification rules) through the system's preset "service registration center" to ensure the security of model calls. Plan-and-Execute Model: The model first generates a complete business plan (including objectives, steps, and dependencies) based on user needs, then executes the plan step by step and adjusts it dynamically. For example, if a user says, "Deposit 50% of my salary into a fixed deposit account and use the remainder to pay off my credit card," the model first generates a plan: "1. Check salary account balance → 2. Calculate the 50% fixed deposit amount and the outstanding credit card balance → 3. Execute the fixed deposit → 4. Execute the credit card repayment," and then calls the corresponding services sequentially. This model is suitable for multi-stage approval and cross-system workflow orchestration scenarios. Visualization and Human-Machine Collaboration Interface: The system provides a natural language interaction interface, allowing users to directly manage business processes through voice or text commands. For example, inputting "I want to transfer money" or "Process corporate transfer business," the system automatically selects and executes the corresponding workflow, while returning the result. Business Execution and Multimodal Feedback Module: Based on the parsed operation intent and business parameters, the system calls the corresponding bank backend service interfaces (such as transfer interfaces and query interfaces) to execute the operation. The execution results will be fed back to the user in the most appropriate modality based on the context and user preferences (such as voice announcement of successful transaction, on-screen display of account details with pictures and text, or pop-up interactive Q&A box requesting confirmation).

[0046] 7. Dynamic card management: such as Figure 6The diagram shows the overall architecture of the card platform, illustrating the element hierarchy (index.json) generated by the card editor, the style files (CSS), data binding (databinding), and the business logic (code) (JS). Cards are deployed to server-side storage, where the server provides internal management APIs, client-side APIs (for accessing cards), and security encryption functions to store multiple versions of card outputs. Clients download, decompress, and update cards through the UI (interface) engine, and execute the card's business logic through the JS engine, calling predefined services via the JS API. The card editor creates cards, producing card artifacts including an element hierarchy file (recording the components or elements contained in the card and their hierarchical relationships), a style file (recording the rendering styles of the components or elements in the card), a data binding file (recording the data source corresponding to the components or elements), and business logic (an executable code bound to the card, used to implement preset processing logic related to the data corresponding to the elements in the card and the corresponding user input). The resulting cards are saved on the server. When a business service needs to use a card, the business service running on the backend server sends the required card code to the client. The client then retrieves and downloads the card artifacts from the card server, renders and displays the card on the client using the UI engine, and executes the corresponding business logic through the JS engine and JSAPI interface. A complete and mature online card platform is provided, along with SDKs for iOS, Android, and HarmonyOS clients. After integration, developers can develop new AI banking scenarios without or with minimal intervention in native client development, and card rendering performance is consistent with native rendering, achieving cost reduction and efficiency improvement. Specific technical features are as follows: Online Card Platform: Through the online card development platform, developers can quickly and easily build cards and connect them to intelligent agent workflow orchestration; preview cards by scanning QR codes via a mobile app to view the actual effects on various terminals in real time; and support one-click online publishing after development, improving development efficiency. Card Hot Update: When a card needs to be updated, developers can modify the UI and business code in the online card editor and click publish. The platform automatically generates an element hierarchy (index.json, defining the view tree structure, such as...) <card> <text id="amount"> 500 yuan< / text> < / card> The core technical solution comprises a compressed package (card package) containing the following elements: styles (index.css, supporting native CSS syntax), data binding (index.databinding, defining the mapping relationship between text and business parameters, such as amount → transferAmount), and business logic (index.js). The client updates the latest card product by reading the template file to achieve hot updates. The core technical solution includes: a client-side rendering engine: reading the view hierarchy files and CSS style files from the card product, and mapping them via "view node parsing → native component mapping (such as JSON)". <text>The process of mapping to iOS's UILabel and Android's TextView, followed by style attribute assignment (converting the font-size property of index.css to the font size property of the native control), transforms non-native templates into UI components with performance consistent with native controls (frame rate ≥ 60fps); client-side business engine: such as Figure 7 As shown, this demonstrates the cross-platform hierarchy of "iOS (Objective-C / Swift) / Android (Java / Kotlin) / HarmonyOS (ArkTS) native layer → C++ intermediate layer → QuickJS virtual machine layer"; it integrates the QuickJS 2024.01 version virtual machine (using the "-O2" optimization level during cross-platform compilation, with a memory limit of 512MB) as a JS container to run the JS files in the card artifacts to execute business logic; it supports JS calling native functions through "FlameJSBridge"—when JS calls native functions, the parameters are first converted to JSON format, and then mapped to native methods through the C++ intermediate layer (e.g., JS's alertEx() is mapped to iOS's UIAlertController, Android's AlertDialog), and after native execution, the result is returned to JS through a callback function, achieving cross-platform (iOS / Android / HarmonyOS) functional compatibility. Canvas components: such as Figure 8 As shown, this demonstrates the component rendering process: "Client Environment → JavaScript Virtual Machine → Canvas Drawing Code → Context2d Proxy → Native Canvas View". It provides a canvas component to meet the needs of various UI design drafts, avoiding frequent SDK updates for the client. The canvas component is developed based on standard JavaScriptCanvas 2D syntax. The client achieves cross-platform compatibility through the Context2d proxy layer—the proxy layer converts Canvas drawing instructions (such as path, arc, text) into native drawing interfaces for various platforms (Core Graphics for iOS, Canvas API for Android, and ArkUI Canvas for HarmonyOS), supporting custom resolution adaptation (automatically adapting to the phone's screen DPI), facilitating front-end development. For example... Figure 9 As shown in the flowchart, when the card's JavaScript calls native functions, the JavaScript code is triggered either when the client loads the card or by the user clicking. First, this code is forwarded to the delegate method in the C++ environment for execution. The delegate method validates the JavaScript code; if valid, it converts the code into an object recognizable by the JavaScript virtual machine environment, which then executes it. Due to the characteristics of the JavaScript language, when code like `ynet.callJSAPI` is executed, the JavaScript virtual machine forwards this line of code to the C++ delegate method for execution. This is the reverse of the previous steps; in this case, C++ converts the JavaScript code into native code for execution, ultimately calling the native function.

[0047] Technical Solution Implementation Principle: By uniformly converting all inputs (including non-verbal images) into language commands, this invention creates a unified, language-based interaction interface at the logic layer, breaking down the technical barriers between different modalities. This enables the system to understand and process any form of user input in a consistent manner and leverage the rich contextual expressive power of language to achieve a continuous and fluent "dialogue" experience, thereby technically solving the problems of modal fragmentation and interaction interruption. The embodiments of this invention have the following technical effects: significantly improved interaction efficiency: In actual testing with 1200 test users (aged 18-75, including 20% ​​elderly users) and 5 mainstream mobile phone models (iPhone 13 / 14, Huawei Mate 50 / P60, Xiaomi 13), the average number of operation steps to complete three common tasks—transfer, balance inquiry, and credit card repayment—was reduced by 52%. The transfer process was reduced from 6 steps in the traditional GUI to 3 steps. The traditional 6 steps for a transfer are: open the mobile banking app, click the "transfer" function entry; select the transfer type (such as intra-bank transfer, inter-bank transfer, mobile number transfer, etc.); enter the recipient information (account number, name, bank, or select by mobile number / contact person); enter the transfer amount; enter the transaction password (or perform biometric recognition such as fingerprint or face recognition); verify the transfer information and submit, and wait for the system to provide feedback. The three steps of multimodal interaction in this invention embodiment are: initiating a multimodal command (e.g., saying "Transfer XXXX yuan to XXX" via voice, while simultaneously displaying images such as the recipient's business card and account screenshot); the system automatically recognizes the command and generates a dynamic transfer confirmation card (displaying information such as the recipient, amount, and arrival time); after the user confirms the card information, the payment is completed via biometrics or password, and the process ends, reducing the average operation time by 41% (from 45 seconds to 27 seconds); significantly improving recognition accuracy and fault tolerance: in a 65-decibel noise environment (simulating shopping mall background noise), 10 In a test of 00 mixed-modal commands (60% voice + image, 20% pure voice, and 20% pure text), the overall command recognition accuracy improved from 90% for single voice recognition to 98%, while the false recognition rate decreased by 87%. This enhanced system functionality and user experience: it can handle unprecedentedly complex commands (such as "Use the account number on the cover of the check I just photographed to transfer an equal amount to this contact (show image)"), expanding the technological boundaries of mobile banking. It also lowered the barrier to entry: providing a more intuitive interaction method, especially convenient for elderly or visually impaired users unfamiliar with GUI operations, thus improving the accessibility of financial services.

[0048] The following is a specific embodiment: A smartphone (such as an iPhone 13 or an equivalent Android device) equipped with a microphone, camera, and touchscreen is installed with a mobile banking app integrating this embodiment of the invention. The backend communicates with the bank server via HTTPS protocol. A transfer operation combining image recognition is executed. User input: The user clicks the voice button and says, "Transfer 500 yuan to the person on this business card." Simultaneously, the user uses the app's built-in camera to take a picture of the recipient's business card (including name and bank card number). Multimodal command unification: The voice channel uses an ASR engine supporting dialect recognition and noise optimization to convert speech into text: "Transfer 500 yuan to the person on this business card."; The image channel uses an OCR engine supporting ID / receipt recognition (character recognition accuracy ≥98%, supports images with tilt angles ≤30°) to process the business card image and extract two key slots: Name = Li Si, Card Number = 6217XXXX7890; Command fusion: Integrating the voice and image information to generate a semantically rich unified natural language command: "Transfer 500 yuan to the person on this business card, Name = Li Si, Card Number = 6217XXXX7890"; Natural language understanding and dialogue management: The large language model recognizes the intent as "transfer" and fills the slots: Amount = 500 yuan, Recipient Name = Li Si, Recipient Card Number = 6217XXXX7890; The dialogue state tracker records the context of the current transfer operation (such as the current operation type and identified payment information). Business execution and feedback: The system calls the business service corresponding to the transfer, executes the transfer process, and sends an instruction to the client: display a "Transfer Confirmation" card with the parameters "Amount = 500 yuan, Recipient Name = Li Si, Recipient Card Number = 6217XXXX7890"; The client card SDK calls the rendering engine to generate a native UI based on the card name and parameters, and displays the "Transfer Confirmation" card; The system automatically announces: "500 yuan will be transferred to Li Si's account ending in 7890. Please confirm."; The user replies with "Confirm"; The system calls the transfer API through the MCP protocol to execute the transaction. After the transaction is successful, the backend returns the "Transfer Successful" card information; After receiving the information, the client displays the "Transfer Successful" card and announces: "Transfer Successful".

[0049] Some embodiments of the present invention provide a multimodal interaction method for AI mobile banking based on a Language User Interface (LUI), including: receiving multiple modal input information from a user, the multiple modal input information including voice input, image input, and / or text input (first text information); and uniformly converting the multiple modal input information into a unified natural language command, specifically including: converting voice input into second text information and corresponding voice information confidence assessment text using ASR technology (ASR technology can convert voice into text and output corresponding confidence values; in this embodiment of the invention, the corresponding confidence value can be converted into corresponding voice information confidence assessment text); and converting image input into third text information containing structured information and corresponding image information confidence assessment text using OCR and object detection technology (OCR and object detection). R and object detection technology can extract key text information of interest from images and output corresponding confidence values. In this embodiment of the invention, the corresponding confidence values ​​can be converted into corresponding image information confidence evaluation text. The first text information, the second text information, and / or the third text information are timestamped and corresponding voice information confidence evaluation text and image information confidence evaluation text are added to generate a unified natural language instruction. Based on the unified natural language instruction, the operation intention and business parameters are determined, specifically including: using a large language model combined with a financial domain knowledge base to identify the user's operation intention (such as transfer, query) and extract (key) business parameters. If the confidence of the intention identification is lower than a preset confidence threshold, multiple rounds of clarification dialogue are triggered; a dialogue state tracker is maintained: the current conversation content is stored in short-term memory for 5-10 rounds, and the user's historical interaction records are stored in long-term memory to achieve context-based continuous interaction; the banking business operation corresponding to the user's intention is executed, and the execution result is fed back according to the user's preference by voice, text, or interactive question and answer method. In some embodiments, the dialogue state tracker uses a vector database to semantically index historical dialogues, supporting referential resolution (e.g., mapping "that account" to a specific card number), omission completion (automatically supplementing parameters not explicitly specified by the user), and multi-device dialogue synchronization. In some embodiments, the graphic feedback method is implemented through dynamic cards, which are generated by an online development platform. The client uses a rendering engine to convert the card template into a native interface and executes business logic through a JS engine, supporting hot updates and eliminating the need for native client development.

[0050] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.

[0051] In the above detailed description, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features of the single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, wherein each claim stands alone as a preferred embodiment of the invention.

[0052] The disclosed embodiments have been described above to enable any person skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit and scope of this disclosure. Therefore, this disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0053] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations falling within the scope of the appended claims. Furthermore, the term "comprising" as used in the specification or claims is used in a manner similar to the term "including." Additionally, the use of any term "or" in the specification of the claims is intended to mean "non-exclusive or." Those skilled in the art will also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of both. To clearly demonstrate the interchangeability of hardware and software, the functions of the various illustrative components, units, and steps described above have been generally described. Whether such functionality is implemented by hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of the present invention. The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.< / text>

Claims

1. A multimodal interaction method for mobile banking based on a language user interface, characterized in that, include: The system acquires input information from multiple modalities from the user, fuses the input information from these multiple modalities, and obtains a unified natural language instruction. The unified natural language instruction is input into a large language model for intent classification and parameter extraction to obtain the operation intent and corresponding business parameters of the unified natural language instruction. The operation intent is executed according to the unified natural language instructions and the business parameters.

2. The mobile banking multimodal interaction method based on a language user interface as described in claim 1, characterized in that, The input information of the multiple modalities includes voice information, image information and / or first text information, as well as the corresponding input time for each; The input information from the multiple modalities is fused to obtain a unified natural language instruction, including: The speech information is converted into corresponding second text information, and the confidence level of the corresponding second text is calculated based on the sound quality of the speech information. Extract business-related third-party text information from the image information, and calculate the corresponding third-party text confidence level based on the image information's clarity; The first text information is time-stamped using the input timestamp corresponding to the first text information, the second text information is time-stamped using the input timestamp corresponding to the voice information, and the third text information is time-stamped using the input timestamp corresponding to the image information. Generate a speech information confidence assessment text for the second text information based on the second text confidence score, and use the speech information confidence assessment text to label the second text information; Generate image information confidence evaluation text for the third text information based on the third text confidence score, and use the image information confidence evaluation text to label the third text information; The unified natural language instruction is obtained by concatenating the first text information marked with the input time corresponding to the first text information, the second text information marked with the confidence evaluation text of the voice information and the input time corresponding to the voice information, and / or the third text information marked with the confidence evaluation text of the image information and the input time corresponding to the image information.

3. The mobile banking multimodal interaction method based on a language user interface as described in claim 1, characterized in that, The unified natural language command is input into a large language model for intent classification and parameter extraction to obtain the operation intent and corresponding business parameters corresponding to the unified natural language command, including: Construct target prompt words, which include: the unified natural language instruction, domain knowledge related to the unified natural language instruction retrieved from a preset financial domain knowledge base, and system prompt words obtained from a preset system prompt word library. The system prompt words are at least used to indicate the role of the large language model. The target prompt word is input into the large language model for semantic analysis to obtain the response text corresponding to the unified natural language instruction. The response text includes the preliminary operation intention, preliminary business parameters and corresponding answer confidence. Determine whether the confidence level of the response is greater than or equal to a preset confidence threshold; If the confidence level of the answer is greater than or equal to the preset confidence threshold, then the preliminary operation intention and the preliminary business parameters are respectively used as the operation intention and the corresponding business parameters of the unified natural language instruction; If the confidence level of the answer is less than a preset confidence threshold, a multi-round clarification dialogue is triggered to confirm the key information in the unified natural language instruction with the user, until the large language model outputs the preliminary operation intention and preliminary business parameters with the confidence level of the answer greater than the preset confidence threshold, and the obtained preliminary operation intention and preliminary business parameters are respectively used as the operation intention and corresponding business parameters of the unified natural language instruction.

4. The mobile banking multimodal interaction method based on a language user interface as described in claim 3, characterized in that, The target prompts also include: user profile information and historical session information; The target prompt word is input into the large language model for semantic analysis to obtain the response text corresponding to the unified natural language instruction, including: Record the unified natural language instructions in the prompt words input to the large language model each time, as well as the response text corresponding to the unified natural language instructions output by the large language model, as historical conversations; The constructed target prompt words include: Generate user profile information based on user-related information and historical operations; The most recent preset number of sessions in the historical sessions are taken as recent sessions. The historical sessions before the most recent preset number of sessions are compressed to extract key semantic information. The key semantic information is taken as early compressed sessions. The recent sessions and the early compressed sessions are taken as the historical session information. Add the user profile information and the historical session information to the target prompt words.

5. The mobile banking multimodal interaction method based on a language user interface as described in claim 1, characterized in that, Executing the operation intent according to the unified natural language instructions and the business parameters includes: Based on the unified natural language instruction and the business parameters, the preset business service corresponding to the operation intent is invoked, and the return data of the preset business service is collected; wherein, different operation intents correspond to different preset business services; The returned data is parsed and displayed to the user according to the business result feedback method defined within the preset business service; wherein, the business result feedback method includes text, voice or video.

6. The mobile banking multimodal interaction method based on a language user interface as described in claim 5, characterized in that, Based on the unified natural language instruction and the business parameters, a preset business service corresponding to the operation intent is invoked, and the return data of the preset business service is collected, including: For the necessary business parameters required during the execution of the preset business service corresponding to the operation intent but not in the business parameters, the constraint rule description of the necessary business parameters is obtained by querying the first preset bank interface. The constraint rule description and the semantic text of querying the necessary business parameters are used as additional prompt words. The unified natural language instruction and the additional prompt words are input to the large language model, and the necessary business parameters are obtained from the response text of the large language model. During the execution of the preset business service corresponding to the operation intent, if the parameter value of a certain parameter in the business parameters does not match the set of parameter values ​​of the certain parameter queried from the second preset bank interface, the parameter value of the certain parameter in the business parameters and the set of parameter values ​​of the certain parameter are sent to the large language model, so that the large language model can determine the N parameter values ​​that best match the parameter value of the certain parameter in the business parameters from the set of parameter values ​​of the certain parameter, where N is an integer greater than or equal to 0.

7. The mobile banking multimodal interaction method based on a language user interface as described in claim 5, characterized in that, Based on the unified natural language instruction and the business parameters, a preset business service corresponding to the operation intent is invoked, and the return data of the preset business service is collected, including: During the execution of the preset business service corresponding to the operation intent, when the execution reaches the pre-arranged interaction node in the preset business service, the pre-bound card code is obtained from the interaction node; The card package corresponding to the card code is retrieved from the card server, the card template is obtained from the card package, the card template is rendered to obtain the card interface, and the card interface is displayed to the user. In response to the user's input operation on the card interface, the card business logic code is obtained from the card package, the card business logic code is executed to process the input operation, the card processing result is obtained, and the card processing result is fed back to the interaction node of the preset business service; The interactive nodes include execution nodes that are pre-programmed in the preset business services and are required to display information to the user or to respond to user input.

8. A mobile banking multimodal interaction system based on a language user interface, characterized in that, include: The multimodal input to unified natural language instruction unit is used to acquire input information from multiple modalities of user input, fuse the input information from multiple modalities, and obtain unified natural language instructions. The operation intent and business parameter determination unit is used to input the unified natural language instruction into the large language model for intent classification and parameter extraction, so as to obtain the operation intent and corresponding business parameters of the unified natural language instruction. An operation intent execution unit is used to execute the operation intent according to the unified natural language instructions and the business parameters.

9. The mobile banking multimodal interaction system based on a language user interface as described in claim 8, characterized in that, The input information of the multiple modalities includes voice information, image information and / or first text information, as well as the corresponding input time for each; The multimodal input-to-unified natural language instruction unit includes: The speech-to-text module is used to convert the speech information into corresponding second text information and calculate the confidence level of the corresponding second text based on the sound quality of the speech information. The image information extraction module is used to extract business-related third-party text information from the image information and calculate the corresponding third-party text confidence level based on the clarity of the image information. The time stamping module is used to stamp the first text information using the input time corresponding to the first text information, to stamp the second text information using the input time corresponding to the voice information, and to stamp the third text information using the input time corresponding to the image information; A confidence rating module is used to generate a speech information confidence rating evaluation text for the second text information based on the second text confidence rating, and to use the speech information confidence rating evaluation text to tag the second text information; and to generate an image information confidence rating evaluation text for the third text information based on the third text confidence rating, and to use the image information confidence rating evaluation text to tag the third text information. The unified natural language instruction generation module is used to concatenate first text information marked with the input time corresponding to the first text information, second text information marked with the confidence evaluation text of the speech information and the input time corresponding to the speech information, and / or third text information marked with the confidence evaluation text of the image information and the input time corresponding to the image information to obtain the unified natural language instruction.

10. The mobile banking multimodal interaction system based on a language user interface as described in claim 8, characterized in that, The operation intent and business parameter determination unit includes: The prompt word construction module is used to construct target prompt words, which include: the unified natural language instruction, domain knowledge related to the unified natural language instruction retrieved from a preset financial domain knowledge base, and system prompt words obtained from a preset system prompt word library. The system prompt words are at least used to indicate the role of the large language model. The prompt word parsing module is used to input the target prompt word into the large language model for semantic analysis to obtain the response text corresponding to the unified natural language instruction. The response text includes the preliminary operation intention, preliminary business parameters and corresponding answer confidence. The confidence level determination module is used to determine whether the confidence level of the answer is greater than or equal to a preset confidence level threshold. If the confidence level of the answer is greater than or equal to the preset confidence level threshold, the preliminary operation intention and the preliminary business parameters are respectively used as the operation intention and corresponding business parameters of the unified natural language instruction. If the confidence level of the answer is less than the preset confidence level threshold, a multi-round clarification dialogue is triggered to confirm the key information in the unified natural language instruction with the user until the large language model outputs the preliminary operation intention and preliminary business parameters with an answer confidence level greater than the preset confidence level threshold, and the obtained preliminary operation intention and preliminary business parameters are respectively used as the operation intention and corresponding business parameters of the unified natural language instruction.

Images