Multi-modal information display method and device of application program, equipment and storage medium

Abstract

The invention provides a multi-modal information display method and device of an application program, equipment and a storage medium, and relates to the technical field of computers. The method comprises the steps that current view information of an application program is obtained, and a user inputs a user operation sequence in a target service scene based on the current view information; determining user demand perception of the user based on the current view information and the user operation sequence; determining a user intention sequence based on the user demand perception, the unified view information of the application program and the user panoramic information of the user; and controlling the application program to display multi-modal information based on the user intention sequence, wherein the multi-modal information is used for the user to execute the business in the target business scene. According to the technical scheme provided by the invention, the application program can be intelligently controlled to display the multi-modal information based on the perceived user intention, so that the user can execute the business in the target business scene, and the business handling efficiency is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, device and storage medium for displaying multimodal information of an application. Background Art

[0002] At present, the applications (Application, APP) in the operator's mobile terminal mainly provide main business services to users in the form of page distribution, such as traditional inquiry services, payment services, processing services, etc., as well as online sales of traffic packages, package services, terminal services, rights and interests services, etc., as well as exploration services involving various content information around life, food, clothing, housing, transportation and other related services.

[0003] Generally, when an application provides services to users, users can use two main methods to complete online services. One method is to operate through the preset function page in the application, and guide users step by step to complete the required services or order goods according to the page. Another method is to enter the service keyword through the search bar at the top of the homepage, find the service you want to handle, and click on the service page to handle it.

[0004] However, when using the above-mentioned approach to implement online business processing, it usually takes multiple steps to complete a simple business processing. The operation process is cumbersome and it is not easy to find the required function page, resulting in low business processing efficiency. Therefore, how to intelligently display relevant business information to users to improve business processing efficiency is a technical problem that needs to be solved by those skilled in the art. Summary of the invention

[0005] The present application provides a method, apparatus, device and storage medium for displaying multimodal information of an application, which is used to solve the defects in the prior art of how to intelligently display relevant business information to users to improve the efficiency of business handling. It can intelligently control the application to display multimodal information based on the perceived user intention, which is used for users to perform business in the target business scenario, thereby effectively improving the efficiency of business handling.

[0006] The present application provides a multimodal information display method for an application, comprising: Acquire current view information of the application, and a user operation sequence input by the user in a target business scenario based on the current view information; Determining a user demand perception of the user based on the current view information and the user operation sequence; Determine a user intention sequence based on the user demand perception, the unified view information of the application and the user panoramic information of the user; wherein the unified view information is used to summarize and display the page views corresponding to all functional points in the target business scenario, and the user panoramic information is used to characterize the characteristic information and behavior information of the user; Based on the user intention sequence, the application is controlled to display multimodal information, and the multimodal information is used for the user to perform the business in the target business scenario.

[0007] According to a multimodal information display method of an application provided by the present application, determining the user demand perception of the user based on the current view information and the user operation sequence includes: Performing multimodal fusion on different modal information included in the current view information to obtain multimodal view fusion perception information; Based on the behavior sequence in the time dimension and the interaction context sequence in the space dimension included in the user operation sequence, obtaining spatiotemporal sequence behavior perception information; Based on the multimodal view fusion perception information and the spatiotemporal sequence behavior perception information, a user demand perception of the user is determined.

[0008] According to a multimodal information display method of an application provided by the present application, determining a user intention sequence based on the user demand perception, the unified view information of the application and the user panoramic information of the user includes: Determining target potential intention information of the user based on the user demand perception, the unified view information of the application and the user panoramic information of the user; Based on the target potential intent information and a preset intent sequence template, the user intent sequence is generated; wherein the intent sequence template is used to characterize the mapping relationship between the potential intent information and the user intent.

[0009] According to a multimodal information display method of an application provided by the present application, controlling the application to display multimodal information based on the user intention sequence includes: For each user intention in the user intention sequence, based on the weights corresponding to the user intention under different classification dimensions and the time when the user intention is perceived, determining an interest score corresponding to the user intention; Sorting the user intentions in the user intention sequence in descending order of interest scores to obtain a target user intention sequence; The application is controlled to display multimodal information based on the target user intention sequence.

[0010] According to a multimodal information display method of an application provided by the present application, controlling the application to display multimodal information based on the target user intention sequence includes: Based on the target user intention sequence and the target business scenario, generating a potential interaction strategy matrix; Analyze the potential interaction strategy matrix based on the big model to generate multiple executable paths corresponding to the business in the target business scenario, and predict the feasibility and execution efficiency of each executable path based on the big model to obtain the prediction results of each executable path; Based on the prediction results of each of the executable paths, the application is controlled to display multimodal information.

[0011] According to a multimodal information display method of an application provided by the present application, the potential interaction strategy matrix is ​​analyzed based on a large model to generate multiple executable paths corresponding to the business in the target business scenario, including: Obtaining global view information, specific domain knowledge information, and specific skill execution capability information of the business in the target business scenario; The global view information, the specific domain knowledge information, the specific skill execution capability information and the potential interaction strategy matrix are input into the large model to generate the multiple executable paths.

[0012] According to a method for displaying multimodal information of an application provided by the present application, controlling the application to display multimodal information based on the prediction results of each executable path includes: Determine the executable path corresponding to the optimal prediction result as the target executable path; The multimodal instruction is generated based on the target executable path, and the multimodal instruction is used to control the application to display the multimodal information based on the independent transmission channels corresponding to the multimodal information.

[0013] The present application also provides a multimodal information display device for an application, comprising: An acquisition unit, configured to acquire current view information of an application program and a user operation sequence input by a user in a target business scenario based on the current view information; A first processing unit, configured to determine a user demand perception of the user based on the current view information and the user operation sequence; a second processing unit, configured to determine a user intention sequence based on the user demand perception, the unified view information of the application, and the user panoramic information of the user; wherein the unified view information is used to summarize and display the page views corresponding to all functional points in the target business scenario, and the user panoramic information is used to characterize the characteristic information and behavior information of the user; A control unit is used to control the application to display multimodal information based on the user intention sequence, and the multimodal information is used for the user to perform the business in the target business scenario.

[0014] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a multimodal information display method of an application as described in any one of the above is implemented.

[0015] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a multimodal information display method for an application as described in any one of the above.

[0016] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the multimodal information display method of any of the above-mentioned applications.

[0017] The present application provides a method, device, equipment and storage medium for displaying multimodal information of an application, which obtains the current view information of the application and the user operation sequence input by the user in the target business scenario based on the current view information; determines the user's user demand perception based on the current view information and the user operation sequence; and determines the user intention sequence based on the user demand perception, the unified view information of the application and the user's user panoramic information; and then controls the application to display multimodal information based on the user intention sequence, and the multimodal information is used for the user to perform the business in the target business scenario. In this way, the user intention sequence is determined based on the user demand perception, the unified view information of the application and the user's user panoramic information, so that the application can be intelligently controlled to display multimodal information based on the perceived user intention, which is used for the user to perform the business in the target business scenario, thereby effectively improving the business handling efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A flowchart of a multimodal information display method for an application provided in an embodiment of the present application.

[0020] Figure 2 A flowchart of a method for controlling an application to display multimodal information based on a user intent sequence is provided in an embodiment of the present application.

[0021] Figure 3 A schematic diagram of a framework for generating a potential interaction strategy matrix provided in an embodiment of the present application.

[0022] Figure 4 A schematic diagram of the structure of an external system provided in an embodiment of the present application.

[0023] Figure 5 A schematic diagram of a framework for generating multimodal instructions provided in an embodiment of the present application.

[0024] Figure 6 A schematic diagram of a framework for executing multimodal instructions provided in an embodiment of the present application.

[0025] Figure 7 A schematic diagram of a framework for displaying multimodal information provided in an embodiment of the present application.

[0026] Figure 8 A schematic diagram of the structure of a multimodal information display device for an application provided in an embodiment of the present application.

[0027] Fig. 9 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] In the embodiments of the present application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In the text description of the present application, the character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0030] The technical solution provided by the embodiment of the present application can be adapted to the business handling scenario. Taking the online business handling scenario as an example, under normal circumstances, when providing business handling to users through an application, users mainly use two ways to realize online handling of business. Among them, one way is to operate through the preset function page in the application, and guide the user step by step to complete the handling of the required business or the ordering of goods according to the page. Another way is to enter the business keyword through the search bar at the top of the homepage, find the business you want to handle, and click on the business page to handle it.

[0031] However, when using the above-mentioned approach to implement online business processing, it usually takes multiple steps to complete a simple business processing. The operation process is cumbersome and it is impossible to easily find the required function page, resulting in low business processing efficiency.

[0032] Therefore, in order to intelligently display relevant business information to users, thereby improving the efficiency of business processing, an embodiment of the present application provides a multimodal information display method for an application, which can determine the user's user demand perception based on the current view information of the application and the user operation sequence input by the user in the target business scenario based on the current view information; and determine the user intention sequence based on the user demand perception, the unified view information of the application and the user's user panoramic information; and then control the application to display multimodal information based on the user intention sequence. In this way, by determining the user's user intention sequence, the application can be intelligently controlled to display multimodal information based on the perceived user intention, which is used for the user to perform business in the target business scenario, thereby effectively improving the efficiency of business processing.

[0033] It can be understood that the executor of the present method can be an electronic device such as a computer or a server, or it can be a multimodal information display device of an application program installed in the electronic device. The multimodal information display device of the application program can be implemented by software, hardware, or a combination of both.

[0034] The following specific embodiments will be used to describe the multimodal information display method of the application provided by the present application in detail. It is understandable that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0035] Figure 1 A flowchart of a multimodal information display method for an application provided in an embodiment of the present application is provided. For example, see Figure 1 As shown, the multimodal information display method of the application may include: S101: Acquire current view information of an application program, and a user operation sequence input by a user in a target business scenario based on the current view information.

[0036] When the user enters the front-end page from the application, the current view information can be obtained, as well as the user operation sequence in the target business scenario input by the user based on the current view information, providing the prerequisite for subsequent perception of user intentions.

[0037] By way of example, the current view information may include the identifier of the current view (i.e., the unique identifier of the current view in the unified view information, which is used to quickly locate and reference a specific view), the location of the current view (including the specific location of the view in the user interface, contextual associations with the unified view, hierarchical relationships, and other associations), real-time parameter information of the current view (referring to view data that changes dynamically with user operations or system events, such as keywords entered by the user in the search box, voice broadcasts of artificial intelligence assistants, etc.), business data information of the current view (referring to business logic data related to the current view, which is a subset of the unified view business data information), etc., which can be set according to actual needs.

[0038] For example, user operation sequence information can be recorded in the form of a timeline and behavior list to understand the user's specific preferences and behavior patterns. For example, user operation sequence information can include interaction information between the user and the APP, instruction information between the user and the artificial intelligence (AI) assistant, user browsing behavior information, etc., which can be set according to actual needs.

[0039] After obtaining the current view information and the user operation sequence respectively, the following S102 may be executed: S102: Determine the user demand perception of the user based on the current view information and the user operation sequence.

[0040] Among them, user demand perception is used to perceive user needs.

[0041] S103. Determine the user intention sequence based on user demand perception, unified view information of the application and user panoramic information of the user; wherein the unified view information is used to summarize and display the page views corresponding to all functional points in the target business scenario, and the user panoramic information is used to characterize the user's characteristic information and behavior information.

[0042] For example, unified view information refers to the structured arrangement of page views corresponding to all functional points that users can reach through AI assistants, which may include information in the following four dimensions: association relationship (referring to the logical relationship between different page views, such as how to jump to another functional point through operation at one functional point, etc., including page jump, functional link, data transfer, context association, hierarchical relationship), touch point information (including the application's main screen, menu, list, table, dialog box, etc.), business data information (business data information related to each scenario, such as product information, order and transaction information, service description, content and media information, etc.), user-available operations (user-available operations in each scenario, such as viewing operations, click-to-jump operations, creating operations, editing operations, deleting operations, sharing operations, searching operations, filtering operations, sorting operations, etc.), which can be set according to actual needs.

[0043] For example, the user's panoramic information can provide comprehensive user insights, laying the foundation for real-time analytical perception and proactive service provision, and can include the following three dimensions: user basic information (including the user's basic information when registering, such as account, third-party account binding information, age, device information, etc.), user business information (including the user's business activity records, such as order history, package usage, participation in activities and transaction behavior, etc.), tag information (including the user's geographic location information (such as province, city), channel affiliation information, customer group classification, etc., as well as various tags assigned by the operation team based on the user's characteristic information and behavior information, for accurate positioning of user groups), which can be set according to actual needs.

[0044] For example, when determining the user intention sequence based on user demand perception, the unified view information of the application and the user's user panoramic information, the user demand perception, the unified view information of the application and the user's user panoramic information can be integrated and analyzed to realize user intention perception, and at the same time, the user's user panoramic information and unified view information can be continuously optimized and updated using machine learning, data mining and other technologies at a certain periodic frequency to improve the accuracy and real-time performance of user intention perception.

[0045] For example, when integrating and analyzing user demand perception, unified view information of applications and user panoramic information of users, it can be done through the method of pre-training - fine-tuning - context understanding - real-time reasoning of large models, or through natural language processing (NLP) technology, deep learning algorithms, reinforcement learning, etc., which can be set according to actual needs.

[0046] After determining the user intention sequence, the application can be controlled to display multimodal information based on the user intention sequence, that is, the following S104 is executed.

[0047] S104: Based on the user intention sequence, the application is controlled to display multimodal information, where the multimodal information is used for the user to perform business in the target business scenario.

[0048] For example, multimodal information may include driving view switching, driving digital humans, card rendering, multi-round question prediction, voice broadcasting, text answers, etc. It may also include voice generation, visual content generation (images, charts) information, multi-language conversion information, etc., which can be set according to actual needs.

[0049] It can be seen that in the embodiment of the present application, by obtaining the current view information of the application, and the user operation sequence input by the user in the target business scenario based on the current view information; based on the current view information and the user operation sequence, the user's user demand perception is determined; and based on the user demand perception, the unified view information of the application and the user's user panoramic information, the user intention sequence is determined; and then based on the user intention sequence, the application is controlled to display multimodal information, and the multimodal information is used for the user to perform the business in the target business scenario. In this way, the user intention sequence is determined based on the user demand perception, the unified view information of the application and the user's user panoramic information, so that the application can be intelligently controlled to display multimodal information based on the perceived user intention, which is used for the user to perform the business in the target business scenario, thereby effectively improving the business handling efficiency.

[0050] Based on the above Figure 1 In the illustrated embodiment, for example, in the above 102, when determining the user's user demand perception based on the current view information and the user operation sequence, multimodal fusion can be performed on the different modal information included in the current view information to obtain multimodal view fusion perception information; and based on the behavior sequence of the time dimension and the interaction context sequence of the space dimension included in the user operation sequence, the spatiotemporal sequence behavior perception information is obtained; and then based on the multimodal view fusion perception information and the spatiotemporal sequence behavior perception information, the user's user demand perception is determined, thereby obtaining the user demand perception.

[0051] For example, when multimodal fusion is performed on different modal information included in the current view information, multimodal information such as the current view identifier, the current view location, the current view real-time parameter information, and the current view business data information can be obtained on the front end of the page through JavaScript, HyperText Markup Language (HTML) and the corresponding application programming interface (API) interface response of each contact point, and the information of different modalities is modally converted, standardized and normalized. The large model adopts a multimodal self-supervised learning method, which does not require a large amount of labeled data. The feature representation of multiple modalities is extracted through the model's own learning, and multimodal fusion is completed to obtain multimodal view fusion perception information. This can improve the generalization performance of the model and avoid labeling deviation.

[0052] For example, when obtaining spatiotemporal sequence behavior perception information based on the behavior sequence in the time dimension and the interaction context sequence in the space dimension included in the user operation sequence, the user operation sequence information can be obtained through event monitoring, gesture recognition, data embedding, log recording, large model instruction feedback, etc., and a comprehensive and multi-dimensional user behavior model can be established in combination with the user's behavior sequence in the time dimension and the interaction context sequence in the space dimension, so as to determine the spatiotemporal sequence behavior perception through the user behavior model and obtain the corresponding spatiotemporal sequence behavior perception information, thereby providing data support for the subsequent perception of user intentions.

[0053] Based on any of the above embodiments, for example, in the above 103, when determining the user intention sequence based on the user demand perception, the unified view information of the application and the user's user panoramic information, the user's target potential intention information can be first determined based on the user demand perception, the unified view information of the application and the user's user panoramic information; and based on the target potential intention information and a preset intention sequence template, a user intention sequence is generated; wherein the intention sequence template is used to characterize the mapping relationship between the potential intention information and the user intention.

[0054] For example, when determining the user's target potential intention information based on user demand perception, the unified view information of the application and the user's user panoramic information, the user demand perception, the unified view information of the application and the user's user panoramic information can be analyzed to determine all the target potential intention information of the user's current login, and combined with the preset intention sequence template, a standardized user intention sequence is generated to achieve accurate matching of user needs and optimal service quality. In this way, after generating the standardized user intention sequence, products and services corresponding to the scenario can be actively displayed and pushed to the user according to the user's user intention sequence, and three actionable behavioral information can be actively provided to the user. For example, on the page for purchasing Xiaomi mobile phones, the user can be displayed in the form of bubbles "Introduce me to the detailed parameters of this mobile phone", "When was this mobile phone launched on the market", and "Recommend more Xiaomi mobile phones" to achieve proactive service.

[0055] Among them, the intent sequence template is a standard metadata model of an intent set, which may include the mapping relationship between potential intent information and user intent that is actively perceived and generated by the big model based on user demand perception, the unified view information of the application and the user's panoramic information, as well as the user operation sequence information during the interaction process.

[0056] Based on any of the above embodiments, for example, in the above S104, when the application is controlled to display multimodal information based on the user intention sequence, each user intention is dynamically sorted based on the weight corresponding to the user intention under different classification dimensions and the moment when the user intention is perceived, so as to realize the real-time dynamic adjustment of the user intention priority, and control the application to display multimodal information based on the dynamically adjusted target user intention sequence. For details, please refer to the following Figure 2 As shown, Figure 2 A schematic flow chart of a method for controlling an application to display multimodal information based on a user intention sequence provided in an embodiment of the present application, the method may include: S201. For each user intention in the user intention sequence, determine an interest score corresponding to the user intention based on the weights corresponding to the user intention under different classification dimensions and the moment when the user intention is perceived.

[0057] For example, different classification dimensions may include: classification according to actual user purpose, classification according to explicit and implicit intent, and classification according to positive and negative intent. For example, it may include schematic diagram, display intent, context-aware intent, or operational intent, etc., which may be set according to actual needs.

[0058] Among them, according to the actual purpose of the user, it can be divided into: query intent (the intention shown when the user wants to obtain certain information or data. For example, query order status, query product information, query return policy, etc.), operation intent (the intention shown when the user wants to perform a certain operation or function. For example, book a flight, compare products, cancel an order, etc.), chat and emotional expression intent (the user does not want to obtain specific information or perform operations, but to chat, express emotions or seek company. For example, chatting about the weather, expressing mood, etc.), etc., which can be set according to actual needs. In actual application scenarios, classification categories and intents under each category can be added, modified, deleted, and other operations according to business needs and views.

[0059] Classify by explicit and implicit intent: Explicit intent refers to the intent that users directly express through voice or text, such as telling the AI ​​assistant "I want to order a mango card" through voice, entering keywords in the dialog box, or clicking buttons or bubbles, etc. Implicit intent refers to the intent that users do not express explicitly, but can be inferred through their behavior and characteristics. For example, when browsing a product detail page, there may be different intentions such as purchasing and comparing products.

[0060] Classify by positive or negative intent: Positive intent refers to the user's inquiry or feedback with positive emotions, such as expressing satisfaction, gratitude or interest. Negative intent refers to the situation where the user expresses negative emotions such as dissatisfaction, complaints or problems. It is necessary to take timely measures to solve the problem or adjust the service strategy to improve customer satisfaction.

[0061] In order to dynamically adjust the priority of user intentions and ensure optimal adaptation to the real-time needs of users, in an embodiment of the present application, a dynamic sorting mechanism can be used to score and sort the intention sequence. When using a dynamic sorting mechanism to sort the intention sequence, the weights corresponding to the user's intentions under different classification dimensions and the timeliness of the user's intentions can be combined. The interest score of the intention sequence can be determined by describing the moment when the user's intentions are perceived, so as to adjust the mechanism based on the determined interest score, thereby improving the service quality and meeting the real-time needs of users.

[0062] For example, suppose that the user intention sequence in a certain user interaction process can be recorded as ,in, Indicates that at time step t, the user intention of user u is perceived by real-time analysis. The corresponding interest score can be found in the following formula 1.

[0063] Formula 1

[0064] in, Indicates the interest score corresponding to the user intention s. Usually, The larger the value of , the higher the priority of the corresponding user intention s in the user intention sequence; The smaller the value is, the better the ranking of the corresponding user intention s in the user intention sequence is; It represents the weight of user intention s in the classification dimension according to the user's actual purpose. represents the weight of user intention s in the explicit and implicit classification dimensions of intention, represents the weight of user intention s in the positive and negative classification dimensions of intention, represents the moment when the user intention s is currently perceived, The normalization method is used to consider the timeliness of user intentions, making it distributed between [0.1-1], so that the services provided are closer to the needs of users at the current moment. Indicates the earliest moment of perception, Indicates the latest moment of perception.

[0065] In combination with the above description, after the interest scores corresponding to the intentions of each user are determined respectively, the following S202 may be executed.

[0066] S202 , sorting the user intentions in the user intention sequence in descending order of interest scores to obtain a target user intention sequence.

[0067] The target user intention sequence is a sequence corresponding to the user intention after the priority is dynamically adjusted.

[0068] In this way, the user intentions in the user intention sequence are sorted in descending order of interest scores, which enables the priority of the intent to be dynamically adjusted according to the user's real-time behavior and contextual information, so that it can adapt to the user's current needs more quickly and provide more personalized and timely services.

[0069] S203: Control the application to display multimodal information based on the target user intention sequence.

[0070] By way of example, in an embodiment of the present application, when controlling an application to display multimodal information based on a target user intention sequence, a potential interaction strategy matrix can be first generated based on the target user intention sequence and the target business scenario; the potential interaction strategy matrix can be analyzed based on a large model to generate multiple executable paths corresponding to the business under the target business scenario, and the feasibility and execution efficiency of each executable path can be predicted based on the large model to obtain a prediction result for each executable path; and then, based on the prediction result for each executable path, the application can be controlled to display multimodal information.

[0071] For example, see Figure 3 As shown, Figure 3 A schematic diagram of a framework for generating a potential interaction strategy matrix provided in an embodiment of the present application, wherein the potential interaction strategy matrix may include strategies in two dimensions: user scenario strategy and user demand strategy. Among them, the user scenario strategy is formulated based on the user's current environment, context, and interaction history, such as page jump, chat companionship, etc.; the user demand strategy is formulated based on the user's actual specific needs, such as query services, operation functions, etc.

[0072] For example, when generating a potential interaction strategy matrix based on the target user intention sequence and the target business scenario, it can be implemented in conjunction with a large model. The large model integrates policy templates for different business scenarios. The policy template is a standard template for the corresponding interaction strategy matrix in process execution, including business priority settings, exception handling mechanisms, pre-dependencies, etc. During the docking process, the corresponding target policy template can be determined from the policy templates integrated in the large model based on the target user intention sequence and the target business scenario, so that the corresponding potential interaction strategy matrix can be generated based on the target policy template. The potential interaction strategy matrix can be understood as a matrix containing multiple interaction strategies and possible actions, which is used to guide the large model on how to autonomously analyze and plan in different business scenarios.

[0073] For example, when analyzing the potential interaction strategy matrix based on the big model and generating multiple executable paths corresponding to the business in the target business scenario, the global view information, specific domain knowledge information and specific skill execution capability information of the business in the target business scenario can be obtained first; and the global view information, specific domain knowledge information, specific skill execution capability information and the potential interaction strategy matrix are input into the big model to generate multiple executable paths. Each executable path is composed of a series of task items.

[0074] For example, the global view information, specific domain knowledge information and specific skill execution capability information of the business in the target business scenario can be obtained through an external system, which can be specifically set according to actual needs.

[0075] For example, the external system docking module consists of a unified task executor and a unified system integrator, which can be seen in Figure 4 As shown, Figure 4 A schematic diagram of the structure of an external system provided in an embodiment of the present application, wherein the unified task executor includes a capability management module and a capability call module, wherein the capability management module is used to implement the adaptation and docking of the business system, and the standardized registration and management of each business. The capability call module is used to implement the capability call of the large model to the business system, and also supports the rapid adjustment and optimization of business processes according to business needs.

[0076] The unified system integrator mainly includes the content library (providing global view information), the business library (providing professional field knowledge information) and the capability library (providing professional skill execution capabilities).

[0077] In the scenario where the capability management module implements the adaptation and docking of the business system, when a new business system is connected, it must first undergo standardization verification by the external system docking module, including data format verification, interface specification verification, and business logic verification. After the new business system is connected, the external system docking module will standardize the data. Define the structure and content of the data, including field definition, data structure, data mapping, etc. By standardizing the data, the external system docking module can ensure data consistency.

[0078] The big model is associated with a specific business system using a business registration template. The business registration template includes information such as the name, type, and specific functions of the business system, as well as the role and responsibilities of the big model in the business system, to achieve business registration, grading, listing, and delisting. Each business system creates a corresponding business registration template and associates it with the corresponding intent sequence template and policy template to achieve capability calls such as view query, knowledge query, and skill call.

[0079] For example, in the embodiment of the present application, the business registration template can realize the process control, view information association, data flow, large model plug-in configuration, etc. of each business scenario in the form of a graph. All business libraries in the back end can be used as basic business capabilities for the large model to autonomously analyze and generate corresponding execution strategies.

[0080] For process control and view information association, the process control and view information association of business scenarios can be completed in the form of nodes + dynamic connection lines in the graph. A node represents a business activity or decision point, which is divided into two types: function nodes and reply nodes: function nodes execute specific business logic, such as data processing, service calls, etc., and contain business data information and function details of the current view; reply nodes are responsible for interacting with users, displaying results or requesting additional information, and contain business data information of the current view and multimodal reply arrangement information (including cards, bubbles, digital human voice and action configuration). Dynamic connection lines represent the flow relationship between nodes and determine the direction of the process. Specifically, it includes trigger conditions, intent details, and real-time parameters of the view. The trigger conditions on the connection line ensure that the process will flow to the next node only when specific conditions are met. It can be simple data validation or complex business rule evaluation. The dynamic characteristics of the connection line allow the process to change flexibly according to different conditions or user input.

[0081] The flexible and customizable orchestration process of dynamic connection lines + function nodes and reply nodes, as well as the realization of page jumps, hierarchical relationships and other associations between views, ensure that the big model enables the autonomy and control of various business scenarios.

[0082] Regarding data flow, it is mainly reflected in three aspects: data binding, real-time update and data display. Among them, data binding is the business data information of the view bound to the node, which is used to associate product information, order and transaction information, service description, content and media information, etc. The real-time parameters of the view are bound to the dynamic connection line, which is used to associate the view data that changes dynamically due to user operations or external factors. Real-time update means that during the process execution, the business management module updates the data changes in real time, responds to user input, external events or execution results of other nodes, and ensures the consistency of data and the continuity of the process. Data display means that on the reply node, the business data information of the current view and the multimodal reply arrangement information (including cards, bubbles, digital human voice and action configuration) will be transmitted to the front end and the digital human model in the form of an API interface for data display and result rendering, so as to achieve effective information communication and dynamic generation of the user interface.

[0083] For the configuration of the big model plug-in, you can select and configure the most suitable big model plug-in according to different scenarios and channels in the graph. The general big model has been widely trained and can handle various types of tasks and data, and is suitable for most standard business scenarios; the customized trained professional big model is optimized for specific tasks or fields. This enables the big model to understand and support the unique needs of various business scenarios, while providing personalized services for users of different channels and customer groups to enhance the user experience.

[0084] In the scenario where the capability calling module above realizes the capability calling of the big model to the business system, the capability calling may include three aspects: intelligent identification of parameter input slots, automatic adaptation of various types of protocols, and intelligent parsing of various types of results.

[0085] Among them, intelligent identification of parameter entry slots means that when calling an interface or other capability services, the big model understands the context of the external request and infers the parameters that need to be filled in. For example, if a user "recommends a large data package of about 100 yuan", the big model needs to identify the 100 yuan price information and the product information of the package that the user wants, and automatically fill it into the corresponding parameter slot, and then obtain the corresponding result data from the unified system integrator.

[0086] Automatic adaptation of various types of protocols means that when the large model calls different external capabilities to provide services to users, it automatically adapts to follow specific communication protocols to ensure that information can be smoothly transmitted across systems and platforms. In terms of specific transmission protocols, for web services and API interfaces, the HyperText Transfer Protocol Secure (HTTPS) protocol is used; for the interaction between the large model and the front-end user, the WebSocket protocol is used; for the interfaces of various back-end distributed systems, the Representational State Transfer (RESTful) API is used to implement data query, creation, update and deletion operations; for communication between distributed systems, protocols such as gRPC are used.

[0087] Intelligent parsing of various types of results means that after the system obtains data through external capabilities, it identifies different data formats, such as JavaScript Object Notation (JSON) format, Extensible Markup Language (XML), Comma-Separated Values ​​(CSV) format, or plain text format, etc., and uses NLP technology to extract key information based on the data structure and convert it into the form required by the current view.

[0088] For example, in the embodiment of the present application, when combining a large model to generate multiple executable paths, the large model can be used to analyze the potential interaction strategy matrix, understand different strategies and applicable business scenarios, evaluate the current target business scenarios and user needs, determine the most matching strategy, and generate multiple executable paths based on the selected strategy, and predict the feasibility and execution efficiency of each executable path based on the large model to obtain the prediction results of each executable path; thus, based on the prediction results of each executable path, the application is controlled to display multimodal information, so that by combining the potential interaction strategy matrix, the large model can dynamically adapt to complex and changeable business scenarios, select the optimal strategy according to user needs and environmental factors, and generate a highly adaptable and efficient task path set, which improves the flexibility, accuracy and efficiency of task execution. Among them, each executable path is composed of a series of task items.

[0089] For example, in an embodiment of the present application, when controlling an application to display multimodal information based on the prediction results of each executable path, the executable path corresponding to the optimal prediction result can be first determined as the target executable path; then, multimodal instructions are generated based on the target executable path, and the multimodal instructions are used to control the application to display multimodal information based on the independent transmission channels corresponding to each of the multimodal information.

[0090] For example, when generating multimodal instructions based on the target executable path, the predefined instruction template can be combined to generate a multimodal instruction that matches the target executable path. For example, see Figure 5 As shown, Figure 5 A framework diagram for generating a multimodal instruction provided in an embodiment of the present application, wherein the multimodal instruction is a multimodal instruction that matches user needs, so that the generated multimodal instruction is distributed to the front end and the back end for operations such as instruction execution.

[0091] Among them, the instruction template defines the specific actions that the big model should take when responding to user requests. Including but not limited to driving view conversion, driving digital human dialogue, front-end interface rendering, multi-round question prediction, text streaming output, voice generation and broadcasting, visual content generation (images, charts), multi-language conversion, etc. It can be understood that in the embodiment of the present application, these instruction templates are closely related to the business registration template to ensure that the big model can accurately implement the functions of the business system and output results as expected.

[0092] After the multimodal instructions are generated, they can be passed to the target execution modules, which may include the backend services of the AI ​​assistant, the frontend applications, the 3D digital human engine, and other functional components in the system. Correspondingly, after receiving the multimodal instructions, the target execution module can display the multimodal information based on the independent transmission channels corresponding to each of the multimodal information. For example, see Figure 6 As shown, Figure 6 A framework diagram for executing multimodal instructions provided in an embodiment of the present application, wherein the multimodal information may include driving view switching, driving digital humans, card rendering, multi-round question prediction, voice broadcasting, text answers, etc., and may also include voice generation, visual content generation (images, charts) information, multi-language conversion information, etc., which may be specifically set according to actual needs.

[0093] For example, the multimodal instruction execution process can take the form of conversational or automated interaction according to the specific requirements of the business scenario. In the conversational mode, the big model will interact with the user to obtain additional input or confirm the execution of the operation; in the automated mode, the system will execute the instruction autonomously without additional user interaction. This flexible execution mechanism ensures that the system can adapt to diverse business needs.

[0094] In this way, based on the independent transmission channels corresponding to each multimodal information, the multimodal information can be displayed in parallel, which can effectively improve the system processing efficiency: by receiving and executing multimodal instructions in parallel through multiple independent transmission channels, the system can handle multiple types of tasks at the same time, such as view conversion, digital human dialogue drive, etc., thereby significantly improving the system's processing efficiency, while avoiding resource waste and improving resource utilization efficiency.

[0095] For example, after displaying multimodal information in parallel based on the independent transmission channels corresponding to each of the multimodal information, considering that the AI ​​assistant has the ability to autonomously monitor the dialogue process and can identify whether the dialogue is proceeding as expected, it can also collect the user's operation results on the multimodal information and feed back the collected operation results to the big model, so that once potential errors or dialogue deviations are detected, the big model will actively seek user feedback to correct the deviation and improve the quality of the dialogue. Users have the option of providing feedback at any time during this process to help guide the big model toward the correct learning path, so that the big model can regularly iterate and optimize the model to form a complete data closed loop. Through closed-loop feedback and iterative optimization, the system can more accurately adapt to user needs and provide more efficient and more accurate services.

[0096] In combination with the above description, in the embodiment of the present application, when displaying the multimodal information of the application, for example, see Figure 7 As shown, Figure 7 A framework schematic diagram for displaying multimodal information provided for an embodiment of the present application can first obtain the current view information of the application, as well as the user operation sequence of the user input in the target business scenario based on the current view information; and based on the current view information and the user operation sequence, determine the user's user demand perception; then based on the user demand perception, the unified view information of the application and the user's user panoramic information, determine the user intention sequence to generate a potential interaction strategy matrix; and combine with the external system docking module to generate multimodal instructions, which are used to control the application to display multimodal information based on the independent transmission channels corresponding to each of the multimodal information. In this way, the user intention sequence is determined based on the user demand perception, the unified view information of the application and the user's user panoramic information, so that the application can be intelligently controlled to display multimodal information based on the perceived user intention, which is used for the user to perform the business in the target business scenario, thereby effectively improving the business handling efficiency.

[0097] For example, the multimodal instructions may include text polishing instructions, digital human driving instructions, voice broadcast instructions, or view conversion instructions, etc., which can be set according to actual needs.

[0098] By adopting the technical solution provided in this application, it is possible to utilize technologies such as user intent perception to achieve accurate identification of user's ambiguous intentions. In addition, it is possible to connect with external systems such as business libraries and searches to achieve digital training optimization. It supports providing users with customized service content and empowering services in multiple scenarios, thereby continuously improving the intelligence level of products and service quality, and enhancing the problem-solving capabilities of self-service links.

[0099] In addition, combined with the user's perception of user needs, it can more accurately identify the user's potential demands, conduct multiple rounds of interactive services, capture marketing interaction opportunities, and recommend suitable products or operations to users, thereby helping to improve business conversion. By combining voice interaction with front-end visual images, an intelligent assistant with image, emotion, and interactivity can be provided to enhance the perception of service experience; around customer service scenarios, scene penetration can be improved, so that users can obtain the desired services anytime, anywhere, and on demand within the APP.

[0100] The multimodal information display device of the application provided by the present application is described below. The multimodal information display device of the application described below and the multimodal information display method of the application described above can be referred to each other.

[0101] Figure 8 A schematic diagram of a multimodal information display device for an application provided in an embodiment of the present application. For example, see Figure 8 As shown, the multimodal information display device 80 of the application may include: The acquisition unit 801 is used to acquire the current view information of the application and the user operation sequence input by the user in the target business scenario based on the current view information; A first processing unit 802 is configured to determine a user demand perception of the user based on the current view information and the user operation sequence; The second processing unit 803 is used to determine the user intention sequence based on the user demand perception, the unified view information of the application and the user panoramic information of the user; wherein the unified view information is used to summarize and display the page views corresponding to all function points in the target business scenario, and the user panoramic information is used to characterize the characteristic information and behavior information of the user; The control unit 804 is used to control the application to display multimodal information based on the user intention sequence, and the multimodal information is used for the user to perform the business in the target business scenario.

[0102] For example, in the embodiment of the present application, the first processing unit 802 is configured to determine the user demand perception of the user based on the current view information and the user operation sequence, including: Performing multimodal fusion on different modal information included in the current view information to obtain multimodal view fusion perception information; Based on the behavior sequence in the time dimension and the interaction context sequence in the space dimension included in the user operation sequence, obtaining spatiotemporal sequence behavior perception information; Based on the multimodal view fusion perception information and the spatiotemporal sequence behavior perception information, a user demand perception of the user is determined.

[0103] For example, in the embodiment of the present application, the second processing unit 803 is used to determine the user intention sequence based on the user demand perception, the unified view information of the application and the user panoramic information of the user, including: Determining target potential intention information of the user based on the user demand perception, the unified view information of the application and the user panoramic information of the user; Based on the target potential intent information and a preset intent sequence template, the user intent sequence is generated; wherein the intent sequence template is used to characterize the mapping relationship between the potential intent information and the user intent.

[0104] For example, in the embodiment of the present application, the control unit 804 is used to control the application to display multimodal information based on the user intention sequence, including: For each user intention in the user intention sequence, based on the weights corresponding to the user intention under different classification dimensions and the time when the user intention is perceived, determining an interest score corresponding to the user intention; Sorting the user intentions in the user intention sequence in descending order of interest scores to obtain a target user intention sequence; The application is controlled to display multimodal information based on the target user intention sequence.

[0105] For example, in the embodiment of the present application, the control unit 804 is used to control the application to display multimodal information based on the target user intention sequence, including: Based on the target user intention sequence and the target business scenario, generating a potential interaction strategy matrix; Analyze the potential interaction strategy matrix based on the big model to generate multiple executable paths corresponding to the business in the target business scenario, and predict the feasibility and execution efficiency of each executable path based on the big model to obtain the prediction results of each executable path; Based on the prediction results of each of the executable paths, the application is controlled to display multimodal information.

[0106] For example, in the embodiment of the present application, the control unit 804 is used to analyze the potential interaction strategy matrix based on the large model to generate multiple executable paths corresponding to the business in the target business scenario, including: Obtaining global view information, specific domain knowledge information, and specific skill execution capability information of the business in the target business scenario; The global view information, the specific domain knowledge information, the specific skill execution capability information and the potential interaction strategy matrix are input into the large model to generate the multiple executable paths.

[0107] For example, in the embodiment of the present application, the control unit 804 is used to control the application to display multimodal information based on the prediction results of each executable path, including: Determine the executable path corresponding to the optimal prediction result as the target executable path; The multimodal instruction is generated based on the target executable path, and the multimodal instruction is used to control the application to display the multimodal information based on the independent transmission channels corresponding to the multimodal information.

[0108] The multimodal information display device 80 of the application provided in the embodiment of the present application can execute the technical solution of the multimodal information display method of the application in any of the above-mentioned embodiments, and its implementation principle and beneficial effects are similar to the implementation principle and beneficial effects of the multimodal information display method of the application. Please refer to the implementation principle and beneficial effects of the multimodal information display method of the application, which will not be repeated here.

[0109] Fig. 9 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application, such as Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the multimodal information display method of the application, the method comprising: obtaining the current view information of the application, and the user operation sequence input by the user in the target business scenario based on the current view information; determining the user demand perception of the user based on the current view information and the user operation sequence; determining the user intention sequence based on the user demand perception, the unified view information of the application and the user panoramic information of the user; wherein the unified view information is used to summarize and display the page views corresponding to all function points in the target business scenario, and the user panoramic information is used to characterize the characteristic information and behavior information of the user; controlling the application to display multimodal information based on the user intention sequence, and the multimodal information is used for the user to perform the business in the target business scenario.

[0110] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0111] On the other hand, the present application also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal information display method of the application provided by the above methods, the method including: obtaining the current view information of the application, and the user operation sequence input by the user in the target business scenario based on the current view information; determining the user's user demand perception based on the current view information and the user operation sequence; determining the user intention sequence based on the user demand perception, the unified view information of the application and the user panoramic information of the user; wherein the unified view information is used to summarize and display the page views corresponding to all functional points in the target business scenario, and the user panoramic information is used to characterize the characteristic information and behavior information of the user; based on the user intention sequence, the application is controlled to display multimodal information, and the multimodal information is used for the user to perform the business in the target business scenario.

[0112] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a multimodal information display method for an application provided by the above methods, the method comprising: obtaining current view information of the application, and a user operation sequence input by a user in a target business scenario based on the current view information; determining the user's user demand perception based on the current view information and the user operation sequence; determining a user intention sequence based on the user demand perception, the unified view information of the application and the user panoramic information of the user; wherein the unified view information is used to summarize and display page views corresponding to all functional points in the target business scenario, and the user panoramic information is used to characterize the characteristic information and behavior information of the user; based on the user intention sequence, the application is controlled to display multimodal information, and the multimodal information is used for the user to perform business in the target business scenario.

[0113] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0114] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multimodal information display method for an application, characterized in that: include: Acquire current view information of the application, and a user operation sequence input by the user in a target business scenario based on the current view information; Determining a user demand perception of the user based on the current view information and the user operation sequence; Determine a user intention sequence based on the user demand perception, the unified view information of the application and the user panoramic information of the user; wherein the unified view information is used to summarize and display the page views corresponding to all functional points in the target business scenario, and the user panoramic information is used to characterize the characteristic information and behavior information of the user; Based on the user intention sequence, the application is controlled to display multimodal information, and the multimodal information is used for the user to perform the business in the target business scenario.

2. The multimodal information display method of an application according to claim 1, characterized in that: The determining, based on the current view information and the user operation sequence, the user's user demand perception includes: Performing multimodal fusion on different modal information included in the current view information to obtain multimodal view fusion perception information; Based on the behavior sequence in the time dimension and the interaction context sequence in the space dimension included in the user operation sequence, obtaining spatiotemporal sequence behavior perception information; Based on the multimodal view fusion perception information and the spatiotemporal sequence behavior perception information, a user demand perception of the user is determined.

3. The multimodal information display method of an application according to claim 1 or 2, characterized in that: The determining of the user intention sequence based on the user demand perception, the unified view information of the application and the user panoramic information of the user includes: Determining target potential intention information of the user based on the user demand perception, the unified view information of the application and the user panoramic information of the user; Based on the target potential intent information and a preset intent sequence template, the user intent sequence is generated; wherein the intent sequence template is used to characterize the mapping relationship between the potential intent information and the user intent.

4. The multimodal information display method of an application according to claim 1 or 2, characterized in that: The controlling the application to display multimodal information based on the user intention sequence includes: For each user intention in the user intention sequence, based on the weights corresponding to the user intention under different classification dimensions and the time when the user intention is perceived, determining an interest score corresponding to the user intention; Sorting the user intentions in the user intention sequence in descending order of interest scores to obtain a target user intention sequence; The application is controlled to display multimodal information based on the target user intention sequence.

5. The multimodal information display method of an application according to claim 4, characterized in that: The controlling the application to display multimodal information based on the target user intention sequence includes: Based on the target user intention sequence and the target business scenario, generating a potential interaction strategy matrix; Analyze the potential interaction strategy matrix based on the big model to generate multiple executable paths corresponding to the business in the target business scenario, and predict the feasibility and execution efficiency of each executable path based on the big model to obtain the prediction results of each executable path; Based on the prediction results of each of the executable paths, the application is controlled to display multimodal information.

6. The multimodal information display method of an application according to claim 5, characterized in that: The analyzing the potential interaction strategy matrix based on the big model to generate multiple executable paths corresponding to the business in the target business scenario includes: Obtaining global view information, specific domain knowledge information, and specific skill execution capability information of the business in the target business scenario; The global view information, the specific domain knowledge information, the specific skill execution capability information and the potential interaction strategy matrix are input into the large model to generate the multiple executable paths.

7. The multimodal information display method of an application according to claim 5, characterized in that: The controlling the application to display multimodal information based on the prediction results of each executable path includes: Determine the executable path corresponding to the optimal prediction result as the target executable path; The multimodal instruction is generated based on the target executable path, and the multimodal instruction is used to control the application to display the multimodal information based on the independent transmission channels corresponding to the multimodal information.

8. A multimodal information display device for an application, characterized in that: include: An acquisition unit, configured to acquire current view information of an application program and a user operation sequence input by a user in a target business scenario based on the current view information; A first processing unit, configured to determine a user demand perception of the user based on the current view information and the user operation sequence; a second processing unit, configured to determine a user intention sequence based on the user demand perception, the unified view information of the application, and the user panoramic information of the user; wherein the unified view information is used to summarize and display the page views corresponding to all functional points in the target business scenario, and the user panoramic information is used to characterize the characteristic information and behavior information of the user; A control unit is used to control the application to display multimodal information based on the user intention sequence, and the multimodal information is used for the user to perform the business in the target business scenario.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the multimodal information display method of the application program according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal information display method of the application program as described in any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multimodal information display method of the application program as described in any one of claims 1 to 7 is implemented.

Images