Intelligent interaction method and system for public service scenarios based on multi-modal large model

Abstract

The application discloses a public service scene intelligent interaction method based on a multimodal large model. It belongs to the technical field of intelligent interaction. The method comprises the following steps: acquiring multimodal user input data in a public service scene; inputting the multimodal user input data into a fine-tuned multimodal large model for unified semantic analysis, and generating a unified semantic state vector fused with timing context information; based on the unified semantic state vector, processing a task through a multi-agent collaborative decision system to generate a service execution scheme; acquiring an interaction ability characteristic parameter of a user, and based on the interaction ability characteristic parameter and a service semantic state of the service execution scheme, generating a corresponding adaptive output strategy; and according to the adaptive output strategy, presenting a service result to the user through a combination of multiple output modalities by executing a control agent; the multiple output modalities at least include voice output, graphic display and action guidance.

Description

Technical Field

[0001] This application relates to the field of intelligent interaction technology, and in particular to an intelligent interaction method and system for public service scenarios based on a multimodal large model. Background Technology

[0002] With the rapid development of artificial intelligence technology, large-scale language models such as ChatGPT, LLaMA, and Claude have made significant progress in natural language understanding, multimodal information fusion, and intelligent interaction. These models possess powerful semantic understanding, contextual modeling, and human-like generation capabilities, demonstrating broad application value in scenarios such as intelligent question answering, information retrieval, public service consultation, smart navigation, and decision support. In recent years, with the maturity of multimodal large-scale model technology, models can now jointly model multi-source information such as text, speech, and images, providing important technical support for building intelligent public service systems. However, in practical applications, existing intelligent service systems are still mainly designed for ordinary users with strong digital operation skills, lacking sufficient adaptability to special groups such as the elderly, visually impaired, hearing impaired, and those with mobility impairments, making it difficult to meet the inclusive service needs of users of all ages. Specifically, existing technologies mainly suffer from the following problems: First, the interaction methods are limited and accessibility is insufficient. Most intelligent interactive systems still rely primarily on touchscreen operation and text display, with low integration of multiple output methods such as voice output, graphic display, gesture guidance, and augmented reality assistance. Furthermore, the lack of a unified accessibility mechanism between different terminal devices results in fixed interaction modes and a single output format. Users struggle to obtain clear and intuitive service guidance in complex public environments, severely impacting the user experience.

[0003] Second, there is a disconnect between multimodal information understanding and service execution. Existing systems generally suffer from problems such as poor coordination between semantic understanding and actual operation, low service scheduling efficiency, and imperfect anomaly feedback mechanisms during the process of multimodal information understanding and service execution. Especially when dealing with elderly or disabled users with declining cognitive abilities or limited operational capabilities, the system is prone to errors, misunderstandings, and service interruptions, which hinders the in-depth application of intelligent service systems in the field of accessibility.

[0004] Third, there is a lack of adaptive interaction mechanisms that cater to differences in user capabilities. Existing systems typically employ a uniform interaction logic and output method, failing to dynamically adjust output strategies based on user interaction capability characteristics such as visual perception, auditory perception, motor operation, and cognitive comprehension. This results in higher comprehension and operational barriers for specific user groups.

[0005] Fourth, multi-agent collaborative decision-making mechanisms have not yet been effectively applied to accessible service scenarios. In existing technologies, task understanding, service planning, instruction execution, and effect feedback are often completed by a single model or fixed rule engine, lacking collaborative division of labor and closed-loop optimization mechanisms among multiple agents, making it difficult to cope with the diversity and dynamic changes in user needs in complex public service scenarios.

[0006] In summary, how to fully utilize multimodal large-scale models and multi-agent collaborative technologies to construct barrier-free intelligent interaction methods and systems that are highly adaptable and user-friendly for all age groups, and achieve end-to-end collaborative optimization from understanding user multimodal input, task planning and execution to adaptive interactive output, has become a crucial technical problem that urgently needs to be solved in the field of smart public services. Currently, no effective solutions have been proposed to address the technical problems existing in the aforementioned technologies, such as the limited interaction methods and insufficient barrier-free adaptability, the disconnect between multimodal understanding and service execution, the lack of adaptive interaction mechanisms to accommodate differences in user capabilities, and the ineffective application of multi-agent collaborative decision-making mechanisms. Summary of the Invention

[0007] The embodiments of this disclosure provide an intelligent interaction method and system for public service scenarios based on a multimodal large model.

[0008] According to one aspect of the present disclosure, an intelligent interaction method for public service scenarios based on a multimodal large model is provided, comprising: acquiring multimodal user input data in a public service scenario, wherein the multimodal user input data includes at least voice data, image data, and behavior perception data; inputting the multimodal user input data into a fine-tuned multimodal large model for unified semantic parsing to generate a unified semantic state vector incorporating temporal contextual information; wherein the multimodal large model is obtained based on a pre-trained base model through multimodal expansion and scenario-based fine-tuning, and is used to map different modal features to a unified semantic space and perform cross-modal fusion; and based on the unified semantic state vector... The system processes tasks and generates service execution plans through a multi-agent collaborative decision-making system. This system includes a task understanding agent for parsing user intent, a service planning agent for generating execution paths, and an execution control agent for driving multi-terminal presentation. It acquires user interaction capability feature parameters and, based on these parameters and the service semantic state of the service execution plan, generates corresponding adaptive output strategies. Following these adaptive output strategies, the execution control agent presents the service results to the user through a combination of multiple output modalities. These multiple output modalities include at least voice output, graphic and text display, and action guidance.

[0009] According to another aspect of the present disclosure, a storage medium is also provided, the storage medium including a stored program, wherein the methods described above are executed by a processor when the program is running.

[0010] According to another aspect of the present disclosure, an intelligent interaction system for public service scenarios based on a multimodal large model is also provided, comprising: a multimodal data acquisition module for acquiring multimodal user input data in public service scenarios, wherein the multimodal user input data includes at least voice data, image data, and behavioral perception data; and a semantic state vector generation module for inputting the multimodal user input data into a fine-tuned multimodal large model for unified semantic parsing to generate a unified semantic state vector that integrates temporal contextual information; wherein the multimodal large model is obtained based on a pre-trained base model through multimodal expansion and scenario-based fine-tuning, and is used to map different modal features to a unified semantic space and perform cross-modal fusion; The service execution plan generation module generates service execution plans based on a unified semantic state vector and through a multi-agent collaborative decision-making system. This system includes a task understanding agent for parsing user intent, a service planning agent for generating execution paths, and an execution control agent for driving multi-terminal presentation. The adaptive output strategy generation module acquires user interaction capability feature parameters and generates corresponding adaptive output strategies based on these parameters and the service semantic state of the service execution plan. Finally, the service result presentation module, based on the adaptive output strategy and through the execution control agent, presents the service results to the user through a combination of multiple output modalities. These modalities include at least voice output, text and image display, and action guidance.

[0011] The beneficial effects of this application are as follows: (1) Significantly improves the semantic understanding accuracy and service reliability in complex public service scenarios. This application adopts a multimodal large model obtained by multimodal expansion and scenario-based fine-tuning based on a pre-trained base model to perform unified semantic parsing on multimodal user input data such as voice data, image data, and behavior perception data, generating a unified semantic state vector that integrates temporal context information; at the same time, it introduces a multi-agent collaborative decision-making system including a task understanding agent, a service planning agent, and an execution control agent, to decompose tasks and plan execution paths based on the unified semantic state vector, and complete task processing through multi-agent division of labor and cooperation. Compared with existing intelligent service systems based on single models or rule engines, this application, through multimodal unified semantic modeling and multi-agent joint reasoning, not only significantly improves the semantic understanding accuracy in complex service scenarios, but also effectively reduces the probability of task conflicts and execution failures, thereby improving the overall service reliability and stability of the system.

[0012] (2) Effectively reduce system deployment and maintenance costs, and achieve efficient expansion and flexible upgrades of functional modules. This application adopts a strategy of scenario-based fine-tuning and transfer learning based on published pre-trained base models, avoiding the high computational costs and data dependency problems caused by large-scale pre-training from scratch; at the same time, by constructing specialized task understanding agents, service planning agents, and execution control agents for different service tasks, and implementing targeted fine-tuning, each agent has higher professional adaptability in subdivided scenarios such as navigation query, appointment processing, and equipment control. The above technical means, while ensuring the overall uniformity of the system, achieve efficient expansion and flexible upgrades of functional modules, significantly reducing system deployment and subsequent maintenance costs.

[0013] (3) To achieve automated matching and secure invocation of natural language commands and external service interfaces, thereby improving the system's versatility and engineering implementation capabilities. After receiving the service execution plan, the execution control agent of this application maps the function call sequence and interaction node sequence contained in the service execution plan into an executable instruction sequence corresponding to each function interface, and sequentially calls the corresponding function interfaces through a unified interface call platform to complete the actual execution of the service operation. The above technical means realize automated matching and secure invocation between natural language commands and external service interfaces, effectively solving the problems of complex manual configuration rules and high interface adaptation costs in traditional systems, enabling the intelligent service platform to quickly connect to multiple types of business systems and hardware terminals, thereby improving the system's versatility and engineering implementation capabilities in practical application environments.

[0014] (4) Effectively reduces the understanding and operation threshold of intelligent service systems for special groups, and improves the accessibility and usability of the system. This application obtains user interaction capability feature parameters, including visual perception capability parameters, auditory perception capability parameters, action operation capability parameters, and cognitive understanding capability parameters, and generates corresponding adaptive output strategies based on the service semantic state of the interaction capability feature parameters and service execution scheme. According to the adaptive output strategy, the service results are presented to the user in a combination of multiple output modalities such as voice output, graphic and text display, gesture guidance, and augmented reality display through the execution control agent. The above technical means construct a multi-terminal barrier-free interactive output mechanism with user interaction capability feature parameters as the core, realizes dynamic adaptive adjustment of interaction forms, effectively solves the problem of insufficient adaptability of existing intelligent service systems to special groups such as the elderly and disabled, significantly reduces the understanding and operation threshold of special groups, and improves the accessibility, usability, and inclusiveness of intelligent service systems.

[0015] (5) To achieve long-term self-optimization and evolution of interaction strategies and service processes, ensuring the stability and advancement of the system during long-term operation. The multi-agent collaborative decision-making system of this application also includes a feedback evaluation agent for monitoring and evaluating the service presentation effect; during and after the multimodal presentation, the feedback evaluation agent collects user response behavior data, attention duration data, and emotional feedback data to the multimodal presentation, and performs comprehensive analysis on the collected data to generate presentation effect evaluation results; the presentation effect evaluation results are fed back to the module that generates adaptive output strategies for subsequent iterative optimization of output strategies. The above technical means construct a closed-loop feedback mechanism of "output-perception-evaluation-optimization", continuously monitors and models user interaction behavior and service effects, enabling the system to continuously adjust the service mode according to changes in user habits and capabilities, thereby possessing continuous learning and adaptive evolution capabilities, ensuring the stability and advancement of the system during long-term operation.

[0016] (6) Possesses strong modular expansion capabilities and promising prospects for industrial application. The overall technical system of this application adopts a modular and hierarchical design concept. The multimodal data acquisition module, semantic state vector generation module, service execution scheme generation module, adaptive output strategy generation module, and service result presentation module are decoupled and connected through standardized interfaces, supporting independent replacement and upgrade deployment. The above design is suitable for rapid verification in university scientific research experimental environments, and can also meet the large-scale promotion needs of government public service platforms and industry application systems, possessing good prospects for industrialization and standardization applications. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this application, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating the intelligent interaction method for public service scenarios based on a multimodal large model, as described in Embodiment 1 of this disclosure. Figure 2 This is a schematic diagram of the overall technical architecture of the barrier-free intelligent service system based on a multimodal large model and a multi-agent collaborative mechanism, as described in Embodiment 1 of this disclosure. Figure 3 This is a schematic diagram of the multi-scenario, multi-modal data acquisition and alignment processing flow according to Embodiment 1 of this disclosure; Figure 4 This is a schematic diagram of the LoRa fine-tuning technique according to Embodiment 1 of this disclosure; Figure 5 This is a schematic diagram of the structure of the multi-agent collaborative decision-making system according to Embodiment 1 of this disclosure; Figure 6 This is a schematic diagram of the structure of an intelligent interactive system for public service scenarios based on a multimodal large model, as described in Embodiment 2 of this disclosure. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this disclosure.

[0019] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0020] Example 1

[0021] According to the first aspect of this embodiment, an intelligent interaction method for public service scenarios based on a multimodal large model is provided. Figure 1 A flowchart illustrating the method is shown. Figure 2 A schematic diagram of the overall technical architecture of the barrier-free intelligent service system that implements this method is shown.

[0022] refer to Figure 2As shown, the overall system architecture of this embodiment adopts a hierarchical design, including a multimodal input layer, a semantic understanding layer, an intelligent decision-making layer, and an accessibility output layer from bottom to top. The multimodal input layer is used to input user voice data, image data, behavioral data, and text data. This layer centrally manages the aforementioned devices through a unified control module and uses a combination of event-driven and periodic sampling to achieve synchronous cross-device data acquisition. The semantic understanding layer, with a finely tuned multimodal large model as its core, receives multi-source heterogeneous user input data output from the multimodal input layer. Through multimodal feature mapping, dynamic gating cross-modal fusion, and temporal context modeling, it generates a unified semantic state vector that integrates temporal context information. This unified semantic state vector serves as the unified cognitive foundation for the intelligent decision-making layer. The intelligent decision-making layer includes a task understanding agent, a service planning agent, an execution control agent, and a feedback evaluation agent. Each agent collaborates based on a shared, unified semantic state vector: the task understanding agent parses user intent and generates a structured task description; the service planning agent generates the optimal service execution path; the execution control agent drives multi-terminal presentation according to adaptive output strategies; and the feedback evaluation agent monitors presentation effects and generates optimization feedback signals. Agents update their states and interact via a publish-subscribe communication mechanism to ensure the stability and reliability of the collaborative process. The accessibility output layer receives adaptive output strategies and service results generated by the multi-agent collaborative decision-making layer. Through a unified multi-terminal output interface, it drives voice broadcast terminals, graphic display terminals, gesture prompt terminals, and AR navigation terminals to present service results to users in combinations of various output modalities. This layer also collects user response data to multi-modal presentations and feeds it back to the feedback evaluation agent to form a closed-loop optimization. The above four-layer architecture is decoupled through standardized interfaces, and each functional module supports independent replacement and upgrade deployment, exhibiting excellent modular scalability.

[0023] The following combination Figure 1 The flowchart shown illustrates the specific steps of the method in this embodiment. (Refer to...) Figure 1 As shown, the method includes: S1: Acquire multimodal user input data in public service scenarios. Multimodal user input data includes at least voice data, image data, and behavioral perception data.

[0024] Specifically, this embodiment is applied, but not limited to, public service scenarios such as hospital outpatient halls, government service halls, university libraries, commercial complexes, and transportation hubs. In each of these application scenarios, multimodal sensing devices are deployed according to a unified deployment standard, including: an array microphone system for collecting voice data, supporting far-field voice acquisition and spatial positioning; a high-resolution camera for collecting image data, supporting the acquisition of RGB images, infrared images, and depth images; inertial sensors, millimeter-wave radar, and WiFi sensing modules for collecting behavioral perception data; and interactive input devices including touchscreens, physical buttons, and mobile terminal interfaces. A unified control module centrally manages these devices, enabling synchronous data acquisition across devices. During the specific acquisition process, the system uses a combination of event-driven and periodic sampling methods for data acquisition. When user interaction behavior is detected (e.g., voice wake-up, touch operation, user entering a preset recognition area), the multimodal synchronous acquisition process is automatically triggered; environmental data and user behavior trajectory data are periodically sampled at fixed time intervals. Through this dual-mechanism combination, comprehensive coverage of key interactive information and background environmental information is achieved.

[0025] In addition, combined Figure 3 As shown, to ensure good fusionability of the collected multi-source heterogeneous data, this embodiment uses a unified standardized format for storing various modal data. Specifically, speech data can use PCM or WAV encoding format, with a sampling rate of 16kHz or 48kHz and a bit depth of 16bit; text data is stored in structured JSON format; image data is uniformly stored in JPEG or PNG format, and video data is stored in MP4 or H.264 format, accompanied by XML or JSON annotation files to record target bounding box information, pose information, etc. Furthermore, after data acquisition, multi-level annotation processing can be performed on the raw data to enhance its semantic value. Annotation content can include: a basic annotation layer, including time labels, scene labels, and device source labels; a semantic annotation layer, including intent category, service type, emotional state, and urgency level; and an accessibility feature annotation layer, annotating speech rate characteristics, operation frequency, and pause duration for the elderly and special populations. The annotation process can employ a three-level mechanism combining manual annotation, semi-automatic annotation, and model verification to ensure annotation quality.

[0026] Therefore, by deploying multimodal sensing devices in public service scenarios and using a combination of event-driven and periodic sampling for data collection, while implementing standardized data format specifications and multi-level semantic annotation, the system can comprehensively and with high quality acquire multi-source heterogeneous information from users during the interaction process, providing a rich and reliable data foundation for subsequent semantic understanding and intelligent decision-making.

[0027] S2: Input multimodal user input data into a fine-tuned multimodal large model for unified semantic parsing to generate a unified semantic state vector that integrates temporal context information; wherein, the multimodal large model is obtained by multimodal expansion and scenario-based fine-tuning based on a pre-trained base model, and is used to map different modal features to a unified semantic space and perform cross-modal fusion; Specifically, in this embodiment, a publicly released pre-trained base model is first selected as the core semantic understanding engine. This pre-trained base model is preferably a large-scale language model or a multimodal large model, whose training corpus covers general knowledge, dialogue corpora, and multimodal semantic information, possessing good generalization and reasoning capabilities. After selecting the pre-trained base model, it undergoes multimodal extension processing to enable it to access non-textual information such as speech data, image data, and behavior-aware data. Specifically, a multimodal feature mapping structure is introduced at the input of the pre-trained base model. This structure encodes features for speech data, image data, and behavior-aware data respectively, obtaining initial feature vectors for each modality. Then, through a semantic space alignment mapping layer, the initial feature vectors of each modality are mapped to a unified semantic space with the same word vector dimension as the pre-trained base model, obtaining a unified semantic representation for each modality. Based on this, cross-modal fusion processing is performed on the unified semantic representation of each modality after mapping. The fused multimodal features are embedded into the context of the pre-trained base model according to the preset template, and joint semantic modeling is performed by combining the temporal information in the historical interaction process. Finally, a unified semantic state vector that integrates temporal context information is generated.

[0028] To enable the multimodal large model to adapt to specific public service scenarios, this embodiment also employs scenario-based fine-tuning to optimize the model. Based on the actual application scenarios of the system deployment (such as hospitals, government service halls, transportation hubs, and commercial venues), sample data subsets for each scenario are constructed. These sample data are then used to fine-tune and train the multimodal large model, allowing it to better understand user intent, service needs, and environmental characteristics in specific scenarios. Thus, by using a multimodal large model obtained through multimodal expansion and scenario-based fine-tuning based on a pre-trained base model, the system can uniformly map heterogeneous user inputs from different perception channels to the same semantic space and perform cross-modal fusion, generating a unified semantic state vector that integrates temporal contextual information. This overcomes the shortcomings of traditional systems where each modality's information is processed independently and joint reasoning is difficult, providing an accurate and complete semantic foundation for subsequent task understanding and decision-making.

[0029] S3: Based on a unified semantic state vector, a multi-agent collaborative decision-making system is used to process tasks and generate service execution plans. The multi-agent collaborative decision-making system includes a task understanding agent for parsing user intent, a service planning agent for generating execution paths, and an execution control agent for driving multi-terminal presentation. Specifically, in this embodiment, the multi-agent collaborative decision-making system uses the unified semantic state vector generated in step S2 as input, and multiple functional agents collaborate to complete the task processing. This system is built on a unified semantic understanding framework and a shared state space. Each agent works collaboratively based on unified cognition, forming a stable and efficient service decision-making chain. First, the task understanding agent performs deep semantic analysis and intent decomposition processing on the unified semantic state vector. Based on a pre-set task ontology library and historical interaction context information, this agent performs structured modeling of user needs, converting natural language and behavioral information into a standard task description format. This task description information includes, but is not limited to, task type, task constraints, and task priority parameters, providing standardized input for subsequent service planning. Second, the service planning agent receives the task description information and, combined with the rule features of the current scenario and the status of available system resources, generates a service execution plan from the current state to the task completion goal. This service execution plan includes the function call order and interaction node sequence, clarifying the key steps required to complete the user task and their sequential relationship.

[0030] In addition, combined Figure 2 As shown, this multi-agent collaborative decision-making system also includes an execution control agent. This execution control agent is used in subsequent steps to drive multiple terminal devices to present service results to the user in combinations of various output modalities based on the generated adaptive output strategy. Thus, through the division of labor and cooperation among the task understanding agent, service planning agent, and execution control agent in the multi-agent collaborative decision-making system, the system can transform complex user needs into operable service execution plans, achieving effective connection from semantic understanding to task planning and execution preparation, thereby improving the efficiency and accuracy of task processing.

[0031] S4: Obtain the user's interaction capability feature parameters, and based on the interaction capability feature parameters and the service semantic state of the service execution plan, generate a corresponding adaptive output strategy; and Specifically, in this embodiment, the system first acquires the user's interaction capability characteristic parameters. These parameters are used to comprehensively evaluate the user's ability levels in visual perception, auditory perception, motor operation, and cognitive understanding. These parameters can be automatically estimated by analyzing the user's historical interaction behavior data, device usage habit data, and real-time perception information. For example, the system can comprehensively evaluate the user's various interaction capabilities based on indicators such as reaction time, operation success rate, and frequency of repeated information requests in the user's historical operation records, combined with the user's perceived behavioral characteristics during the current interaction. Simultaneously, the system acquires the service semantic state corresponding to the service execution plan generated in step S3. This service semantic state characterizes the content type, information complexity, and urgency of the service result to be output. For example, for navigation guidance service results, the content type is spatial location information, with relatively low information complexity but potentially high urgency; for business processing service results, the content type is a process step description, with higher information complexity, requiring the user to gradually understand and execute.

[0032] Based on this, the system uses user interaction capability characteristic parameters and service semantic state as decision-making criteria to automatically determine the appropriate output strategy for the current user and task. This adaptive output strategy defines the output methods and their combinations that should be used when presenting service results to the user. For example, for users with weak visual perception but normal auditory perception, the output strategy may prioritize voice output with supplementary text and graphics display; for users with limited operational capabilities, the output strategy may add action guidance and simplify the interface. Thus, by acquiring user interaction capability characteristic parameters and combining them with service semantic state to generate an adaptive output strategy, the system can dynamically adjust the output method according to the differences in abilities of different users and the information characteristics of different tasks, providing a decision-making basis for achieving barrier-free and personalized service interaction.

[0033] S5: Based on the adaptive output strategy, the service results are presented to the user through a combination of multiple output modalities by executing the control agent; the multiple output modalities include at least voice output, graphic display and action guidance.

[0034] Specifically, in this embodiment, the execution control agent first receives the adaptive output strategy generated in step S4 and parses the output methods and combination relationships defined therein. Then, the execution control agent adapts the service result content to be presented according to the output strategy. During the output process, the system can first perform semantic reconstruction and hierarchical expression processing on the service results, breaking down complex service result information into multi-level, progressive content units to avoid information overload and improve the understanding efficiency for elderly users and users with cognitive impairments. Subsequently, the system maps the reconstructed information to the multi-terminal output module according to the output strategy. Regarding voice output, the system can present the service results via a speaker device using voice broadcasting, and dynamically adjust the speech rate, volume, and repetition frequency according to the user's ability characteristics to ensure clear information reception. Regarding graphic display, the system can present the service results via a display screen device using text and images, and dynamically adjust the font size, contrast, and color mode according to the user's visual ability to ensure readability for visually impaired users. In terms of motion guidance, the system can combine dynamic indicator lights, arrow indicators, or synchronous prompts generated after real-time recognition of the user's posture by a camera and depth sensor to guide the user through the operation process and reduce the difficulty of operation for users with physical disabilities.

[0035] The aforementioned multiple output modalities can be presented synchronously or sequentially according to the instructions of the adaptive output strategy. A unified multi-terminal output interface drives the corresponding output devices to complete the multi-modal presentation of service results. Therefore, by combining multi-modal outputs based on the adaptive output strategy, the system can present service results in a way that is most easily perceived and understood by users, effectively reducing the understanding and operational barriers for users of different skill levels and improving the accessibility and usability of the intelligent service system.

[0036] As mentioned in the background section above, how to fully utilize multimodal large models and multi-agent collaborative technology to construct barrier-free intelligent interaction methods and systems that are highly adaptable and user-friendly for users of all ages, and to achieve end-to-end collaborative optimization from understanding user multimodal input, task planning and execution to adaptive interactive output, has become an important technical problem that urgently needs to be solved in the field of smart public services.

[0037] In view of this, this application provides an intelligent interaction method for public service scenarios based on a multimodal large model. By acquiring multimodal user input data in public service scenarios and employing standardized data format specifications and multi-level semantic annotation mechanisms, the fusionability and semantic richness of multi-source heterogeneous data are ensured. By utilizing a multimodal large model obtained through multimodal expansion and scenario-based fine-tuning based on a pre-trained base model, different modal features are mapped to a unified semantic space and cross-modal fusion is performed to generate a unified semantic state vector that integrates temporal contextual information. This overcomes the shortcomings of traditional systems where each modal information is processed independently and joint reasoning is difficult. Through a multi-agent collaborative decision-making system comprising a task understanding agent, a service planning agent, and an execution control agent, based on the unified semantic state vector, [the method is further developed]. The system processes tasks and generates service execution plans, effectively linking semantic understanding with task planning and facilitating division of labor and collaboration among multiple agents. By acquiring user interaction capability feature parameters, including visual perception capability parameters, auditory perception capability parameters, action operation capability parameters, and cognitive understanding capability parameters, and generating corresponding adaptive output strategies based on the service semantic state of the service execution plan, the system presents service results in a combination of various output modalities such as voice output, graphic display, and action guidance, thereby achieving dynamic matching between output methods and user capability characteristics and task features, and reducing the understanding and operation threshold for users with different ability levels.

[0038] Therefore, this application, through the organic combination of the above technical solutions, provides effective solutions to the following technical problems existing in the prior art: Regarding the problem of a single interaction method and insufficient accessibility adaptation, this application constructs a multimodal adaptive output mechanism with user interaction capability feature parameters as the core, realizing dynamic adjustment of interaction forms and collaborative presentation of multiple output modalities; Regarding the problem of the disconnect between multimodal understanding and service execution, this application achieves seamless connection from multimodal information understanding to service execution scheme generation through unified semantic parsing of a large multimodal model and task planning of a multi-agent collaborative decision-making system; Regarding the problem of the lack of adaptive interaction mechanisms oriented towards user capability differences, this application obtains user interaction capability feature parameters and generates adaptive output strategies based on these parameters, achieving precise matching between output methods and individual user capabilities; Regarding the problem that multi-agent collaborative decision-making mechanisms have not been effectively applied, this application constructs a multi-agent collaborative decision-making system including a task understanding agent, a service planning agent, and an execution control agent, realizing the division of labor and collaborative execution of complex service tasks. Furthermore, this application comprehensively solves the aforementioned technical problems existing in the prior art, and significantly improves the usability, inclusiveness, and user satisfaction of intelligent service systems in public service scenarios.

[0039] Optionally, the multimodal large model includes a multimodal feature mapping module, a dynamic gating cross-modal fusion module, and a temporal context modeling module. Furthermore, the operation of inputting multimodal user input data into the fine-tuned multimodal large model for unified semantic parsing to generate a unified semantic state vector incorporating temporal context information includes: Through the multimodal feature mapping module, feature encoding is performed on the user input data for each modality, and the encoded feature vectors of each modality are mapped to a unified semantic space with the same word vector dimension as the pre-trained base model, obtaining a unified semantic representation for each modality; Through the dynamic gating cross-modal fusion module, attention weights for each modality are dynamically calculated based on the current task context, and the unified semantic representations of each modality are weighted and fused based on the attention weights to obtain the cross-modal fusion representation at the current moment; Through the temporal context modeling module, sequence modeling is performed on the cross-modal fusion representations at multiple consecutive moments, including the current moment, to extract historical temporal context information. The cross-modal fusion representation of the current moment, historical temporal context information, and context embedding of the pre-trained base model are concatenated and linearly transformed to generate a unified semantic state vector that integrates temporal context information.

[0040] In this embodiment of the invention, the multimodal large-scale model is constructed based on existing publicly released large-scale pre-trained models, through multimodal expansion and scenario-based fine-tuning, to achieve unified understanding and expression of multi-source heterogeneous information. First, a suitable pre-trained base model is selected as the core semantic understanding engine according to application requirements. The pre-trained base model preferably adopts a publicly released large language model or multimodal large-scale model, whose training corpus covers general knowledge, dialogue corpus, and multimodal semantic information, possessing good generalization and reasoning capabilities. The word vector dimension of the pre-trained base model is typically 512 to 4096, depending on the structural design of the selected model. After selecting the pre-trained base model, a multimodal expansion structure is designed to enable it to access non-textual information such as images, speech, and behavioral perception data. Specifically, a multimodal feature mapping module is introduced at the input of the pre-trained base model. This module consists of a multimodal encoder, a semantic space alignment layer, and a dynamic fusion layer, used to uniformly map different modal features to the semantic space of the language model. Simultaneously, a dynamic gating cross-modal fusion module is introduced during the feature fusion stage to adaptively adjust the fusion weights of each modality based on the task context. Furthermore, a temporal context modeling module is introduced to capture historical temporal dependencies during user interactions. These three functional modules together constitute the core processing front-end of the multimodal large model. After completing the multimodal extended structure design, to enable the model to adapt to specific public service scenarios, a scenario-based fine-tuning approach is adopted to optimize the model. Specifically, based on the actual application scenarios of the system deployment (such as hospitals, government service halls, transportation hubs, and commercial venues), corresponding sample data subsets are constructed, and these sample data are used to fine-tune and train the multimodal large model.

[0041] During the scene fine-tuning stage, combined with Figure 4 As shown, the parameter-efficient fine-tuning technique LoRA (Low-Rank Adaptation) can be used to optimize the model, avoiding the high computational burden of full parameter updates. The core idea of ​​LoRA is to introduce low-rank trainable parameters as a bypass to a specific weight matrix in the pre-trained base model, while keeping the original pre-trained weights unchanged. Specifically, for a certain weight matrix W to be fine-tuned in the model (i.e., the original parameter matrix in the pre-trained base model), a low-rank increment matrix is ​​introduced. This makes the fine-tuned weight matrix Represented as: ; Where W is the original weight matrix of the pre-trained pedestal model, which remains frozen during fine-tuning and is not updated with gradients. The low-dimensional trainable parameter increment matrix can be further decomposed into the product of two low-rank matrices B and A, i.e. =BA. Effective adaptation of model behavior can be achieved by training only a small number of parameters in matrices B and A. The typical value range for the low-rank dimension r is 4 to 64, which can be set according to the model size and task complexity.

[0042] Through the LoRA fine-tuning process described above, the model can better understand user intent, service requirements, and environmental characteristics in specific scenarios, while significantly reducing the consumption of computing resources and storage costs during the fine-tuning phase.

[0043] In the application phase, after inputting the multimodal user input data obtained in step S1 into the multimodal large model, the multimodal feature mapping module first performs feature encoding processing on each modality's data. This module first constructs a dedicated encoder for each type of input modality. For image modalities, a visual encoding network is used to extract spatial feature vectors. This visual encoding network can be a pre-trained convolutional neural network or a visual Transformer network, capable of extracting high-dimensional feature representations containing information such as target location, scene layout, and object attributes from RGB images, infrared images, or depth images. Let the feature output by the m-th modality encoder be... ,in This represents the feature dimension of the modality. For the image modality, The typical value range is 512 to 2048, which can be determined based on the output dimension of the selected coding network. For the speech modality, a speech feature coding network is used to extract joint acoustic and semantic features. This speech feature coding network can be a feature extraction layer based on a pre-trained speech recognition model, capable of extracting feature representations containing phoneme information, intonation features, and semantic cues from speech waveforms or acoustic features. Feature Dimensions of Speech Modality The typical value range is also 128 to 2048, which can be determined based on the speech sampling rate and the coding network structure. For behavior-aware modalities (such as user gesture trajectories, touch operation sequences, or positional movement paths), a sequence coding network is used for temporal feature extraction. This sequence coding network can be a recurrent neural network or a one-dimensional convolutional neural network, capable of extracting feature vectors representing user behavior patterns from discrete sequences of behavioral events or continuous sensor readings.

[0044] After completing the feature encoding for each modality, the multimodal feature mapping module further maps the feature vectors of each modality to a unified semantic space with the same word vector dimension as the pre-trained base model through a semantic space alignment mapping layer. This mapping process can be achieved through a linear transformation, and its mathematical expression is: ; in, Let m be the feature vector output by the encoder for the m-th mode. The modality mapping matrix, For bias terms, This represents the mapped unified semantic representation vector. The dimension 'd' of the unified semantic space is consistent with the word vector dimension of the pre-trained base model, typically ranging from 512 to 4096, to ensure that the mapped multimodal features can be effectively jointly processed with the text embeddings of the language model. Through the above feature encoding and semantic space mapping operations, the multimodal feature mapping module uniformly transforms heterogeneous modal data such as images, speech, and behavior perception into vector representations in the same semantic space, providing standardized input for subsequent cross-modal fusion.

[0045] Furthermore, after obtaining the unified semantic representation vectors of each modality, the dynamic gating cross-modal fusion module dynamically calculates the attention weights of each modality based on the current task context, and performs weighted fusion based on these attention weights. In public service interaction scenarios, the contribution of different modal information to accurately understanding user intent is not constant. For example, in noisy environments, the reliability of voice data decreases, and the system should rely more on image data and behavioral perception data to infer user needs; while in dimly lit environments, the quality of image data is limited, and the importance of voice data increases accordingly. To address these issues, this embodiment introduces an adaptive fusion strategy based on an attention mechanism. The dynamic gating cross-modal fusion module first calculates the attention weights of the unified semantic representation vectors of each modality. Let there be M modal inputs at the current time t, and the unified semantic representation vectors output by the multimodal feature mapping module for each modality are as follows: The calculation process for attention weights will be explained in detail below.

[0046] After obtaining the attention weights corresponding to each modality, the dynamic gating cross-modal fusion module uses the attention weights as weighting coefficients to perform a weighted summation of the unified semantic representation vectors of each modality, thus obtaining the cross-modal fusion representation at the current time step. The mathematical expression for this weighted fusion process is: ; in, Let m be the attention weights for the m-th modality. Let z be the unified semantic representation vector of the m-th modality, and z be the cross-modal fusion representation vector obtained after weighted fusion.

[0047] Compared to the simple splicing or fixed weighted fusion methods used in traditional approaches, the dynamic weighted fusion strategy based on attention weights can adaptively adjust the contribution of each modality according to different service scenarios and task contexts, thereby improving the robustness and accuracy of multimodal semantic understanding in complex environments.

[0048] In public service interaction scenarios, user behavior and voice commands often exhibit distinct time-series characteristics. For example, users may gradually clarify their needs through continuous multi-turn dialogues or express navigation intentions through continuous movement trajectories. Relying solely on the cross-modal fusion representation at the current moment is insufficient to fully capture the user's true intent; a comprehensive judgment based on historical context information is necessary. Therefore, this embodiment utilizes a temporal context modeling module to perform sequence modeling on the cross-modal fusion representation across multiple consecutive moments, including the current moment, and extracts historical temporal context information. Let the sequence of cross-modal fusion representations for consecutive time steps be... Where t is the current time. The temporal context modeling module uses a sequence encoding network based on the Transformer architecture to encode the sequence. The Transformer architecture can effectively capture the long-distance dependencies between time points in the sequence through a multi-head self-attention mechanism. The typical range of its number of layers is 2 to 12, and the typical range of the number of attention heads H is 4 to 16, which can be configured according to computing resources and temporal modeling accuracy requirements. After processing by the sequence encoding network, a temporal context representation incorporating historical information is obtained. This timing context indicates It encodes the evolution trajectory of user behavior and the trend of semantic state changes from the start of the interaction to the current moment, providing a historical reference for understanding the complete intent of the user's current input.

[0049] Obtain the cross-modal fusion representation at the current moment. Historical time sequence context information and context embedding of pre-trained pedestal models The system then fuses these three types of information to generate a final unified semantic state vector. This includes the context embedding of the pre-trained base model. This is the semantic context representation established by the base model prior to the current interaction round, including dialogue history, identified entity information, and task progress status. Combining multimodal fusion information with the context embedding of the language model itself enables multimodal semantic understanding and the reasoning ability of the language model to complement each other. The specific fusion method is as follows: first, the cross-modal fusion representation at the current moment is... Historical time sequence context information Context embeddings of pre-trained pedestal models Vector concatenation is performed to obtain a concatenated joint representation vector; then, a linear transformation layer is used to perform feature fusion and dimensionality adjustment on this joint representation vector, the mathematical expression of which is: ; in, This indicates that the three vectors are concatenated along the feature dimension. To fuse the weight matrix, For bias terms, This is the generated unified semantic state vector that incorporates temporal context information. It integrates current multimodal input information, historical interaction sequence information, and language model context information, and can comprehensively and accurately represent the user's complete intent and semantic state at the current interaction moment.

[0050] Thus, through the collaborative work of the aforementioned multimodal feature mapping module, dynamic gating cross-modal fusion module, and temporal context modeling module, the system transforms heterogeneous user input data from different perception channels into a unified semantic state vector, providing an accurate and complete semantic foundation for task understanding and service planning in subsequent multi-agent collaborative decision-making systems.

[0051] Optionally, the dynamic gating cross-modal fusion module dynamically calculates the attention weights of each modality based on the current task context. This includes: obtaining the unified semantic representation vectors of each modality output by the multimodal feature mapping module; performing inner product operations on the unified semantic representation vectors of each modality with preset query vectors to obtain the original relevance scores corresponding to each modality; wherein the query vector is used to represent the semantic attention tendency of the current task context; and normalizing the original relevance scores of each modality so that the sum of the attention weights of all modalities is one, and the attention weight of any modality is proportional to the exponential function value of the original relevance score to obtain the attention weights corresponding to each modality.

[0052] In this embodiment of the invention, the dynamic gating cross-modal fusion module first receives the unified semantic representation vectors of each modality output by the multimodal feature mapping module. Let M be the number of modalities received by the system at the current moment, where M typically ranges from 3 to 6, and may include text modalities, image modalities, speech modalities, behavior-aware modalities, etc. The unified semantic representation vectors corresponding to each modality are denoted as follows: Each of them Let be a vector of dimension d, where d is the same dimension as the word vectors in the pre-trained base model. To measure the relevance of each modal information to the current task context, the dynamic gating cross-modal fusion module introduces a trainable query vector. The query vector q has the same dimension as the unified semantic representation vector d, and its parameters are optimized and updated during model training using the backpropagation algorithm. The role of the query vector q is to represent the semantic focus of the current task context, i.e., what type of information the system should prioritize in the current interaction scenario. After obtaining the query vector q, the dynamic gating cross-modal fusion module integrates the unified semantic representation vectors of each modality... Perform an inner product operation with each query vector q, and calculate the vector dot product of the two as the raw relevance score for that modality. The mathematical expression for this inner product operation is: ; in, This represents the transpose of the query vector q. Let be the unified semantic representation vector for the m-th modality. This is the original relevance score for the m-th modality. The result of the inner product operation reflects the directional consistency and magnitude matching degree of the two vectors in the semantic space: the original relevance score. The higher the value, the better the information of the m-th modality matches the semantic focus of the current task context, and the greater the importance of that modality in understanding the user's intent.

[0053] Original correlation score The value range of is unrestricted, making it unsuitable for direct use as a weighting coefficient in weighted fusion. To transform the original relevance scores into attention weights usable for weighted fusion, the dynamic gating cross-modal fusion module normalizes the original relevance scores for each modality. This normalization is implemented using the Softmax function, whose mathematical expression is: ;in, Let M be the original relevance score of the m-th mode, and M be the total number of modes. Let be the attention weights corresponding to the m-th modality after normalization. The above normalization process has the following mathematical properties: First, the sum of the attention weights for all modalities is one, i.e. This allows the attention weights to have a probabilistic interpretation, facilitating subsequent weighted fusion operations; secondly, the attention weights for any modality The exponential function value of the original correlation score of this mode The attention weight is proportional to the original relevance score of the modality. This relationship is a non-linear exponential mapping, which can amplify the weight difference between important and secondary modalities, making the fusion result more focused on key information.

[0054] Through the above steps, the dynamic gating cross-modal fusion module has completed the calculation of attention weights for each modality. The resulting attention weights... It is used in subsequent cross-modal fusion representation generation operations, that is, with A unified semantic representation vector of weighted coefficients for each modality We perform a weighted summation to obtain the cross-modal fusion representation at the current time step. Therefore, by introducing trainable query vectors and an attention weighting mechanism based on the Softmax function, the system can automatically learn and dynamically adjust the importance weights of each modality according to the current task context, making the multimodal fusion process task-adaptive and effectively improving the system's semantic understanding robustness in complex public service scenarios.

[0055] Optionally, the operation of generating a service execution plan by processing tasks through a multi-agent collaborative decision-making system based on a unified semantic state vector includes: using a task understanding agent to perform intent recognition and semantic decomposition on the unified semantic state vector, and combining historical interaction context information to transform user requirements into structured task description information; the task description information includes at least the task type, task constraints, and task priority parameters; using a service planning agent to receive the task description information, and based on a preset scenario rule base and current system resource status information, searching in the feasible strategy space to generate the optimal service execution path from the current state to the task completion goal; the optimal service execution path includes the function call order, interaction node sequence, and exception handling branch strategy; and outputting the optimal service execution path as the service execution plan.

[0056] In this embodiment of the invention, step S3 can be further achieved through the collaborative work of the task understanding agent and the service planning agent in a multi-agent collaborative decision-making system. Figure 5 A schematic diagram of the multi-agent collaborative decision-making system is shown. (Reference) Figure 5 As shown, the multi-agent collaborative decision-making system is built upon a unified semantic understanding framework and a shared state space. Each functional agent works collaboratively based on a unified understanding, forming a stable and efficient service decision-making chain. The following section combines... Figure 5 The above operations are explained in detail.

[0057] Specifically, in step S2, a unified semantic state vector that integrates temporal context information is generated. Subsequently, the unified semantic state vector is first input to the task understanding agent. The task understanding agent is the first processing node in the multi-agent collaborative decision-making system, and its core function is to transform the fused multimodal semantic representation into structured task description information that can be directly used by subsequent agents. The task understanding agent performs structured modeling of user needs based on a predefined task ontology library and context memory module. The task ontology library is a predefined structured knowledge base that defines various user task types and their associated attribute frameworks that may occur in public service scenarios. Task types may include, but are not limited to: navigation query tasks, appointment processing tasks, information consultation tasks, equipment control tasks, etc. Each task type is associated with corresponding semantic slot definitions in the ontology library, clarifying the key information items required to complete that type of task.

[0058] The context memory module is used to store and manage historical interaction context information between the user and the system. In multi-turn interaction scenarios, user needs are often not expressed completely at once, but rather gradually clarified through multiple rounds of dialogue or continuous actions. The context memory module records the user intent identified in previous rounds of interaction, the entity information extracted, the completed task steps, and the user's historical behavioral characteristics, providing contextual reference for understanding the intent in the current round. The task-understanding agent responds to the unified semantic state vector of the input. The process of performing deep semantic analysis and intent decomposition can be abstractly represented by the following function mapping relationship: ; in, This is the unified semantic state vector generated by the multimodal large model at the current moment. This refers to the historical interaction context information stored by the context memory module up to the previous moment. This represents the intent parsing function of the task-understanding agent. This provides structured task description information for the output.

[0059] Specifically, the intent parsing process described above includes the following sub-steps: First, intent recognition is performed. The task-understanding agent is based on a unified semantic state vector. The semantic features contained therein, combined with the historical interaction information provided by the context memory module, are used to... The first step is to determine the task type to which the user's current core intent belongs. This process can be implemented through a classification model or matching algorithm, mapping the user's natural language expression and behavioral features to a predefined task category in the task ontology library. For example, when a user says, "I want to register for an appointment at a cardiology department," the task understanding agent identifies the task type as "appointment processing - registration." Next, semantic decomposition is performed. After determining the task type, the task understanding agent, based on the semantic slot definitions associated with that task type in the task ontology library, decomposes the semantic state vector... and historical context information Extract the required slot values. Taking the registration task as an example, the slots to be extracted may include: department name, doctor's name, consultation time, patient identity information, etc. For necessary slot information missing in the current expression, the task understanding agent can mark it as "to be clarified" for use in subsequent interaction processes. Finally, generate structured task description information. The task-understanding agent organizes the results of intent recognition and semantic decomposition into a unified, structured format, which is the task description information. It includes at least the following three components: Task Type: Indicates the task category to which the user's intent belongs, converting natural language and behavioral information into a task type field in a standard task description format; Task Constraints: Describes the restrictions that must be met to complete the task, including extracted slot information, user-specific needs, and system-side business rule constraints; Task Priority Parameter: Indicates the urgency of the current task, which can be determined comprehensively based on urgent words in the user's expression, user identity characteristics, and the timeliness requirements of the task itself. Thus, through the deep analysis of the unified semantic state vector by the task understanding agent, the system transforms fuzzy multimodal user input into structured task description information containing task type, constraints, and priority parameters. This provides a standardized input basis for the subsequent processing of service planning agents.

[0060] The task understanding agent outputs structured task description information. Subsequently, the task description information is passed to the service planning agent. The service planning agent is the core planning node in the multi-agent collaborative decision-making system. Its function is to generate the optimal service execution path from the current state to the task completion goal based on the task description information and the current system state. Internally, the service planning agent maintains or has access to two key data sources: a pre-set scenario rule base and current system resource state information.

[0061] The pre-defined scenario rule base is a knowledge base tailored to specific public service scenarios, storing the business process rules required to complete various tasks. The content of the scenario rule base can be customized according to different deployment scenarios (such as hospitals, government service halls, transportation hubs, and commercial venues), enabling the service planning agent to adapt to different scenarios. The scenario rule base defines the standard business processes corresponding to each task type, the dependencies between steps, and the principles for handling exceptions.

[0062] Current system resource status information It reflects the real-time availability of various functional interfaces and hardware terminals of the system. This information includes at least: the availability status of each functional interface, its current load, and expected response time. By sensing the system resource status, the service planning agent can avoid unavailable or high-load interfaces when planning execution paths, improving the reliability and efficiency of task execution. The service planning agent receives task description information... Then, a method combining strategy search and path planning is used to model the task execution process. The specific processing steps are as follows: First, the service planning agent parses the task description information. It includes task types, task constraints, and task priority parameters, clearly defining the core objectives and limitations of the current task.

[0063] Secondly, the service planning agent retrieves relevant business process reference information from the scenario rule base based on task type, and searches for possible execution paths in the feasible strategy space by combining task constraints and task priority parameters. Each candidate execution path defines a complete function call sequence, interaction node sequence, and exception handling branch strategy. The function call sequence specifies the functional interfaces that need to be called sequentially and their order; the interaction node sequence indicates at which stages information confirmation or supplementary interaction with the user is required; and the exception handling branch strategy defines alternative solutions or rollback mechanisms when a step fails.

[0064] Finally, the service planning agent scores and compares multiple candidate execution paths through a utility evaluation mechanism, and selects the candidate execution path with the highest comprehensive utility score as the optimal service execution path. The utility evaluation mechanism comprehensively considers multiple dimensions of factors, including but not limited to: path length, estimated execution time, resource consumption cost, task priority matching degree, and historical success rate. This selection process can be represented by the following optimization model: ;in, U represents the set of feasible strategies (i.e., the set of all generated candidate execution paths), and U represents the utility evaluation function, used to evaluate the performance of a given task description. and current system resource status Under the given conditions, the overall utility score of execution path p, The selected optimal service execution path.

[0065] Through the above planning mechanism, the service planning agent forms a complete service flowchart, including function call order, interaction nodes, and exception handling strategies, i.e., the optimal service execution path. Thus, through intelligent planning by the service planning agent based on the scenario rule base and system resource status, the system can transform abstract task description information into specific and operable optimal service execution paths. These execution paths clarify the function call order, interaction node sequence, and exception handling branch strategy, providing a complete blueprint for the subsequent instruction mapping and operation execution of the execution control agent.

[0066] The service planning agent generates the optimal service execution path. Subsequently, the system outputs the optimal service execution path as a complete service execution plan. This plan is encapsulated in a structured data format, containing all information such as the aforementioned function call order, interaction node sequence, and exception handling branch strategy. This service execution plan will be passed to subsequent processing stages. In this embodiment, the service execution plan serves two purposes: firstly, it acts as the basic input for generating the service semantic state; that is, the system can extract features such as the content type, information complexity, and urgency of the service result to be output based on the function call results defined in the service execution plan, thus forming the service semantic state; secondly, the function call order information in the service execution plan provides a basis for the execution control agent to map executable instruction sequences in subsequent steps.

[0067] Furthermore, within the multi-agent collaborative decision-making system, the task understanding agent and the service planning agent exchange states and messages via a publish-subscribe communication mechanism, with the communication format adhering to a unified protocol specification. When an agent completes its task and outputs a result, that result is published to the shared state space. Other agents that have subscribed to that state can then obtain the update and continue executing subsequent tasks. This decoupled communication method allows each agent to evolve and upgrade independently, enhancing the system's modularity and scalability, and ensuring a stable and reliable collaborative process.

[0068] Thus, by outputting the optimal service execution path as the service execution plan, the multi-agent collaborative decision-making system completes the entire process from a unified semantic state vector to an executable task plan. Compared to existing intelligent service systems that primarily rely on single models or rule engines, this embodiment, through the division of labor and collaboration between task understanding agents and service planning agents, unifies the semantic modeling of natural language and behavioral information inputted by users from multiple modalities. Furthermore, by collaborating with multiple agents to complete task decomposition and execution planning, it not only significantly improves the accuracy of semantic understanding in complex service scenarios but also effectively reduces the probability of task conflicts and execution failures, laying a solid foundation for subsequent adaptive output strategy generation and multi-modal service result presentation.

[0069] Optionally, based on a preset scenario rule base and current system resource status information, the operation of searching and generating the optimal service execution path from the current state to the task completion goal in the feasible strategy space includes: parsing the task type, task constraints, and task priority parameters contained in the task description information; retrieving candidate service process templates corresponding to the task type from the preset scenario rule base based on the task type; the candidate service process templates define the standard functional nodes required to complete the task type and the corresponding logical dependencies; obtaining the current system resource status information, which includes at least the availability status of each functional interface, the current load, and the expected response time; using the candidate service process templates as the framework, combined with the task constraints, task priority parameters, and current system resource status information, scoring and comparing multiple candidate execution paths through a preset utility evaluation function, and selecting the candidate execution path with the highest comprehensive utility score as the optimal service execution path.

[0070] In this embodiment of the invention, the service planning agent receives structured task description information output by the task understanding agent. Next, the task description information is parsed to extract the task type, task constraints, and task priority parameters. The task type indicates the category of the user's intent, such as "appointment booking," "department navigation," "information query," or "equipment control." The task type is the main index for the service planning agent to retrieve the corresponding business process template from the scenario rule base. Task constraints describe the restrictions that must be met to complete the task. Constraints may include: user-side constraints, such as extracted slot information (target department, expected time range, patient identity information, etc.), special user needs (such as accessibility requirements, priority identification for the elderly); and system-side constraints, such as business rule restrictions (department outpatient time range, appointment quota limit, required pre-application procedures, etc.). When generating the execution path, the service planning agent must ensure that the generated path satisfies all constraints. The task priority parameter indicates the urgency of the current task. Priority parameters can be quantified as discrete levels (such as high, medium, low) or continuous values ​​(such as normalized scores in the range of 0 to 1). Priority parameters influence the scoring weights of each candidate execution path in the subsequent utility evaluation function, ensuring that high-priority tasks receive better execution path allocations. Thus, by parsing the task type, constraints, and priority parameters in the task description information, the service planning agent obtains the core input elements required for path planning, providing clear goal guidance and constraint boundaries for subsequent template retrieval and path generation.

[0071] After parsing and obtaining the task type, the service planning agent retrieves the corresponding candidate service process template from a pre-set scenario rule base. The pre-set scenario rule base is a structured knowledge base oriented towards specific public service scenarios, storing standard service process templates for each task type. The content of the scenario rule base can be customized according to the specific application scenario of the system deployment (such as hospital outpatient halls, government service halls, transportation hubs, commercial complexes, etc.), enabling the service planning agent to adapt to different scenarios. The candidate service process template defines the standard functional nodes required to complete the task type and their corresponding logical dependencies. Specifically, a candidate service process template includes the following structural elements: Standard functional nodes: Define the functional processing units that typically need to be processed to complete this type of task. For example, for the "appointment registration" task type, standard functional nodes may include: identity verification node, department selection node, doctor scheduling query node, appointment time confirmation node, payment processing node, appointment voucher generation node, etc. Each functional node corresponds to a functional interface or interaction step that can be called by the system. Logical dependencies: Define the execution order and conditional branching relationships between the standard functional nodes. Logical dependencies can include: sequential dependencies (node ​​B can only execute after node A has completed), conditional branches (selecting to execute node B or node C based on the output of node A), and parallel execution (nodes B and C can execute simultaneously). These logical dependencies form the skeleton of the service flow. Therefore, by retrieving candidate service flow templates corresponding to the task type, the service planning agent obtains a standard flow reference framework for completing the current task, providing a structural foundation for subsequently generating instantiated execution paths based on specific constraints and resource states.

[0072] While or after retrieving candidate service process templates, the service planning agent obtains current system resource status information. The current system resource status information reflects the availability of various functional interfaces and hardware terminals in real time, serving as a crucial basis for the service planning agent to make optimal path selection decisions. Information should include at least the following three dimensions: Availability status of each functional interface: This indicates whether the functional interface corresponding to each standard functional node in the candidate service process template is currently available. Availability status can include: online (can be called normally), offline (unavailable), under maintenance (temporarily unavailable), etc. For currently unavailable functional interfaces, the service planning agent needs to consider alternative solutions or waiting strategies when generating execution paths. Current load: This indicates the current request processing load level of each functional interface. Load can be quantified as indicators such as the number of currently queued requests and the concurrent processing thread utilization rate. The service planning agent can use this to estimate the response latency of each functional interface and tend to select interfaces with lower loads to improve overall execution efficiency. Expected response time: Based on historical statistical data or real-time monitoring data, this estimates the average response time of each functional interface for a single request. Expected response time provides the basic data for calculating time costs in the utility evaluation function.

[0073] Furthermore, system resource status information can also include auxiliary information such as the occupancy status of various hardware terminals (e.g., printers, displays, voice broadcasting devices) and network bandwidth utilization, to support more refined resource awareness planning. Thus, by acquiring current system resource status information, the service planning agent can perceive the real-time status of the system's operating environment, ensuring that the generated execution path not only meets the task logic requirements but also conforms to the actual availability constraints of system resources, thereby improving the reliability and efficiency of task execution.

[0074] After obtaining the candidate service process template, task constraints, task priority parameters, and current system resource status information, the service planning agent uses the candidate service process template as its framework and, in conjunction with the above information, generates multiple candidate execution paths in the feasible strategy space. The process of generating candidate execution paths includes the following sub-steps: First, based on the standard functional nodes and logical dependencies defined in the candidate service process template, an initial execution path framework is constructed. Second, the initial execution path framework is instantiated and populated according to the task constraints. For slot information already defined in user-side constraints (e.g., the target department is "cardiology"), it is populated into the parameters of the corresponding functional nodes; for system-side constraints (e.g., the department's outpatient time range), it is transformed into conditional branches or time window restrictions in the path. Third, the instantiation scheme of each functional node is adjusted according to the current system resource status information. For example, when multiple functional interfaces corresponding to a certain standard functional node are available, multiple candidate path branches bound to different functional interfaces can be generated based on the current load and expected response time of each interface; when a certain functional interface is currently unavailable, candidate paths can be generated that include exception handling branches such as waiting for retry, calling backup interfaces, or requesting manual intervention.

[0075] Finally, the generation tendency of candidate paths is adjusted according to the task priority parameters. For high-priority tasks, candidate paths can be generated that include acceleration strategies such as resource reservation, parallel execution, or priority queuing. Each generated candidate execution path contains the following complete information: function call order (clearly specifying each function interface and its parameters called in sequence), interaction node sequence (indicating the steps that require information confirmation or supplementary input from the user), and exception handling branch strategy (defining alternative solutions or rollback mechanisms when each step fails). Thus, by instantiating and expanding the candidate service process template as a framework and combining it with multi-dimensional information, the service planning agent generates multiple candidate execution paths covering different execution schemes in the feasible strategy space, providing a rich selection space for subsequent path optimization.

[0076] After generating multiple candidate execution paths, the service planning agent scores and compares each candidate path using a pre-defined utility evaluation function, selecting the candidate path with the highest overall utility score as the optimal service execution path. The pre-defined utility evaluation function U is a multi-objective evaluation function used to quantitatively evaluate the performance of a given task description. and current system resource status Under certain conditions, the overall utility score of a candidate execution path p is calculated. The utility evaluation function considers factors from multiple evaluation dimensions, which may include: Execution efficiency dimension: evaluating the estimated completion time of the candidate execution path. This dimension comprehensively considers factors such as path length (the number of functional nodes required to be executed), the expected response time of each functional node, and the opportunity for parallel execution. The shorter the estimated completion time, the higher the score for this dimension.

[0077] Resource Consumption Dimension: Evaluates the resource consumption of candidate execution paths. This dimension comprehensively considers factors such as the call cost of each functional interface, computational resource consumption, and network bandwidth usage. The lower the resource consumption, the higher the score in this dimension. Reliability Dimension: Evaluates the success probability of candidate execution paths. This dimension comprehensively considers factors such as the historical success rate of each functional interface, current availability status, and the completeness of exception handling branches. The higher the expected success rate, the higher the score in this dimension. Task Priority Matching Degree: Evaluates the responsiveness of candidate execution paths to task priority parameters. For high-priority tasks, candidate paths that include acceleration strategies (such as resource reservation and priority queuing) receive higher scores in this dimension. User Experience Dimension: Evaluates the impact of candidate execution paths on user experience. This dimension comprehensively considers factors such as the number of interaction nodes, waiting time distribution, and information confirmation frequency. The better the expected user experience, the higher the score in this dimension.

[0078] The utility evaluation function weights and synthesizes the scores of each evaluation dimension, and the weighting coefficients can be configured according to the actual needs of the system deployment scenario. For example, in a hospital emergency room scenario, the weights of execution efficiency and reliability dimensions can be appropriately increased; in a regular consultation scenario, the weights of resource consumption and user experience dimensions can be relatively increased. The service planning agent will generate each candidate execution path... Enter the utility evaluation functions sequentially and calculate their overall utility score. After scoring all candidate paths, the service planning agent compares the scores and selects the candidate execution path with the highest overall utility score as the optimal service execution path. Its mathematical expression has been given above and will not be repeated here.

[0079] Therefore, by using a utility evaluation function to quantitatively score and compare multiple candidate execution paths across multiple dimensions, the service planning agent can automatically select the optimal service execution path from numerous feasible solutions. This ensures that the generated execution plan achieves a comprehensive and optimal balance in terms of efficiency, resources, reliability, and user experience, providing high-quality planning output for subsequent service execution. Through the aforementioned collaborative operations, the service planning agent completes a full planning process, from parsing task description information, retrieving process templates, perceiving resource status, generating candidate paths, to selecting the optimal path. The final output includes the optimal service execution path, encompassing the function call order, interaction node sequence, and exception handling branch strategy, which is then passed on as a core component of the service execution plan to subsequent processing stages.

[0080] Optionally, the operation of obtaining the user's interaction capability feature parameters and generating a corresponding adaptive output strategy based on the interaction capability feature parameters and the service semantic state of the service execution plan includes: obtaining the user's current interaction capability feature parameters; the interaction capability feature parameters include at least visual perception capability parameters, auditory perception capability parameters, action operation capability parameters, and cognitive understanding capability parameters; obtaining the service semantic state corresponding to the service execution plan; the service semantic state is used to characterize the content type, information complexity, and urgency of the service result to be output; determining at least one applicable output mode from a preset candidate output mode set based on the interaction capability feature parameters and the service semantic state; the candidate output mode set includes at least voice broadcast mode, graphic and text display mode, gesture guidance mode, and augmented reality display mode; generating an adaptive output strategy according to the determined output mode; the adaptive output strategy defines the combination of output modes used when presenting the service result to the user and the collaborative relationship between each output mode.

[0081] In this embodiment of the invention, the system first obtains the interaction capability feature parameters of the current user. These interaction capability feature parameters are used to comprehensively evaluate the user's ability level in visual perception, auditory perception, motor operation, and cognitive understanding, forming a personalized interaction capability description vector. This vector can be represented as: ; in, Parameters representing visual perception ability. This represents parameters of auditory perception ability. Indicates the parameter of action operation capability. These represent parameters indicating cognitive comprehension ability. The values ​​of all the above parameters can be normalized to the interval [0,1], with higher values ​​indicating stronger abilities in the corresponding dimension.

[0082] The aforementioned interactive capability characteristic parameters can be automatically estimated by analyzing users' historical interaction behavior data, device usage habit data, and real-time perception information. Specifically, the system first constructs a set of user capability characteristics. This set of features includes, but is not limited to: device usage features (such as user-preferred interaction device types and frequently used functional modules), real-time behavioral characteristics (e.g., operation speed, gesture stability, voice clarity), environmental characteristics (such as current ambient noise levels, light intensity) and historical interaction characteristics (Such as the efficiency of completing historical tasks, the frequency of repeated information requests). That is: ; Subsequently, the system uses a feature fusion model to comprehensively process the aforementioned multidimensional features, generating a comprehensive capability feature vector. : ;in, For the feature fusion weight matrix, This is a bias term.

[0083] Based on this, the system calculates four types of interaction capability parameters: visual perception capability parameters. This reflects a user's perception of visual output formats such as text and images, and video. This parameter can be estimated by analyzing indicators such as the user's response speed to visual information, font size preferences, and reliance on visual cues in past interactions. Its calculation method can be expressed as follows: ,in For normalization function, Estimating the weight vector for visual ability. Auditory perception ability parameters. This reflects a user's perception of auditory output formats such as voice broadcasts and prompts. This parameter can be estimated by analyzing indicators such as the user's speech recognition accuracy in noisy environments, volume adjustment habits, and whether they frequently request repeated broadcasts. Its calculation method can be expressed as follows: ,in Estimating the weight vector for auditory ability. Motor performance parameters. This reflects a user's ability to perform actions such as touch, gestures, and button presses. This parameter can be estimated by analyzing indicators such as user operation speed, operation trajectory stability, accidental touch frequency, and gesture recognition success rate. Its calculation method can be expressed as follows: ,in Estimating the weight vector for motor ability. Cognitive comprehension ability parameters. This parameter reflects a user's ability to understand complex information and their task execution efficiency. It can be estimated by analyzing indicators such as the user's historical task completion time, frequency of repeated information requests, and preference for simplified interfaces. Its calculation method can be expressed as follows: ,in Estimate the weight vector for cognitive ability. The above normalization function... The Sigmoid function can be used to ensure that all output capability parameters are within the range [0,1]. Thus, by obtaining the user's four-dimensional interaction capability characteristic parameters, the system establishes a quantitative description of the current user's interaction capability level, providing a core decision-making basis on the user side for the subsequent generation of adaptive output strategies.

[0084] Next, the system obtains the service semantic state corresponding to the service execution plan generated in step S3. The service semantic state characterizes features such as the content type, information complexity, and urgency of the service result to be output. The service semantic state can be extracted from the structured information of the service execution plan. For example, based on the task type and function call results defined in the service execution plan, the system can determine the content type of the service result to be output: navigation results are spatial location information, appointment processing results are process status information, and information consultation results are knowledge-based text information. Information complexity can be quantitatively assessed based on factors such as the amount of information contained in the service result, the depth of its structural hierarchy, and whether the user needs to perform multiple steps of understanding. Urgency can be comprehensively determined based on task priority parameters, the timeliness requirements of the service result, and the current system resource status. Thus, by obtaining the service semantic state corresponding to the service execution plan, the system grasps the information characteristics of the service result to be output, providing a core decision-making basis on the task side for the subsequent selection of the output mode.

[0085] In obtaining user interaction capability characteristic parameters Service semantic state Subsequently, based on the above two types of information, the system determines at least one applicable output mode from a preset set of candidate output modes. This invention employs a capability-constrained guided multimodal strategy generation model. This model determines the optimal output combination by calculating the adaptation scores of different output modes.

[0086] The system first defines a set of candidate output modes. : ;in, Indicates voice broadcast mode. Indicates the text and image display mode. Indicates action guidance mode. This indicates the augmented reality display mode.

[0087] The system then calculates the adaptation score for each output method. For the i-th output mode, its adaptation score is... The calculation formula is: ;in, This is the current user's interactive capability feature parameter vector. For the current service semantic state, These are environmental parameters (such as environmental noise levels, light intensity, and other external factors). For the user capability fit scoring function, For the service semantic matching score function, This is a scoring function for environmental constraint matching. , , These are preset weighting coefficients used to adjust the contribution ratio of the above three factors to the final fit score. Specifically, the meanings of each scoring function are as follows: User capability suitability score. Based on the user's four-dimensional interaction capability characteristic parameters, this score assesses how well the output mode suits the user's current capability level. For example, when the user's auditory capability parameter is low, the suitability score for the voice broadcast mode decreases; when the user's visual capability parameter is low, the suitability score for the graphic display mode decreases. Service semantic matching score. Based on the content type and information complexity represented by the current service semantic state, this score assesses the degree of matching between the output mode and the information features of the service result to be presented. For example, spatial location information for navigation tasks is more suitable for action guidance mode and augmented reality display mode; knowledge-based text for information consulting is more suitable for voice broadcast mode and graphic display mode. Environmental constraint matching score. This assesses the applicability of the output mode under current environmental conditions based on existing environmental parameters. For example, the matching score for the voice broadcast mode decreases when the ambient noise level is high; the matching score for the graphic display mode decreases when the ambient lighting is insufficient.

[0088] The system selects the set of candidate output modes. After calculating the adaptation score for each output mode, the combination of output modes with the highest overall adaptation score is selected as the optimal output strategy. ; Through the above-mentioned adaptation scoring calculation and optimal selection mechanism, the system can automatically determine the optimal combination of output modes under the current user capability conditions, service semantic features and environmental constraints.

[0089] In the specific output process, the system first performs semantic reconstruction and hierarchical expression processing on the service results, breaking down complex information into multi-level, progressive content units: The system uses a hierarchical semantic decomposition algorithm to transform service results into an information structure suitable for accessible output. The algorithm flow is as follows: (1) Semantic Unit Extraction: The system extracts the core semantic units of the task from the service result to be output. Let the original service result be R, and the set of extracted core semantic units be T: ; (2) Task ontology matching: The system matches the extracted semantic units with a preset task ontology library to identify key steps of the task. Let the task ontology library be... The matching result is : Through ontology matching, the system identifies the key steps of the task and their logical sequence.

[0090] (3) Semantic granularity control: The system controls the semantic granularity based on the user's cognitive ability parameters. Dynamically adjust information granularity. The lower the cognitive ability, the finer the information granularity and the more detailed the information breakdown. The formula for calculating the semantic granularity factor g is: ;in, Here, e represents the user's cognitive comprehension ability, and g is a natural constant. The granularity factor g determines the level of detail in information breakdown. The lower the value, the smaller the g value, and the more finely the information is broken down.

[0091] (4) Task Priority Ranking: When multiple task intents need to be presented, the system selects the task to be presented first based on context relevance calculation. For the i-th task, its priority score is... The calculation formula is: ;in, This indicates the degree of association between the i-th task and the current interaction context. This indicates the urgency level of the i-th task. and These are the preset weighting coefficients.

[0092] (5) Generate hierarchical information structure: Based on the semantic granularity control results and task priority sorting results, the system finally forms a hierarchical information structure. : ;in, This is the core task information layer (such as task completion status and main conclusions). This is the operation step information layer (such as specific operation instructions and explanations of subsequent steps). To supplement the information layer (such as precautions and supplementary information), this hierarchical structure allows information to be presented to the user progressively from core to supplementary information. This approach avoids information overload at once and improves the comprehension efficiency for elderly users and users with cognitive impairments. Subsequently, the system maps the reconstructed information to the multi-terminal output module according to the output strategy.

[0093] After determining the applicable output mode, the system generates an adaptive output strategy based on the determined output mode. The adaptive output strategy defines the combination of output modes used to present service results to users and the collaborative relationships between these modes. The output mode combination specifies which output modes will be used for this service result presentation and their relative importance. For example, for elderly users with weak vision but normal hearing, the output strategy can define a combination of primary voice broadcast mode and secondary text / image display mode; for users with limited operational capabilities, the output strategy can define a combination of primary gesture guidance mode and simultaneous voice broadcast prompts. The collaborative relationships between output modes define the coordination rules when multiple output modes are used simultaneously or sequentially. Collaborative relationships can include: synchronous presentation (e.g., voice broadcast and text / image display occur simultaneously, allowing users to listen and view simultaneously), sequential presentation (e.g., first broadcasting summary information via voice, then displaying detailed information via text / image display), and complementary presentation (e.g., gesture guidance indicates the operation location, while voice broadcast simultaneously explains the operation content), etc.

[0094] In terms of voice interaction output, the system employs adaptive speech synthesis technology to dynamically adjust speech rate, volume, tone, and repetition frequency. For elderly and visually impaired users, a slow, clear playback mode is prioritized, and it supports automatic repetition of key content and a voice confirmation feedback mechanism. Regarding text and image display, the system dynamically adjusts font size, contrast, and color mode based on accessibility display standards, supporting high-contrast mode, large font size mode, and simplified interface mode to ensure readability for visually impaired users.

[0095] Thus, by generating an adaptive output strategy that includes combinations of output modes and collaborative relationships, the system provides explicit execution instructions for the subsequent multimodal presentation operations of the control agent. Through the aforementioned collaborative operations, the system completes the entire process from user capability assessment, service semantic perception, output mode selection to adaptive strategy generation, achieving dynamic adaptive adjustment of the interaction form and effectively reducing the understanding and operation threshold of the intelligent service system for special user groups.

[0096] Optionally, based on the adaptive output strategy, the operation of presenting the service results to the user through a combination of multiple output modalities by executing the control agent includes: receiving the adaptive output strategy through the control agent and parsing the output mode combination methods and the coordination relationships between the output modes defined therein; the output mode combination methods include at least one or more combinations of voice broadcast mode, graphic and text display mode, gesture guidance mode, and augmented reality display mode; mapping the service results to be presented to the corresponding driving instruction sequences of each output modality according to the output mode combination methods; and executing the driving instruction sequences of each output modality synchronously or sequentially according to the coordination relationships through a unified multi-terminal output interface to complete the multimodal presentation of the service results.

[0097] In this embodiment of the invention, the execution control agent first receives the adaptive output strategy generated in step S4. The adaptive output strategy is transmitted in a structured data format, which defines the combination of output modes and the coordination relationships between them. The execution control agent parses the received adaptive output strategy and extracts the defined combination of output modes. The combination of output modes specifies the type of output mode to be used in this service result presentation, which includes at least one or more combinations of voice broadcast mode, graphic and text display mode, gesture guidance mode, and augmented reality display mode. Simultaneously, the execution control agent extracts the coordination relationships between the output modes, clarifying whether each mode is presented synchronously, sequentially, or complementaryly. Thus, by parsing the adaptive output strategy, the execution control agent obtains all the control parameters and coordination rules required to drive multi-terminal presentation.

[0098] After parsing the output mode combination, the execution control agent maps the service result content to be presented to the corresponding drive instruction sequence of each output mode according to the determined output modes. The service result content is the information to be presented to the user after the service execution plan is executed. In the mapping process, the execution control agent first performs semantic reconstruction and hierarchical expression processing on the service result. The system uses a hierarchical semantic decomposition algorithm to convert the service result into an information structure suitable for accessible output.

[0099] Specifically, the system extracts the core semantic units from the service results and analyzes them based on the user's cognitive and comprehension ability parameters. The information granularity is dynamically adjusted; the lower the cognitive ability, the finer the information granularity. This ultimately forms a hierarchical information structure. For example, the first level is core task information (e.g., "Registration successful"), the second level is operational steps (e.g., "Please go to the cardiology department on the second floor of the outpatient building tomorrow at 9:00 AM"), and the third level is supplementary information (e.g., "Please bring your ID card and medical insurance card"). This method avoids information overload at once and improves the comprehension efficiency for elderly users and those with cognitive impairments.

[0100] Subsequently, the execution control agent maps the layered processed service results to the corresponding driver instruction formats for each output modality. For example, for the voice broadcast mode, the text content is mapped to speech synthesis driver instructions, including the broadcast text, speech rate parameters, volume parameters, and repetition count; for the image and text display mode, the text and image content is mapped to interface rendering driver instructions, including the displayed content, font size, contrast parameters, and layout style; for the gesture guidance mode, the spatial location and operation steps are mapped to dynamic indicator driver instructions, including arrow direction, flashing frequency, and synchronized prompt text; and for the augmented reality display mode, virtual information and spatial anchors are mapped to AR rendering driver instructions, including virtual label content, overlay position, and display duration. Thus, by mapping the service results to the driver instruction sequences of each output modality, the execution control agent completes the conversion from abstract output strategies to specific device-executable instructions.

[0101] After generating the drive instruction sequence for each output modality, the execution control agent executes the drive instruction sequence for each output modality synchronously or sequentially according to the cooperative relationship through a unified multi-terminal output interface, thus completing the multi-modal presentation of the service result. The unified multi-terminal output interface is a standardized device driver abstraction layer that shields the differences in interface protocols, data formats, and calling methods between different hardware terminals. This interface interfaces with various output devices, including but not limited to: voice broadcasting devices (speakers, directional speakers), graphic display devices (LCD screens, e-ink screens, mobile terminal screens), gesture guidance devices (dynamic indicator light arrays, projection guidance devices), and augmented reality display devices (AR glasses, mobile terminal AR camera views), etc.

[0102] The execution control agent controls the execution order of the driving instruction sequence of each output modality according to the timing rules defined in the cooperative relationship: for synchronous presentation mode, the driving instructions of multiple output modalities are triggered simultaneously to achieve multi-channel parallel output; for sequential presentation mode, the driving instructions of each modality are triggered one by one in a preset order; for complementary presentation mode, the driving instructions of each modality partially overlap in time and complement each other in content to form a cooperative information transmission effect.

[0103] In terms of voice interaction output, the execution control agent can dynamically adjust the speech rate, volume, and repetition frequency of voice broadcasts based on the user's hearing ability parameters. For elderly and visually impaired users, a slow, clear broadcast mode is prioritized, and automatic repetition of key content and voice confirmation feedback mechanisms are supported. Regarding text and image display, the execution control agent dynamically adjusts font size, contrast, and color mode based on accessibility display standards, supporting high-contrast mode, large font size mode, and simplified interface mode to ensure readability for visually impaired users. For gesture guidance, the execution control agent can combine a camera and depth sensor to recognize the user's posture in real time and guide the user through the operation process using dynamic arrows, flashing indicators, and synchronized voice prompts, reducing the operational difficulty for users with physical disabilities. Thus, by driving multiple terminal output devices to present service results in a combination of various output modalities, the system can deliver information in a way that is most easily perceived and understood by users, effectively reducing the understanding and operation barriers for users of different ability levels and improving the accessibility and usability of the intelligent service system.

[0104] Optionally, the multi-agent collaborative decision-making system also includes a feedback evaluation agent for monitoring and evaluating the service presentation effect; and the method further includes: during and after the multimodal presentation, the feedback evaluation agent collects user response behavior data, attention duration data, and emotional feedback data in response to the multimodal presentation; the feedback evaluation agent comprehensively analyzes the collected multimodal presentation data to generate a presentation effect evaluation result, and feeds the presentation effect evaluation result back to the module that generates an adaptive output strategy for iterative optimization of the subsequent output strategy.

[0105] In this embodiment of the invention, the multi-agent collaborative decision-making system, in addition to including a task understanding agent, a service planning agent, and an execution control agent, further introduces a feedback evaluation agent to construct a closed-loop feedback mechanism of "output-perception-evaluation-optimization". The specific operation of the feedback evaluation agent is described in detail below. Specifically, during and after the multimodal presentation, the feedback evaluation agent collects user feedback data on the multimodal presentation through multiple perception channels. The types of data collected include: Response behavior data: refers to the observable behavioral responses of users to the service result presentation. This includes, but is not limited to, whether the user completed the expected operation according to the instructions, the accuracy rate of the operation, the time of operation completion, the number of repeated requests, and the task abandonment rate. Response behavior data can be obtained from sources such as touchscreen operation logs, camera behavior recognition, and sensor data. Attention duration data: refers to the distribution of user attention duration for each output modality. This includes, but is not limited to, the duration of the user's gaze at the screen, the completeness of listening to voice broadcasts, and the response delay to gesture guidance. Attention duration data can be obtained through eye tracking, facial orientation detection, and interaction event timestamp analysis. Emotional feedback data refers to information about the emotional state displayed by users during interaction. This includes, but is not limited to, facial expression features, voice tone features, and body language features, used to infer the user's satisfaction, confusion, or anxiety level. Emotional feedback data can be extracted using facial expression recognition models, voice sentiment analysis models, etc. The feedback evaluation agent continuously collects the above data during multimodal presentation and summarizes it after presentation to form a complete perception record of the service presentation effect.

[0106] Then, the feedback evaluation agent comprehensively analyzes the collected response behavior data, attention duration data, and emotional feedback data to generate a presentation effect evaluation result. The analysis process may include the following dimensions: Task completion evaluation: Based on response behavior data, it determines whether the user successfully understood and completed the operation guided by the service result. Task completion can be quantified as completion rate, accuracy rate, and other indicators. Interaction efficiency evaluation: Based on attention duration data and response behavior data, it assesses the time cost required for the user to understand the service result and respond. Interaction efficiency can be quantified as task completion time, number of repeated information requests, and other indicators. User satisfaction evaluation: Based on emotional feedback data and response behavior data, it comprehensively infers the user's satisfaction with this service interaction. Satisfaction can be quantified as the proportion of positive emotions, task abandonment rate, and other indicators. The feedback evaluation agent weights and synthesizes the evaluation results from each dimension to generate a presentation effect evaluation result. This evaluation result is output in the form of structured quantitative indicators, reflecting the overall effectiveness level of this service presentation.

[0107] After generating the presentation effect evaluation results, the feedback evaluation agent uses these results as a feedback signal and transmits them to the module that generates the adaptive output strategy. This feedback signal is used for iterative optimization of subsequent output strategies. Specifically, the adaptive output strategy generation module can calibrate and update the estimation model of user interaction capability characteristic parameters based on the presentation effect evaluation results of multiple historical interactions, making the capability evaluation more accurate. Simultaneously, it can adjust the parameters of the decision model for output mode selection, making the output strategies generated for similar users or similar tasks more aligned with actual needs. Thus, by introducing the feedback evaluation agent and constructing a closed-loop feedback mechanism of "output-perception-evaluation-optimization," the system can continuously monitor and model user interaction behavior and service effects, achieving long-term self-optimization and evolution of interaction strategies and service processes. The system can continuously adjust its service mode according to changes in user habits and capabilities, thus possessing continuous learning and adaptive evolution capabilities, ensuring the stability and advancement of the system during long-term operation.

[0108] Furthermore, according to this embodiment, a storage medium is also provided. The storage medium includes a stored program, wherein, during program execution, a processor executes any of the methods described above.

[0109] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0110] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0111] Example 2

[0112] Figure 6 This illustration shows an intelligent interaction system for public service scenarios based on a multimodal large model, corresponding to the method described in Embodiment 1. (Reference) Figure 6As shown, the system includes: a multimodal data acquisition module 610, used to acquire multimodal user input data in public service scenarios, the multimodal user input data including at least voice data, image data, and behavior perception data; and a semantic state vector generation module 620, used to input the multimodal user input data into a fine-tuned multimodal large model for unified semantic parsing, generating a unified semantic state vector that integrates temporal contextual information; wherein, the multimodal large model is obtained based on a pre-trained base model through multimodal expansion and scenario-based fine-tuning, used to map different modal features to a unified semantic space and perform cross-modal fusion; The service execution plan generation module 630 is used to generate a service execution plan based on a unified semantic state vector and through a multi-agent collaborative decision-making system. The multi-agent collaborative decision-making system includes a task understanding agent for parsing user intent, a service planning agent for generating execution paths, and an execution control agent for driving multi-terminal presentation. The adaptive output strategy generation module 640 is used to obtain the user's interaction capability feature parameters and generate a corresponding adaptive output strategy based on the interaction capability feature parameters and the service semantic state of the service execution plan. The service result presentation module 650 is used to present the service result to the user through a combination of multiple output modalities according to the adaptive output strategy and through the execution control agent. The multiple output modalities include at least voice output, graphic and text display, and action guidance.

[0113] It should be noted that the intelligent interaction system for public service scenarios based on a multimodal large model provided in this embodiment can realize all the functions and steps in the above method embodiments, solve the same technical problems, and achieve the same technical effects. The similarities will not be repeated here.

[0114] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0115] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The system embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0117] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0118] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0119] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0120] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for intelligent interaction in public service scenarios based on a multimodal large model, characterized in that, include: Acquire multimodal user input data in public service scenarios, wherein the multimodal user input data includes at least voice data, image data, and behavioral perception data; The multimodal user input data is input into a fine-tuned multimodal large model for unified semantic parsing to generate a unified semantic state vector that integrates temporal context information. The multimodal large model is obtained by multimodal expansion and scenario-based fine-tuning based on a pre-trained base model and is used to map different modal features to a unified semantic space and perform cross-modal fusion. Based on the unified semantic state vector, a multi-agent collaborative decision-making system is used to process tasks and generate service execution plans. The multi-agent collaborative decision-making system includes a task understanding agent for parsing user intent, a service planning agent for generating execution paths, and an execution control agent for driving multi-terminal presentation. Obtain the user's interaction capability feature parameters, and generate a corresponding adaptive output strategy based on the interaction capability feature parameters and the service semantic state of the service execution scheme; as well as Based on the adaptive output strategy, the execution control agent presents the service results to the user through a combination of multiple output modalities; the multiple output modalities include at least voice output, graphic display and action guidance.

2. The method according to claim 1, characterized in that, The multimodal large model includes a multimodal feature mapping module, a dynamic gating cross-modal fusion module, and a temporal context modeling module; and the operation of inputting the multimodal user input data into the fine-tuned multimodal large model for unified semantic parsing to generate a unified semantic state vector that integrates temporal context information includes: The multimodal feature mapping module encodes the user input data for each modality and maps the encoded modal feature vectors to a unified semantic space with the same word vector dimension as the pre-trained base model, thereby obtaining a unified semantic representation for each modality. The dynamic gating cross-modal fusion module dynamically calculates the attention weights of each modality based on the current task context, and performs weighted fusion of the unified semantic representation of each modality based on the attention weights to obtain the cross-modal fusion representation at the current moment. The temporal context modeling module performs sequence modeling on the cross-modal fusion representation of multiple consecutive moments, including the current moment, and extracts historical temporal context information. The cross-modal fusion representation of the current moment, the historical temporal context information, and the context embedding of the pre-trained base model are concatenated and linearly transformed to generate a unified semantic state vector that integrates temporal context information.

3. The method according to claim 2, characterized in that, The operation of dynamically calculating the attention weights of each modality based on the current task context through the dynamically gated cross-modal fusion module includes: Obtain the unified semantic representation vectors of each modality output by the multimodal feature mapping module; The unified semantic representation vector of each modality is multiplied by a preset query vector to obtain the original relevance score for each modality; wherein the query vector is used to characterize the semantic attention tendency of the current task context. The original relevance scores of each modality are normalized so that the sum of the attention weights of all modalities is one, and the attention weight of any modality is proportional to the exponential function value of the original relevance score, thus obtaining the attention weight corresponding to each modality.

4. The method according to claim 1, characterized in that, Based on the unified semantic state vector, the operation of generating service execution plans through a multi-agent collaborative decision-making system for task processing includes: The task-understanding agent performs intent recognition and semantic decomposition on the unified semantic state vector, and combines historical interaction context information to transform user needs into structured task description information; the task description information includes at least task type, task constraints, and task priority parameters. The service planning agent receives the task description information and, based on a preset scenario rule base and current system resource status information, searches and generates the optimal service execution path from the current state to the task completion target in the feasible strategy space. The optimal service execution path includes the function call order, the interaction node sequence, and the exception handling branch strategy. The optimal service execution path is output as the service execution scheme.

5. The method according to claim 4, characterized in that, Based on a pre-defined scenario rule base and current system resource status information, the system searches and generates the optimal service execution path from the current state to the task completion goal within the feasible strategy space, including: Parse the task type, task constraints, and task priority parameters contained in the task description information; Based on the task type, a candidate service process template corresponding to the task type is retrieved from the preset scenario rule base; the candidate service process template defines the standard functional nodes required to complete the task type and the corresponding logical dependencies. Obtain current system resource status information, which includes at least the availability status of each functional interface, current load status, and expected response time; Using the candidate service process template as a framework, and combining the task constraints, task priority parameters, and current system resource status information, multiple candidate execution paths are scored and compared using a preset utility evaluation function. The candidate execution path with the highest comprehensive utility score is selected as the optimal service execution path.

6. The method according to claim 1, characterized in that, The operation of obtaining user interaction capability feature parameters and generating corresponding adaptive output strategies based on the interaction capability feature parameters and the service semantic state of the service execution scheme includes: Obtain the user's current interaction capability characteristic parameters; the interaction capability characteristic parameters include at least visual perception capability parameters, auditory perception capability parameters, motor operation capability parameters, and cognitive comprehension capability parameters; Obtain the service semantic state corresponding to the service execution plan; the service semantic state is used to characterize the content type, information complexity, and urgency of the service result to be output. Based on the interactive capability feature parameters and the service semantic state, at least one applicable output mode is determined from a preset set of candidate output modes; the set of candidate output modes includes at least a voice broadcast mode, a graphic display mode, a gesture guidance mode, and an augmented reality display mode. Based on the determined output mode, the adaptive output strategy is generated; the adaptive output strategy defines the combination of output modes used when presenting service results to the user and the collaborative relationship between each output mode.

7. The method according to claim 1, characterized in that, Based on the adaptive output strategy, the execution control agent presents the service results to the user through a combination of multiple output modalities, including: The execution control agent receives the adaptive output strategy and parses the output mode combination method and the cooperative relationship between each output mode as defined therein; the output mode combination method includes at least one or more combinations of voice broadcast mode, graphic display mode, gesture guidance mode and augmented reality display mode; The execution control agent maps the service results to be presented to the corresponding output modal drive instruction sequences according to the output mode combination method. Through the execution control agent, via a unified multi-terminal output interface, the driving instruction sequence of each output mode is executed synchronously or sequentially according to the cooperative relationship, thereby completing the multimodal presentation of the service result.

8. The method according to claim 7, characterized in that, The multi-agent collaborative decision-making system further includes a feedback evaluation agent for monitoring and evaluating the service presentation effect; and the method further includes: During and after the multimodal presentation, the feedback evaluation agent collects user response behavior data, attention duration data, and emotional feedback data in response to the multimodal presentation. The feedback evaluation agent comprehensively analyzes the collected data of the multimodal presentation, generates a presentation effect evaluation result, and feeds the presentation effect evaluation result back to the module that generates the adaptive output strategy for subsequent iterative optimization of the output strategy.

9. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, the method described in any one of claims 1 to 8 is performed by a processor.

10. An intelligent interactive system for public service scenarios based on a multimodal large model, characterized in that, include: A multimodal data acquisition module is used to acquire multimodal user input data in public service scenarios. The multimodal user input data includes at least voice data, image data, and behavioral perception data. The semantic state vector generation module is used to input the multimodal user input data into the fine-tuned multimodal large model for unified semantic parsing, and generate a unified semantic state vector that integrates temporal context information; wherein, the multimodal large model is obtained by multimodal expansion and scenario-based fine-tuning based on a pre-trained base model, and is used to map different modal features to a unified semantic space and perform cross-modal fusion. The service execution plan generation module is used to generate a service execution plan based on the unified semantic state vector and through a multi-agent collaborative decision-making system. The multi-agent collaborative decision-making system includes a task understanding agent for parsing user intent, a service planning agent for generating execution paths, and an execution control agent for driving multi-terminal presentation. An adaptive output strategy generation module is used to obtain user interaction capability feature parameters and, based on the interaction capability feature parameters and the service semantic state of the service execution scheme, generate a corresponding adaptive output strategy; and The service result presentation module is used to present the service result to the user through a combination of multiple output modalities by the execution control agent, based on the adaptive output strategy; the multiple output modalities include at least voice output, graphic display and action guidance.

Images