A multimodal AI digital human service system for transportation hubs

Abstract

The application discloses a kind of multi-modal AI digital human service systems for traffic hub, belong to artificial intelligence technical field, it includes local knowledge base module and is realized dynamic update by versioning difference loading;Multi-modal sensing pre-processing module carries out signal fusion and noise suppression to interactive request;Intention analysis module completes multilingual recognition and visual analysis;Knowledge base matching module carries out similarity determination and selects path;Large model module generates and checks safe compliance answer when not covered;Multi-modal synthesis module synchronously generates voice and animation;Session management module records complete log.The application adopts multi-modal signal fusion and noise suppression technology, combines local knowledge base difference update and large model dynamic generation safety cooperative mechanism, can realize multilingual intelligent interaction service under the complex environment of traffic hub, improve the accuracy and response efficiency of service.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multimodal AI digital human service system for transportation hubs. Background Technology

[0002] A multimodal AI digital human service system is an intelligent interactive terminal that integrates multiple technologies such as artificial intelligence, computer graphics, speech recognition, and synthesis. In specific commercial service scenarios such as transportation hubs, this system typically appears in the form of a virtual customer service representative, aiming to provide passengers with automated services such as information inquiry, route guidance, and ticketing consultation. This alleviates the pressure on human customer service representatives, improves service efficiency, and enhances the travel experience for passengers, making it an important component of the fields of intelligent transportation and commercial automation services.

[0003] In related technologies, Chinese invention patent CN120317879B discloses a multimodal digital human interaction method and system based on a large language model, including: acquiring video data, audio signals, and text data of a single dialogue when a user interacts with a digital human; extracting the envelope of the audio signal to determine the emotion assessment value and semantic contrast of the single dialogue; identifying the emotions contained in each frame of the video data to form an emotion sequence of the single dialogue; obtaining the semantic deviation of the single dialogue; obtaining the semantic guidance value of the single dialogue; and generating a corresponding emotional dialogue by combining an emotional dialogue generation model.

[0004] However, the aforementioned existing technical solutions have the following technical shortcomings: While introducing multimodal interaction, they fail to specifically address the issues of strong noise and multi-voice interference in scenarios such as train stations and airports. Their solutions focus on emotion guidance and generation, lacking key technologies such as beamforming noise reduction, lip movement feature extraction, and adaptive weighted fusion of audiovisual elements. Existing solutions overly rely on the generation capabilities of large models, lacking a dual-path decision-making mechanism that prioritizes accurate matching from the local knowledge base while supplementing with large model generation. Existing technologies are open-loop systems, focusing on the experience of a single interaction while neglecting the quantitative evaluation of long-term service quality. The system cannot automatically analyze user dwell time and the frequency of repeated questions through session logs, and cannot accurately locate and mark weak points in the knowledge base with low matching rates and poor user experience. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a multimodal AI digital human service system for transportation hubs. It employs multimodal signal fusion and noise suppression technologies, combined with a local knowledge base differential update and a large model dynamic generation security collaboration mechanism, enabling multilingual intelligent interactive services in the complex environment of transportation hubs, thereby improving service accuracy and response efficiency.

[0006] The above objectives can be achieved through the following approach: A multimodal AI digital human service system for transportation hubs includes a local knowledge base module that dynamically updates its content through versioned differential loading; a multimodal perception preprocessing module that performs signal fusion and noise suppression on interactive requests; an intent parsing module that performs multilingual recognition and visual analysis; a knowledge base matching module that performs similarity judgment and selects a path; a large model module that generates and verifies safe and compliant answers when not covered; a multimodal synthesis module that synchronously generates speech and animation; and a conversation management module that records complete logs.

[0007] Furthermore, the local knowledge base module includes: a version interaction and incremental acquisition unit, a security verification and conflict detection unit, and a differential loading and library update unit; wherein, the version interaction and incremental acquisition unit is used to periodically send the current local knowledge base version identifier to the backend server and receive the corresponding incremental update package; the security verification and conflict detection unit, connected to the version interaction and incremental acquisition unit, is used to perform digital signature verification and content conflict detection on the incremental update package to obtain the verified update content; the differential loading and library update unit, connected to the security verification and conflict detection unit, is used to replace the corresponding entries in the local knowledge base with the verified update content in a differential manner to generate an updated local knowledge base.

[0008] Furthermore, the multimodal perception preprocessing module includes: a multi-source heterogeneous data acquisition unit, a spatiotemporal feature enhancement and extraction unit, and a cross-modal temporal alignment and fusion unit; wherein, the multi-source heterogeneous data acquisition unit, connected to the differential loading and library update unit, is used to acquire speech signals and synchronous visual signals to obtain raw multi-source data; the spatiotemporal feature enhancement and extraction unit, connected to the multi-source heterogeneous data acquisition unit, is used to perform beamforming noise reduction and lip movement feature extraction on the raw multi-source data to obtain enhanced speech features and visual auxiliary features; the cross-modal temporal alignment and fusion unit, connected to the spatiotemporal feature enhancement and extraction unit, is used to perform temporal alignment and fusion of the enhanced speech features and visual auxiliary features to generate standardized interactive data.

[0009] Furthermore, the intent parsing module includes: a speech recognition and decoding unit, an audiovisual fusion error correction unit, and an intent parsing text generation unit; wherein, the speech recognition and decoding unit is connected to the cross-modal temporal alignment and fusion unit, and is used to perform acoustic model decoding and language model correction on the speech portion in the standardized interaction data to obtain a candidate text sequence; the audiovisual fusion error correction unit is connected to the speech recognition and decoding unit, and is used to correct easily confused pronunciations in the candidate text sequence using the visual auxiliary features to obtain a corrected text sequence; the intent parsing text generation unit is connected to the audiovisual fusion error correction unit, and is used to perform semantic slot filling and traffic hub scene rule matching on the corrected text sequence to generate intent parsing text.

[0010] Furthermore, the knowledge base matching and path decision module includes: a semantic vectorization and similarity calculation unit, a coverage determination unit, and a response path selection unit; wherein, the semantic vectorization and similarity calculation unit is connected to the intent parsing text generation unit, and is used to convert the intent parsing text into semantic vectors and perform distance calculation with the structured question-answer pair vectors in the updated local knowledge base to obtain a matching score; the coverage determination unit is connected to the semantic vectorization and similarity calculation unit, and is used to determine whether it belongs to the coverage range of the updated local knowledge base based on the relationship between the matching score and a preset threshold, and obtain a coverage determination result; the response path selection unit is connected to the coverage determination unit, and is used to select a direct knowledge base answer path or a large language model fallback path based on the coverage determination result, and generate a response path selection result.

[0011] Furthermore, the large-scale intelligent response and security audit module includes: a large-scale model inference unit, a multi-dimensional security verification unit, and a content decision and text generation unit; wherein, the large-scale model inference unit, connected to the response path selection unit, is used to input the intent parsing text and the updated transportation hub-specific constraint prompts from the local knowledge base into the large language model when the response path selection result is not covered, to obtain the initial response content; the multi-dimensional security verification unit, connected to the large-scale model inference unit, is used to sequentially perform policy compliance detection, sensitive information shielding, and factual accuracy verification on the initial response content to obtain the verification result; the content decision and text generation unit, connected to the multi-dimensional security verification unit, is used to rewrite or confirm the initial response content according to the verification result, and generate a safe and compliant response text.

[0012] Furthermore, the multimodal synthesis module includes: a text prosody and sentiment analysis unit, a natural language synthesis unit, and a digital human animation generation unit; wherein, the text prosody and sentiment analysis unit is connected to the content decision and text generation unit, and is used to perform prosody analysis and sentiment annotation on the safe and compliant answer text to obtain a prosodic text sequence; the natural language synthesis unit is connected to the text prosody and sentiment analysis unit, and is used to generate naturally synthesized speech data based on the prosodic text sequence; the digital human animation generation unit is used to synchronously generate a digital human lip shape, facial expression, and gesture sequence consistent with the speech rhythm using the naturally synthesized speech data to obtain a digital human animation sequence.

[0013] Furthermore, the session management module includes: a multi-channel output rendering unit, a visual interaction recording unit, and a session log generation unit; wherein, the multi-channel output rendering unit is connected to the digital human animation generation unit, and is used to send the natural synthesized speech data to a full-band speaker and simultaneously send the digital human animation sequence to a high-definition touch screen for real-time rendering output; the visual interaction recording unit is connected to the multi-channel output rendering unit, and is used to sequentially display the user intent parsing text and the safety and compliance response text in the form of dialog bubbles on the high-definition touch screen to obtain a visual interaction record; the session log generation unit is connected to the visual interaction recording unit, and is used to package the visual interaction record, the synthesized speech data, and the digital human animation sequence to form a complete interactive session log.

[0014] Furthermore, the session management module also includes a service quality analysis and optimization feedback unit; wherein, the service quality analysis and optimization feedback unit is connected to the session log generation unit, and is used to obtain the user dwell time and repeated question frequency in the complete interaction session log to obtain service quality feedback indicators; according to the service quality feedback indicators, the question-answer pairs with lower matching scores in the updated local knowledge base are prioritized to obtain an optimization mark set and sent to the backend.

[0015] Compared with the prior art, the present invention has the following advantages: This invention improves the quality and reliability of customer service by constructing a dual-path response decision mechanism. The system prioritizes accurate matching using a local knowledge base, ensuring the authority and speed of responses to standard business inquiries; for long-tail questions not covered by the knowledge base, it invokes a constrained and reviewed large language model for intelligent responses.

[0016] This invention establishes a rigorous, multi-dimensional content security review module to ensure the compliance and security of commercial service operations. While providing more flexible services using a large language model, the system forms a solid security barrier by performing multiple checks on generated content, including policy compliance, sensitive information, and factual accuracy.

[0017] This invention improves customer service experience and efficiency by deeply integrating multimodal interaction technology. The system can not only accurately capture user intent in noisy environments like transportation hubs through multimodal perception technology, but also generate digital human expressions and movements synchronized with voice emotion and rhythm through multimodal synthesis technology.

[0018] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a framework diagram of a multimodal AI digital human service system for transportation hubs according to an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of the structure of a multimodal AI digital human service system for transportation hubs according to an embodiment of the present invention.

[0022] Figure 3 This is a flowchart of the multimodal sensing preprocessing module according to an embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Reference Figure 1One embodiment of the present invention proposes a multimodal AI digital human service system for transportation hubs. It adopts multimodal signal fusion and noise suppression technology, combined with a local knowledge base differential update and a large model dynamic generation security collaboration mechanism, which can realize multilingual intelligent interactive services in the complex environment of transportation hubs, and improve the accuracy and response efficiency of services.

[0025] like Figure 2 As shown, the system in this embodiment specifically includes: a local knowledge base module, a multimodal perception preprocessing module, an intent parsing module, a knowledge base matching and path decision module, a large-scale intelligent response and security auditing module, a multimodal synthesis module, and a session management module; wherein, S1. The local knowledge base module is used to obtain the approved knowledge base update package and perform a versioned differential loading operation to generate the updated local knowledge base. Furthermore, the local knowledge base module includes: a version interaction and incremental acquisition unit, a security verification and conflict detection unit, and a differential loading and library update unit; wherein, The version interaction and incremental acquisition unit is used to periodically send the current local knowledge base version identifier to the backend server and receive the corresponding incremental update package. The security verification and conflict detection unit is connected to the version interaction and incremental acquisition unit, and is used to perform digital signature verification and content conflict detection on the incremental update package to obtain the verified update content. The differential loading and library update unit is connected to the security verification and conflict detection unit, and is used to replace the corresponding entries in the local knowledge base with the verified update content in a differential manner to generate an updated local knowledge base.

[0026] Specifically, the update process is initiated by the version interaction and incremental acquisition unit. A synchronization mechanism is established between the local knowledge base module and the backend server to ensure efficient and timely acquisition of the latest knowledge base content. This unit has a built-in timer that proactively sends a version synchronization request to the backend knowledge base management server at a configurable polling period, typically set between 30 and 60 minutes. The request includes a version identifier for the current local knowledge base, a globally unique string, such as a UUID generated based on content hash and timestamp, used to uniquely identify the current state of the knowledge base. Upon receiving the version identifier, the backend server compares it with the latest version in its central knowledge base. If the versions match, a status code indicating no update is needed is returned; if a new version exists, the server generates an incremental update package. The incremental update package is a structured data file, such as in JSON format, containing all changed entries since the previous version, including added, modified, or deleted question-answer pairs, rather than the entire knowledge base, thus reducing network transmission load. The version interaction and incremental acquisition unit receives this incremental update package and passes it to the next processing unit.

[0027] Upon receiving the incremental update packet, the security verification and conflict detection unit takes over the processing. This unit ensures the credibility of the incremental update packet's source and the integrity of its content, preventing logical conflicts between the updated content and the existing knowledge base. This unit first performs digital signature verification. Digital signatures are an application of asymmetric encryption technology, using the private key of the backend server to encrypt the hash value of the incremental update packet, ensuring that the update packet has not been tampered with during transmission and that its source is credible. This unit decrypts the signature using the server's public key pre-installed in the system to obtain the original hash value, and simultaneously independently calculates the hash value of the received incremental update packet content, for example, using the SHA-256 algorithm. If the two hash values ​​match perfectly, the source credibility and content integrity verification passes. Subsequently, the unit performs content conflict detection to avoid introducing semantically repetitive but contradictory knowledge. For each newly added or modified question-and-answer entry in the incremental update packet, the system extracts its question text, converts it into a semantic vector using a pre-trained sentence vector model, and calculates the cosine similarity with the semantic vectors of all existing questions in the local knowledge base. Conflict determination is based on the following formula: when a conflict is detected... When the value is 1, the entry is marked as conflicting. A value of 0 indicates no conflict. The formula is: , in, For the semantic vector of the new entry question, This is the semantic vector of a problem for an existing entry. This function represents the cosine similarity calculation for two vectors. The specific form is as follows: , For the semantic vector of the new entry question, The semantic vector of the existing entry question; The dimension of the semantic vector; For vectors The Dimensional components; For vectors The Dimensional components; For the answer text of the new entry, The answer text for an existing entry. This represents the normalized edit distance between two answer texts, with a value ranging from [0,1]. A smaller value indicates more similar answers, while a larger value indicates greater differences. (Function) The specific form is: , in The edit distance between the two answer texts; These represent the character lengths of the corresponding answer text.

[0028] function The logic is that when The value is higher than a preset high similarity threshold, such as 0.98, and When the value is greater than or equal to the difference threshold, it indicates that the questions are highly similar but the answers are different, and the function... An output of 1 indicates a conflict. Entries marked as conflicting will be isolated and sent to the backend for manual review. After these two verification steps, the set of entries that pose no security risks and have no content conflicts constitutes the verified update content.

[0029] The differential loading and library update unit performs the final update operation of the knowledge base. It accurately and efficiently applies the validated updates to the local knowledge base, completing version iteration while ensuring service continuity during the update process. Differential loading is a technique that updates only the changed parts of the dataset, rather than completely replacing the entire dataset. By directly manipulating specific entries, it reduces the time and system resource overhead required for updates. This unit reads the validated update content, which consists of a series of explicit instructions, such as "ADD", "UPDATE", and "DELETE", along with the corresponding data. The unit iterates through these instructions and performs the corresponding operations, such as inserting new question-answer pairs and their semantic vectors according to the "ADD" instruction; locating old entries by unique IDs and replacing their content according to the "UPDATE" instruction; and removing specified entries according to the "DELETE" instruction. To ensure data consistency, the entire update process is encapsulated in a transaction. The system first applies the changes to a temporary copy of the knowledge base. After the operation is complete, an atomic pointer switch points the main service to this new copy. If any step in the update process fails, the system will automatically roll back, discard the temporary copy, and continue using the old version of the knowledge base, thus preventing the knowledge base from being corrupted or inconsistent due to update failure. After a successful update, the system will update the version identifier stored locally to match the latest version on the backend server, completing a full knowledge base update cycle and generating the updated local knowledge base.

[0030] For example, the version interaction and incremental acquisition unit sends the current version identifier to the backend and obtains the incremental update package containing the new entry when the polling cycle arrives. After receiving the incremental update package, the security verification and conflict detection unit first decrypts the signature hash using the public key and compares it with the SHA-256 hash value of the current package content to complete the security verification. Then, conflict detection is performed, assuming the semantic vector of the new entry problem is... The semantic vector for the locally existing entry problem is: Vector dimension The answer to the new entry is... The answer to "in the waiting room on the second floor" already exists. "Waiting in the lobby on the first floor." First, calculate semantic similarity. Next, the difference in the answer text is calculated, given that... The length is 6. The edit distance between two strings of length 7. If the value is 4, then the degree of difference is Define the conflict logic determination function The high similarity threshold is 0.98 and the difference threshold is 0.5, because and Decision function An output of 1 indicates that the entry has a semantic conflict and needs to be isolated for review. For non-conflicting entries that pass the validation, the differential loading and library update unit applies them to the temporary copy according to the instruction. If the instruction is UPDATE, the unit locates the old entry using a unique ID and replaces its content. Finally, the main service is switched from the old library to the new copy using an atomic pointer, and the local version identifier is updated, completing the differential update of the knowledge base.

[0031] S2. The multimodal perception preprocessing module is used to acquire multimodal interaction requests and perform signal fusion and noise suppression processing to generate standardized interaction data; Furthermore, the multimodal perception preprocessing module includes: a multi-source heterogeneous data acquisition unit, a spatiotemporal feature enhancement and extraction unit, and a cross-modal temporal alignment and fusion unit; wherein, The multi-source heterogeneous data acquisition unit is connected to the differential loading and library update unit and is used to acquire speech signals and synchronous visual signals to obtain raw multi-source data. The spatiotemporal feature enhancement and extraction unit is connected to the multi-source heterogeneous data acquisition unit and is used to perform beamforming noise reduction and lip movement feature extraction on the original multi-source data to obtain enhanced speech features and visual auxiliary features. The cross-modal temporal alignment and fusion unit is connected to the spatiotemporal feature enhancement and extraction unit, and is used to temporally align and fuse the enhanced speech features and visual auxiliary features to generate standardized interactive data.

[0032] Specifically, the workflow of the multimodal perception preprocessing module is as follows: Figure 3 As shown, the initial data acquisition is performed by a multi-source heterogeneous data acquisition unit. This unit precisely and synchronously acquires the user's voice and visual signals. It operates through a microphone array and a high-definition camera integrated into the digital human terminal device. The microphone array, typically composed of 4 to 8 MEMS microphones, records the sound field in multiple channels at a sampling rate of 16kHz or higher; simultaneously, the high-definition camera captures video of the user's face and upper body at a resolution of at least 1920×1080 and a frame rate of 30fps. Crucially, the system uses a unified hardware clock source or the Network Time Protocol (NTP) to timestamp audio sampling points and video frames with millisecond-level precision, ensuring strict synchronization of audio and video in subsequent processing, forming the original multi-source data stream, and sending it to the next unit.

[0033] The spatiotemporal feature enhancement and extraction unit processes the raw multi-source data. It extracts useful speech signals from the noisy raw data and extracts visual cues that aid speech recognition. For audio signals, the unit first applies beamforming noise reduction technology. Beamforming is a technique that utilizes the spatial characteristics of a microphone array to enhance sound from a specific direction. The system uses sound source localization algorithms, such as Generalized Cross-Correlation-Phase Transform (GCC-PHAT), to determine the direction of the user's speech in real time. Then, it dynamically adjusts the delay and gain of each microphone signal to form a virtual "auditory beam" pointing towards the user, thereby improving the signal-to-noise ratio of the user's speech by 5 to 10 dB, resulting in enhanced speech features. For synchronized video signals, the unit performs lip movement feature extraction. This process uses a pre-trained deep learning model to first detect the face, locating 68 key points on the face, and then focusing on 20 key points in the lip region. The system calculates the displacement and shape changes of these key points between consecutive video frames, generating a time-varying vector sequence, which is the visual auxiliary feature that quantifies the lip movement pattern during speech.

[0034] The cross-modal temporal alignment and fusion unit integrates processed speech and visual features. It aligns and fuses feature streams from different modalities with varying temporal resolutions into a unified, more information-rich, standardized interactive data. Since audio features are typically analyzed in 10-millisecond frames, while video features are analyzed in approximately 33-millisecond frames, a difference in temporal resolution exists. This unit uses methods such as upsampling or linear interpolation to improve the temporal resolution of visual auxiliary features to match that of audio features, achieving frame-level temporal alignment. After alignment, the unit employs a weighted fusion strategy to generate the final feature vector. The fusion process can be represented by the following formula: , In this formula, Represents the standardized interactive data frame after fusion. It is the enhanced speech feature vector of the current time frame. It is a visual auxiliary feature vector that is aligned with it. It is an estimate of the ambient noise level or signal-to-noise ratio based on the audio signal in real time. and There are two functions, which are based on the ambient noise level. Weights are generated dynamically. In a quiet environment, The value will be close to 1, while A smaller value indicates that the system relies more on voice information; in noisy environments, The value will decrease. The value of increases, thereby increasing the weight of visual lip movement information to compensate for the unreliability of speech information. Through this adaptive weighted fusion, the system generates more robust standardized interaction data, which is then passed to the intent parsing module for further processing. and The specific form is as follows: , , For real-time signal-to-noise ratio, This represents the signal-to-noise ratio threshold, which can be adjusted according to the noise level of the actual deployment environment. This is the threshold for extremely low signal-to-noise ratio.

[0035] For example, the multi-source heterogeneous data acquisition unit simultaneously acquires raw multi-source data through a microphone array and a high-definition camera. The spatiotemporal feature enhancement and extraction unit uses the GCC-PHAT algorithm to perform beamforming noise reduction on the audio, thereby improving the signal-to-noise ratio. Simultaneously, the unit locates key facial points and extracts lip movement pattern sequences as visual auxiliary feature vectors. Cross-modal temporal alignment and fusion units use linear interpolation to... The temporal resolution is aligned to the audio frame level. Assuming the current environment is noisy, the real-time estimated signal-to-noise ratio... And preset threshold Calculate the weighting coefficients according to the formula. , If the enhanced speech feature vector of the current frame Aligned visual auxiliary feature vectors Then standardized interactive data frames The calculation process is as follows Through the aforementioned adaptive weighted fusion, the system generates standardized interactive data that is robust in noisy environments, thus compensating for the unreliability of a single voice modality.

[0036] S3. The intent parsing module is used to perform multilingual speech recognition and visual-assisted intent analysis on the standardized interaction data to generate intent parsing text. Furthermore, the intent parsing module includes: a speech recognition and decoding unit, an audiovisual fusion error correction unit, and an intent parsing text generation unit; wherein, The speech recognition and decoding unit is connected to the cross-modal temporal alignment and fusion unit, and is used to perform acoustic model decoding and language model correction on the speech part of the standardized interactive data to obtain candidate text sequences. The audiovisual fusion error correction unit is connected to the speech recognition and decoding unit and is used to correct easily confused pronunciations in the candidate text sequence using the visual auxiliary features to obtain a corrected text sequence. The intent parsing text generation unit is connected to the audiovisual fusion error correction unit and is used to perform semantic slot filling and traffic hub scene rule matching on the corrected text sequence to generate intent parsing text.

[0037] Specifically, the speech recognition and decoding unit performs initial translation of the input data. It converts the acoustic portion of the standardized interactive data, containing both speech and visual information, into preliminary candidate text. This unit employs an end-to-end acoustic model based on a Conformer architecture, trained to handle multilingual input, capable of directly mapping the enhanced speech feature sequence of the input to probability distributions of character or word units. Next, a pre-trained Transformer-based language model, combined with a beam search algorithm, decodes the probability distribution output by the acoustic model. The beam search algorithm retains a fixed number of optimal candidate sequences at each time step, rather than just one best choice, thus generating a list containing multiple candidate text sequences and their corresponding acoustic and language model scores; this list constitutes the candidate text sequences.

[0038] The audiovisual fusion error correction unit refines the initial candidate text. Utilizing synchronized visual auxiliary features, it performs a secondary judgment on acoustically easily confused pronunciations, thereby improving the final accuracy of speech recognition. For example, the pronunciations 'b' and 'p' have similar acoustic features in many noisy environments, but their lip movements, i.e., the visual isographs, differ significantly. This unit iterates through each candidate text sequence output by the previous unit and calculates the matching degree between each phoneme and the lip movement sequence extracted from the visual auxiliary features. The matching degree is evaluated using a small neural network model that specifically learns the association between phonemes and their corresponding lip movement patterns. The system re-scores each candidate text sequence using the following weighted formula to select the optimal result. The formula is: , in, It is the final score of the candidate sequence. It is the original acoustic score. It is a visual matching score calculated through a neural network model. It is a dynamic weighting coefficient whose value depends on the signal-to-noise ratio. Adaptive adjustment The value is calculated by the front-end module. When the value is low, i.e., the noise is high, Decrease, thereby increasing visual score The influence of the unit. The unit selects the candidate sequence with the highest final score S as the corrected text sequence and passes it to the next processing stage.

[0039] The intent parsing text generation unit performs deep semantic understanding and structuring on the corrected text. It transforms the corrected text sequence from plain text into a structured data object containing clear intents and key information slots. At the core of this unit is a Natural Language Understanding (NLU) model fine-tuned for the transportation hub domain, such as a BERT-based model. This model performs two tasks in parallel: semantic slot filling and intent classification. Semantic slot filling identifies and extracts key entities from the text, such as "destination," "departure point," "train / flight number," and "date / time." Intent classification determines the overall purpose of the user's request, such as "ticket inquiry," "route guidance," or "lost and found." For example, for the input text "Are there any tickets left for the G5 train to Beijing South Station tomorrow?", the model will output the intent as "ticket inquiry" and fill the semantic slots as {date, tomorrow; destination, Beijing South Station; train number, G5}. After NLU processing, the system also performs a transportation hub scenario rule matching step, a set of hard rules based on expert knowledge encoding, used to handle ambiguous expressions and supplement default information. For example, when a user only asks "go to the airport," the rules will automatically complete the question to "route from the current train station to the airport" based on the current deployment location of the digital human device. After this series of processing steps, the module finally outputs a JSON-formatted intent parsing text, which includes the original text, confidence score, classification intent, and filled semantic slot entities.

[0040] For example, the speech recognition and decoding unit receives standardized interactive data, decodes it using a Conformer-based acoustic model and language model, and generates a list containing candidate text sequences. The audiovisual fusion error correction unit uses visual auxiliary features to perform a secondary judgment on the candidate sequences. Assuming there are two candidate text sequences, sequence one is "go to Dalian", its acoustic score... Visual matching score Sequence two is "to Taiyuan", and its acoustic score is... Visual matching score Current environmental signal-to-noise ratio The system adaptively adjusts the dynamic weighting coefficients when the value is low. The final score for each sequence is calculated according to the scoring formula. Unit selection with the highest score The corresponding text serves as the corrected text sequence. The intent parsing text generation unit then performs semantic slot filling on "go to Taiyuan," identifying the intent as route guidance and the destination as Taiyuan. It also completes the information by combining the transportation hub scenario rules, ultimately generating the intent parsing text.

[0041] S4. The knowledge base matching and path decision module is used to perform similarity matching judgment between the intent parsing text and the structured question-answer pairs in the updated local knowledge base, select the corresponding response path, and generate a response path selection result. Furthermore, the knowledge base matching and path decision module includes: a semantic vectorization and similarity calculation unit, a coverage determination unit, and a response path selection unit; wherein, The semantic vectorization and similarity calculation unit is connected to the intent parsing text generation unit and is used to convert the intent parsing text into semantic vectors and perform distance calculation with the structured question-answer pair vectors in the updated local knowledge base to obtain a matching score. The coverage determination unit is connected to the semantic vectorization and similarity calculation unit, and is used to determine whether it belongs to the updated local knowledge base coverage range based on the relationship between the matching score and the preset threshold, and obtain the coverage determination result. The response path selection unit is connected to the coverage determination unit and is used to select either the direct knowledge base answer path or the large language model fallback path based on the coverage determination result, and generate a response path selection result.

[0042] Specifically, the semantic vectorization and similarity calculation unit performs quantitative matching of intent. It performs semantic similarity measurement between the user's textual intent and the standard questions in the updated local knowledge base to find the most relevant existing answers. Upon receiving the intent parsing text, this unit first extracts the core question portion. Then, using a pre-trained sentence vector model, such as a Sentence-BERT model fine-tuned for the Chinese traffic domain, it encodes the question into a high-dimensional semantic vector in real time, typically with dimensions of 768 or 1024. Simultaneously, all standard questions in the updated local knowledge base have been pre-encoded into semantic vectors of the same dimension and indexed when the knowledge base is loaded or updated. This unit then performs an efficient nearest neighbor search within the index, calculating the cosine similarity between the user's question vector and all question vectors in the knowledge base. This process can be represented as: , in, The semantic vector representing the user's intent is parsed from the converted text. This represents the question vector of a structured question-answer pair in the updated local knowledge base. Represents the semantic vector of user questions The length of the mold, Represents the vector of questions in the knowledge base The length of the mold, The function represents cosine similarity calculation, and its output matching score M is a scalar between -1 and 1, with values ​​closer to 1 indicating greater semantic similarity. The unit returns a score based on the matching score. A list of candidate question-answer pairs sorted from highest to lowest.

[0043] The coverage determination unit makes decisions based on the matching results. According to the set confidence level, it determines whether the user's question can be effectively covered and directly answered by an entry in the updated local knowledge base. This unit obtains the score of the candidate question-answer pair with the highest matching score output from the previous unit. Then, this highest score is compared to a preset coverage threshold. The coverage threshold is a key engineering parameter, typically set between 0.90 and 0.95, and its specific value is derived through optimization on a large amount of offline test data using an objective function that balances accuracy and recall. When the score is greater than or equal to the coverage threshold, the system determines that the user's intent has been highly covered by the local knowledge base, and the coverage judgment result is marked as "covered". Conversely, if the highest matching score also fails to reach the threshold, it is considered that there are not enough similar question-answer pairs in the updated local knowledge base to accurately answer the user's question, and the coverage judgment result is marked as "not covered". This judgment process is the watershed of the entire system response path.

[0044] The response path selection unit performs the final routing based on the coverage judgment result. This directs the processing flow to two distinct response generation paths: a deterministic knowledge base path or a generative large model path. This unit functions similarly to a logic switch. When the received coverage judgment result is "covered," it selects the direct knowledge base answer path. Specifically, it packages the unique identifier ID of the highest-scoring structured question-and-answer pair and its corresponding answer text into an internal response instruction and sends it to the subsequent synthesis module to generate a standard, secure answer. When the received coverage judgment result is "not covered," the unit selects the large language model fallback path. Specifically, it packages the unmatched, fully context-rich original intent parsing text, along with a flag indicating that subsequent modules should call the large model, into the response path selection result.

[0045] For example, after receiving the intent parsing text, the semantic vectorization and similarity calculation unit uses the Sentence-BERT model to encode the user's question into a semantic vector in real time. Assuming At this point, the candidate question vector with the highest matching score in the updated local knowledge base The unit calculates the cosine similarity matching score between the two using a formula, given that the vector magnitude is calculated as follows. Substitute into the formula to get the matching score. The coverage decision unit obtains the highest matching score. If the preset coverage threshold is set to 0.92, because... The coverage determination unit marks the coverage judgment result as coverage. The response path selection unit performs routing distribution based on this result, selects the direct knowledge base answer path, and packages the corresponding question-answer pair identifier ID and answer text into an internal response instruction and sends it to the subsequent synthesis module. If the matching score is lower than the threshold, the large language model fallback path is selected, triggering the intervention of the large model intelligent response and security review module.

[0046] S5. The large model intelligent response and security audit module is used to call the preset large language model and generate initial response content based on the intent parsing text when the response path selection result is not covered, and perform multi-dimensional content security verification to generate safe and compliant response text. Furthermore, the large-scale model intelligent response and security audit module includes: a large-scale model inference unit, a multi-dimensional security verification unit, and a content decision and text generation unit; wherein, The large model reasoning unit is connected to the response path selection unit. When the response path selection result is not covered, it inputs the intent parsing text and the updated traffic hub-specific constraint prompts in the local knowledge base into the large language model to obtain the initial answer content. The multi-dimensional security verification unit is connected to the large model reasoning unit and is used to sequentially perform policy compliance detection, sensitive information shielding, and factual accuracy verification on the initial response content to obtain the verification result. The content decision and text generation unit is connected to the multi-dimensional security verification unit and is used to rewrite or confirm the initial answer content based on the verification result to generate a safe and compliant answer text.

[0047] Specifically, the large-scale model inference unit generates an initial response text. Based on the user's intent, it calls a large language model to generate an initial response relevant to the transportation hub scenario. Upon receiving the intent-parsed text, this unit does not directly use it as input. To constrain the model's output and avoid generating unrelated or irrelevant responses, the system constructs a carefully designed prompt. This prompt consists of three parts: a role-playing instruction, such as "You are a professional transportation hub service digital human; your response should be concise, accurate, and polite," a transportation hub-specific constraint, such as "You know you are currently at a certain train station, it is currently a certain time, and you can only answer questions related to transportation," and the user's intent-parsed text. In this way, the inference task is confined to a clear domain and role framework. Then, the unit uses this composite prompt to call a pre-defined large language model via API, such as a 7B or 13B parameter-level model fine-tuned for transportation domain instructions. After receiving the input, the model performs inference calculations, generates an initial response, and passes it to the next stage for review.

[0048] The multi-dimensional security verification unit filters the initial response content generated by the model layer by layer. It identifies and removes various inappropriate, false, or harmful information that may exist in the initial response, ensuring that the final output fully complies with public service requirements. This unit's verification process is executed serially, following a coarse-to-fine, high-risk-to-low-risk order. The first step is policy compliance detection: the system compares the initial response content with a blacklist containing keywords such as the latest policies and regulations, and place name standards. If any match is found, an interception or rewriting strategy is triggered directly. The second step is sensitive information masking: the system uses regular expressions and named entity recognition models to detect potentially leaked personal privacy information in the text, such as phone numbers and ID card numbers, and then anonymizes it. The third step is factual accuracy verification, which is the most complex step. The system extracts key factual statements from the response content, such as "Train G5 checks tickets at platform 3," and then attempts to cross-validate them in an updated local knowledge base or a trusted external fact database. The verification result can be expressed as a confidence score. To quantify. The value depends on the information source and the degree of matching, as shown in the following formula: , This indicates the number of valid information entries matched in a trusted external fact database. This indicates the total number of information entries retrieved. This represents the matching weight coefficient, which directly finds a completely matching entry in the updated local knowledge base. It can be set to 1.0; if information from high-authority websites is found through search engines, It can be 0.9; if no supporting information can be found, then... The result is relatively low. After these three layers of verification, each verification will generate a result tag, which together constitute the comprehensive verification result.

[0049] The content decision and text generation unit determines the final output text based on the verification results. Based on multi-dimensional security review results, it makes a final judgment on the initial response content, generating a response text that both meets user needs and is absolutely safe. This unit adopts three different processing strategies based on the verification results. If the verification results show serious policy compliance issues, the unit will directly reject the answer and generate a preset polite reply. If the verification results show a risk of factual inaccuracies, i.e., a factual accuracy verification score... If the score is below a preset decision threshold, such as 0.7, the unit will conservatively rewrite the initial response, adding more uncertainty or suggestive language. If all verification items pass with a high score, i.e. If the response exceeds the decision threshold and there are no other security issues, the unit confirms that the initial response has passed and makes no modifications. Ultimately, regardless of the processing path, the unit will output a text, which is the security and compliance response text, ready to be sent to the multimodal synthesis module for final presentation.

[0050] For example, after receiving the intent parsing text, the large model inference unit concatenates it with prompts containing role-playing instructions and transportation hub-specific constraints, and inputs this concatenation into the large language model to obtain the initial response. The multi-dimensional security verification unit then sequentially performs policy compliance checks and sensitive information masking on this response, and initiates factual accuracy verification. Assuming the initial response contains information about a train's ticket gate, the system calculates a credibility score through cross-validation. Given that 3 relevant supporting information entries were found in a trusted external fact database, the total number of retrieved entries is 4, and the matching weight coefficient is 0.8, calculate the fact credibility using the preset formula. The content decision and text generation unit processes the result, with a known preset decision threshold of 0.7. Because... The unit determined that the initial answer was at risk of factual inaccuracies, so it performed a conservative rewrite of the text, changing the certain statements into suggestive terms, and finally generated a safe and compliant answer text and output it to the synthesis module.

[0051] S6. The multimodal synthesis module is used to simultaneously generate synthesized speech data and digital human animation sequences based on the safe and compliant response text; Furthermore, the multimodal synthesis module includes: a text prosody and sentiment analysis unit, a natural language speech synthesis unit, and a digital human animation generation unit; wherein, The text prosody and sentiment analysis unit, connected to the content decision and text generation unit, is used to perform prosody analysis and sentiment annotation on the safe and compliant answer text to obtain a prosodic text sequence. The natural speech synthesis unit is connected to the text prosody and sentiment analysis unit and is used to generate natural synthesized speech data based on the prosodic text sequence. The digital human animation generation unit is used to synchronously generate digital human lip-shape, facial expression, and gesture sequences that are consistent with the speech rhythm using the natural synthesized speech data, thereby obtaining a digital human animation sequence.

[0052] Specifically, the text prosody and sentiment analysis unit preprocesses the input text. The engineering goal is to infuse the synthesized speech and animation with semantically appropriate prosody and emotional nuances. Upon receiving a safe and compliant response text, the unit immediately launches a Transformer-based text analysis model. This model is trained to identify sentiment tendencies in the text, such as neutral, happy, and apologetic, and outputs a global sentiment classification label. Simultaneously, the model performs prosodic analysis, segmenting the text into words and predicting appropriate pauses, stresses, and intonation patterns for each word or phrase. For example, for the sentence "Please go to gate 3," the model identifies "gate 3" as the stressed part and predicts a brief pause between "please" and "go to." After processing, the unit outputs a text sequence rich in prosodic and sentiment annotations. This is an internal data structure containing the original text along with additional SSML-style tags, providing fine-grained control for subsequent speech and animation generation.

[0053] The Natural Speech Synthesis Unit (TSU) generates audio based on annotated text. It converts prosodic text sequences into auditoryally natural, clear, and emotionally resonant speech waveforms. This unit employs an advanced end-to-end TTS (Text-to-Speech) model, such as a VITS-based model. This model can directly generate high-quality speech waveforms from text and prosodic annotations in a single pass, eliminating intermediate steps like acoustic feature generation and vocoder synthesis found in traditional TTS systems, thus ensuring speech coherence and naturalness. Emotional tags serve as conditional inputs to the model, guiding it to generate timbre and intonation that match the specified emotions. Prosodic markers directly control the pause length, pitch variations, and energy distribution of the synthesized speech. After inference, the model outputs a raw PCM format audio waveform, typically sampled at 24kHz or 48kHz to ensure high fidelity; this is the natural synthesized speech data.

[0054] The digital human animation generation unit synchronously generates visual animations. It creates digital human facial expressions, lip movements, and body movements that perfectly match the synthesized speech in terms of timing, rhythm, and emotional expression, achieving audio-visual synchronization. This unit receives natural synthesized speech data as its core driving signal. Internally, the unit employs an audio-driven animation generation model that analyzes the phoneme sequence, energy envelope, and pitch curve of the input speech in real time. Based on these acoustic features, the model predicts and generates a series of keyframe data controlling the digital human skeletal animation system. Specifically, the phoneme sequence is primarily used to drive precise lip-sync animation, ensuring a one-to-one correspondence between mouth shape and pronunciation; energy and pitch curves are used to generate natural facial expressions, such as eyebrow twitching, eye blinking, and subtle head posture changes; while emotion tags trigger preset macroscopic actions, such as a slight nod to express happiness or a slight forward lean to express apology. Furthermore, the model can generate non-verbal feedback actions based on the user's visual characteristics at the start of the interaction, such as mimicking a slight head tilt to enhance the intimacy of the interaction. Finally, the unit outputs a digital human animation sequence, which is a time-series data containing the skeletal parameters of each frame of the animation. This sequence, along with the natural language synthesized speech data, is sent to the session management module for final rendering output.

[0055] For example, after receiving a safe and compliant response text, the text prosody and sentiment analysis unit uses a Transformer-based text analysis model to identify sentiment tags and predict pauses and stresses, outputting a prosodic text sequence with SSML annotations. The natural speech synthesis unit receives this sequence and generates PCM-formatted natural synthesized speech data in one go based on an end-to-end model such as VITS, ensuring speech coherence and emotional expression. The digital human animation generation unit uses this speech data as the core driving signal, analyzing phoneme sequences, energy envelopes, and pitch curves in real time through an audio-driven animation generation model. During reproduction, the model drives precise lip-sync animation based on the phoneme sequence, generates facial expressions based on the energy and pitch curves, and triggers preset macroscopic gestures based on sentiment tags. Finally, the system generates a sequence of digital human lip shapes, expressions, and gestures that perfectly match the speech rhythm, resulting in a digital human animation sequence that is then output.

[0056] S7. The session management module is used to generate a complete interactive session log based on the synthesized speech data and the digital human animation sequence.

[0057] Furthermore, the session management module includes: a multi-channel output rendering unit, a visual interaction recording unit, and a session log generation unit; wherein, The multi-channel output rendering unit is connected to the digital human animation generation unit and is used to send the natural synthesized speech data to the full-band speaker and simultaneously send the digital human animation sequence to the high-definition touch screen for real-time rendering output. The visual interaction recording unit is connected to the multi-channel output rendering unit and is used to sequentially display the user intent parsing text and the safety and compliance answer text in the form of dialog bubbles on the high-definition touch screen to obtain a visual interaction record. The session log generation unit is connected to the visual interaction recording unit and is used to package the visual interaction record, synthesized speech data and digital human animation sequence to form a complete interactive session log.

[0058] Specifically, the front-end rendering is handled by a multi-channel output rendering unit. This unit combines independent speech data and animation sequences into a unified and coordinated audiovisual experience in real time on the hardware. It receives naturally synthesized speech data and digital human animation sequences from the multimodal synthesis module. It sends the PCM-formatted naturally synthesized speech data stream to the system's digital signal processor (DSP) for decoding and then plays it out through the built-in full-range speaker. Simultaneously, it sends the digital human animation sequence to the graphics processing unit (GPU). Based on the skeletal parameter data for each frame, the GPU drives a digital human rendering pipeline built on a game engine such as Unity or Unreal Engine in real time. This pipeline is responsible for calculating model skinning, lighting, shading, etc., ultimately rendering the dynamic digital human image onto a high-definition touchscreen at a frame rate of no less than 60fps.

[0059] The visual interaction recording unit is responsible for providing auxiliary information on the interactive interface. By displaying dialogue text on the screen, it provides users with a clear and reviewable interaction history, which is especially important for users who may have difficulty hearing speech in noisy environments. This unit works in parallel with the multi-channel output rendering unit. When the system parses the user's intent and generates intent parsing text, this unit displays the intent parsing text in a preset area of ​​the high-definition touchscreen display, usually on the side or bottom of the screen, as a speech bubble representing the user. When the system generates safe and compliant response text and begins playing speech and animation, this unit displays the response text sentence by sentence or in whole paragraphs in another speech bubble representing the digital human.

[0060] The session log generation unit archives all data from this interaction. It packages all relevant data from a complete user session into a structured log file for subsequent offline analysis and system optimization. This unit is activated when a session ends, such as when the user hasn't interacted further for a preset timeout (e.g., 30 seconds), or when the user explicitly expresses gratitude or leaves. It collects all data fragments from the session, including but not limited to data captured by the original multi-source heterogeneous data acquisition unit, intent parsing text for each round, response path selection results, safe and compliant response text, natural language synthesized speech data, and digital human animation sequences. Additionally, it captures screenshots or text copies of the visual interaction records generated by the previous unit. Then, this unit organizes and encapsulates all these data items according to a predefined JSON or XML format and assigns a globally unique session ID to the log file. The resulting compressed file is the complete interaction session log.

[0061] For example, after a single interaction task is completed, the conversation log generation unit aggregates all the data from the user's original voice signal, intent parsing text, knowledge base matching path, and digital human synthesis feedback to generate a structured and complete interaction conversation log. The service quality analysis and optimization feedback unit then extracts and quantifies service quality feedback indicators, calculates user dwell time by analyzing timestamps in the conversation log, and uses semantic similarity algorithms to identify the frequency of repeated questions with the same intent within the same conversation. If a low matching score is detected for an interaction and the user repeatedly asks the same question, the system associates this anomaly with the corresponding knowledge base entry. Based on the severity of the feedback indicators, the service quality analysis and optimization feedback unit prioritizes and labels poorly performing question-and-answer pairs in the updated local knowledge base, compiling entries requiring urgent optimization into an optimization tag set. Finally, the system sends this set to the backend server as a basis for knowledge base maintenance personnel to make decisions regarding entry correction, expansion, or model fine-tuning.

[0062] Furthermore, the session management module also includes: a service quality analysis and optimization feedback unit; wherein, The service quality analysis and optimization feedback unit is connected to the session log generation unit and is used to obtain the user dwell time and the frequency of repeated questions in the complete interaction session log to obtain service quality feedback indicators. Based on the service quality feedback metrics, prioritize question-answer pairs with lower matching scores in the updated local knowledge base, obtain an optimized tag set, and send it to the backend.

[0063] Specifically, two key service quality feedback metrics are obtained: user dwell time and repeat question frequency. User dwell time refers to the total time of a single session from start to finish, while repeat question frequency is calculated by counting the number of occurrences of intent parsing text with a semantic similarity higher than a certain threshold, such as 0.9, in the same session log. When a session is judged to be of low quality, for example, with a repeat question frequency greater than 1, it indicates that the user is not satisfied with the initial answer, and the system will focus its analysis on that session. The system will trace the question-answer pair IDs selected by the knowledge base matching and path decision module in that session and their matching scores at that time. To quantify the urgency of optimizing the question-answer pair, the system will calculate a priority label score for it. The score is determined by the following formula: , In this formula, The final priority marker score; This is the total frequency of the question-and-answer pair being triggered and resulting in repeated questions within the statistical period; It is the original matching score calculated by the semantic vectorization and similarity calculation unit when the question-answer pair is triggered, and its value is between 0 and 1. and These are pre-configured weighting coefficients used to balance the impact of repetition rate and match confidence, for example... It can be set to 0.6. It can be set to 0.4.

[0064] Calculated priority mark score It will be compared with a preset optimization trigger threshold. All question-answer pairs IDs that exceed the threshold and their corresponding priority scores will be aggregated to form a structured set of optimization tags, such as a JSON array. This set is finally sent to the backend knowledge base management platform through a dedicated API interface, where it will be highlighted on the management interface for operations personnel to review, modify, or supplement, thus completing a full closed loop from end-user feedback to backend knowledge base optimization.

[0065] For example, the service quality analysis and optimization feedback unit obtains complete interaction session logs from the session log generation unit. The system extracts service quality feedback indicators and identifies whether there are repeated questions with the same intent in the same session by analyzing the semantic similarity in the logs. Suppose that in a consultation about "parking fee standards," the user asks the question three times consecutively. The system identifies the total frequency of this question-and-answer pair being triggered and causing repeated questions within the statistical period. The question-and-answer pair obtained the original matching score in the semantic vectorization and similarity calculation unit. The system's preset weighting coefficients are known. The priority marker score is calculated as follows: The calculated priority score will be used to mark the priority. The value is compared with the preset optimization trigger threshold, such as 1.5. Since 1.94 > 1.5, the question and answer pair ID and its priority score are aggregated into the optimization tag set and sent to the back-end knowledge base management platform for operations personnel to make targeted entry corrections or expansions.

[0066] It should be noted that the electrical connections between the various units described above do not necessarily represent direct or indirect connections. Any indirect connection method can be applied to the embodiments of the present invention as long as it achieves the purpose of the present invention. The above descriptions are merely exemplary embodiments of the present invention and should not be construed as limiting the scope of the present invention.

[0067] All equivalent changes and modifications made in accordance with the teachings of this invention are still within the scope of this invention. Those skilled in the art will readily conceive of other embodiments of this invention upon considering the specification and the disclosure of practical truth. This application is intended to cover any variations, uses, or adaptations of this invention that follow the general principles of this invention and include common knowledge or conventional techniques in the art not described herein.

Claims

1. A multimodal AI digital human service system for transportation hubs, characterized in that, The system includes: a local knowledge base module, a multimodal perception preprocessing module, an intent parsing module, a knowledge base matching and path decision module, a large-scale intelligent response and security auditing module, a multimodal synthesis module, and a session management module; among which... The local knowledge base module is used to obtain the approved knowledge base update package and perform a versioned differential loading operation to generate the updated local knowledge base; The multimodal perception preprocessing module is used to acquire multimodal interaction requests and perform signal fusion and noise suppression processing to generate standardized interaction data; The intent parsing module is used to perform multilingual speech recognition and visual-assisted intent analysis on the standardized interaction data to generate intent parsing text; The knowledge base matching and path decision module is used to perform similarity matching judgment between the intent parsing text and the structured question-answer pairs in the updated local knowledge base, select the corresponding response path, and generate a response path selection result. The large-scale intelligent response and security audit module is used to call a preset large language model and generate initial response content based on the intent parsing text when the response path selection result is not covered, and to perform multi-dimensional content security verification to generate a safe and compliant response text. The multimodal synthesis module is used to simultaneously generate synthesized speech data and digital human animation sequences based on the safe and compliant response text; The session management module is used to generate a complete interactive session log based on the synthesized speech data and the digital human animation sequence.

2. The multimodal AI digital human service system for transportation hubs according to claim 1, characterized in that, The local knowledge base module includes: a version interaction and incremental acquisition unit, a security verification and conflict detection unit, and a differential loading and library update unit; wherein... The version interaction and incremental acquisition unit is used to periodically send the current local knowledge base version identifier to the backend server and receive the corresponding incremental update package. The security verification and conflict detection unit is connected to the version interaction and incremental acquisition unit, and is used to perform digital signature verification and content conflict detection on the incremental update package to obtain the verified update content. The differential loading and library update unit is connected to the security verification and conflict detection unit, and is used to replace the corresponding entries in the local knowledge base with the verified update content in a differential manner to generate an updated local knowledge base.

3. A multimodal AI digital human service system for transportation hubs according to claim 2, characterized in that, The multimodal perception preprocessing module includes: a multi-source heterogeneous data acquisition unit, a spatiotemporal feature enhancement and extraction unit, and a cross-modal temporal alignment and fusion unit; wherein... The multi-source heterogeneous data acquisition unit is connected to the differential loading and library update unit and is used to acquire speech signals and synchronous visual signals to obtain raw multi-source data. The spatiotemporal feature enhancement and extraction unit is connected to the multi-source heterogeneous data acquisition unit and is used to perform beamforming noise reduction and lip movement feature extraction on the original multi-source data to obtain enhanced speech features and visual auxiliary features. The cross-modal temporal alignment and fusion unit is connected to the spatiotemporal feature enhancement and extraction unit, and is used to temporally align and fuse the enhanced speech features and visual auxiliary features to generate standardized interactive data.

4. A multimodal AI digital human service system for transportation hubs according to claim 3, characterized in that, The intent parsing module includes: a speech recognition and decoding unit, an audiovisual fusion error correction unit, and an intent parsing text generation unit; wherein... The speech recognition and decoding unit is connected to the cross-modal temporal alignment and fusion unit, and is used to perform acoustic model decoding and language model correction on the speech part of the standardized interactive data to obtain candidate text sequences. The audiovisual fusion error correction unit is connected to the speech recognition and decoding unit and is used to correct easily confused pronunciations in the candidate text sequence using the visual auxiliary features to obtain a corrected text sequence. The intent parsing text generation unit is connected to the audiovisual fusion error correction unit and is used to perform semantic slot filling and traffic hub scene rule matching on the corrected text sequence to generate intent parsing text.

5. A multimodal AI digital human service system for transportation hubs according to claim 4, characterized in that, The knowledge base matching and path decision module includes: a semantic vectorization and similarity calculation unit, a coverage determination unit, and a response path selection unit; wherein... The semantic vectorization and similarity calculation unit is connected to the intent parsing text generation unit and is used to convert the intent parsing text into semantic vectors and perform distance calculation with the structured question-answer pair vectors in the updated local knowledge base to obtain a matching score. The coverage determination unit is connected to the semantic vectorization and similarity calculation unit, and is used to determine whether it belongs to the updated local knowledge base coverage range based on the relationship between the matching score and the preset threshold, and obtain the coverage determination result. The response path selection unit is connected to the coverage determination unit and is used to select either the direct knowledge base answer path or the large language model fallback path based on the coverage determination result, and generate a response path selection result.

6. A multimodal AI digital human service system for transportation hubs according to claim 5, characterized in that, The large-scale model intelligent response and security audit module includes: a large-scale model inference unit, a multi-dimensional security verification unit, and a content decision and text generation unit; wherein... The large model reasoning unit is connected to the response path selection unit. When the response path selection result is not covered, it inputs the intent parsing text and the updated traffic hub-specific constraint prompts in the local knowledge base into the large language model to obtain the initial answer content. The multi-dimensional security verification unit is connected to the large model reasoning unit and is used to sequentially perform policy compliance detection, sensitive information shielding, and factual accuracy verification on the initial response content to obtain the verification result. The content decision and text generation unit is connected to the multi-dimensional security verification unit and is used to rewrite or confirm the initial answer content based on the verification result to generate a safe and compliant answer text.

7. A multimodal AI digital human service system for transportation hubs according to claim 6, characterized in that, The multimodal synthesis module includes: a text prosody and sentiment analysis unit, a natural language speech synthesis unit, and a digital human animation generation unit; wherein... The text prosody and sentiment analysis unit, connected to the content decision and text generation unit, is used to perform prosody analysis and sentiment annotation on the safe and compliant answer text to obtain a prosodic text sequence. The natural speech synthesis unit is connected to the text prosody and sentiment analysis unit and is used to generate natural synthesized speech data based on the prosodic text sequence. The digital human animation generation unit is used to synchronously generate digital human lip-shape, facial expression, and gesture sequences that are consistent with the speech rhythm using the natural synthesized speech data, thereby obtaining a digital human animation sequence.

8. A multimodal AI digital human service system for transportation hubs according to claim 7, characterized in that, The session management module includes: a multi-channel output rendering unit, a visual interaction recording unit, and a session log generation unit; wherein... The multi-channel output rendering unit is connected to the digital human animation generation unit and is used to send the natural synthesized speech data to the full-band speaker and simultaneously send the digital human animation sequence to the high-definition touch screen for real-time rendering output. The visual interaction recording unit is connected to the multi-channel output rendering unit and is used to sequentially display the user intent parsing text and the safety and compliance answer text in the form of dialog bubbles on the high-definition touch screen to obtain a visual interaction record. The session log generation unit is connected to the visual interaction recording unit and is used to package the visual interaction record, synthesized speech data and digital human animation sequence to form a complete interactive session log.

9. A multimodal AI digital human service system for transportation hubs according to claim 1, characterized in that, The session management module further includes: a service quality analysis and optimization feedback unit; wherein... The service quality analysis and optimization feedback unit is connected to the session log generation unit and is used to obtain the user dwell time and the frequency of repeated questions in the complete interaction session log to obtain service quality feedback indicators. Based on the service quality feedback metrics, prioritize question-answer pairs with lower matching scores in the updated local knowledge base, obtain an optimized tag set, and send it to the backend.

Images