Airport intelligent inquiry service system and method based on multi-modal interaction

Abstract

The invention discloses an airport intelligent inquiry service system based on multi-mode interaction, which comprises a user interaction terminal, a multi-mode information processing center, an airport comprehensive database, a service response executor and the like, and adopts an attention weighted feature fusion algorithm to effectively overcome the limitation of a single mode. And a high recognition rate can still be kept in a noisy environment. By establishing a unified airport comprehensive database, an information island is broken, and real-time coordination of flight dynamics, facility navigation and service resources is ensured. The system has a self-learning capability, and can continuously optimize an intention recognition model according to historical interaction data, so that the response accuracy is continuously improved along with the use frequency. A multi-mode interaction mode is especially suitable for the use requirements of the elderly and international passengers, and a response mechanism combining physical and digital services provides barrier-free service support for the disabled. According to the system, a manual counter is liberated from repeated inquiry, so that airport human resources can concentrate on more complex service scenes.

Description

Technical Field

[0001] This invention relates to the field of airport service system technology, and in particular to an intelligent airport information service system and method based on multimodal interaction. Background Technology

[0002] As major transportation hubs, modern airports directly impact passenger travel experiences through their operational efficiency and service quality. Traditional airport information services rely primarily on manual counters, facing challenges such as limited human resources, restricted service hours, and outdated information. Digital information devices that have emerged in recent years often employ a single interaction mode, failing to adapt to diverse needs in complex environments: low voice recognition rates in noisy environments, inconvenience for passengers carrying large luggage with touchscreen operation, and text input barriers for the elderly and international travelers. Existing systems lack coordination between service modules, resulting in widespread information silos and an inability to provide passengers with a seamless multimodal interactive experience. Particularly in emergency scenarios such as flight delays and gate changes, traditional systems struggle to provide timely and accurate personalized guidance, leading to increasingly prominent issues such as passenger congestion and low service efficiency. Summary of the Invention

[0003] In view of this, in order to solve the problems existing in the technical background, the present invention proposes an intelligent airport information service system and method based on multimodal interaction. Specifically, it includes the following:

[0004] An intelligent airport information service system based on multimodal interaction includes a user interaction terminal, a multimodal information processing center, an airport integrated database, and a service response actuator. The user interaction terminal establishes a bidirectional data connection with the multimodal information processing center through the airport's internal communication network, collecting user voice, gesture, and text input information, and receiving feedback results from the multimodal information processing center for visualization. The multimodal information processing center interacts with the airport integrated database through a data interface, performing semantic parsing, intent recognition, and multimodal information fusion processing on the received raw interaction data to generate structured user query instructions. The airport integrated database stores real-time and historical data on airport flight dynamics, facility layout, service resources, and emergency procedures, providing data support for the multimodal information processing center. The service response actuator, based on the instructions output by the multimodal information processing center, drives physical service devices or calls digital service interfaces to complete the specific operations of the information service.

[0005] Furthermore, the user interaction terminal integrates voice acquisition, gesture sensing, text input, and display output modules; the voice acquisition module captures voice signals through a microphone array, the gesture sensing module recognizes hand gestures through a depth camera, the text input module receives text queries, and the display output module presents the response content.

[0006] Furthermore, the multimodal information processing center includes a multi-source information fusion engine, a natural language understanding unit, and a decision control unit; the multi-source information fusion engine employs a feature-level fusion method, and its fused features are represented as follows:

[0007] F_fused=α·F_voice+β·F_gesture+γ·F_text;

[0008] Where F_voice, F_gesture, and F_text are the feature vectors of speech, gesture, and text, respectively, and α, β, and γ are weight coefficients calculated based on the attention mechanism; the natural language understanding unit performs intent classification based on the fused features, and the decision control unit generates service instructions based on the classification results.

[0009] Furthermore, the airport integrated database includes a flight information database, a facility map database, a service rule database, and a user profile database; the flight information database stores flight dynamic data, the facility map database stores terminal facility location information, the service rule database stores standard response templates, and the user profile database stores user history records and preference settings under authorization.

[0010] Furthermore, the service response executor includes a digital interface calling module and a physical device driving module; the digital interface calling module sends information to the user's mobile terminal via API, and the physical device driving module controls the self-service check-in equipment, signage display screen or robot guide device to perform service operations.

[0011] A method for providing intelligent airport information services based on multimodal interaction includes: collecting multimodal information requests input by users via voice, gestures, or text through a user interaction terminal; transmitting the multimodal information requests to a multimodal information processing center, where the center performs fusion analysis and semantic understanding on the multi-source interactive data to generate a structured user query intent; querying relevant real-time data and service resource information in the airport's comprehensive database based on the user query intent; generating service instructions and response content based on the query results, and sending them to a service response executor; and driving a digital interface or physical device through the service response executor to provide feedback on the information request results or perform corresponding service operations to the user.

[0012] The above technical solution has the following beneficial effects:

[0013] This invention employs an attention-weighted feature fusion algorithm to effectively overcome the limitations of single-modality systems, maintaining a high recognition rate even in noisy environments. By establishing a unified airport comprehensive database, information silos are broken down, ensuring real-time coordination of flight dynamics, facility navigation, and service resources. The system possesses self-learning capabilities, continuously optimizing the intent recognition model based on historical interaction data, thus improving response accuracy with usage frequency. The multimodal interaction mode is particularly suitable for the needs of elderly users and international travelers, while the response mechanism combining physical and digital services provides barrier-free service support for those with mobility impairments. This system frees up manual counters from repetitive inquiries, allowing airport human resources to focus on more complex service scenarios, improving overall operational efficiency by approximately 40%. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of an airport intelligent information service system based on multimodal interaction according to the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] See Figure 1 The system illustrates an intelligent airport information service system based on multimodal interaction, comprising a user interaction terminal, a multimodal information processing center, an airport integrated database, and a service response actuator. The user interaction terminal establishes a bidirectional data connection with the multimodal information processing center via the airport's internal communication network, collecting user voice, gesture, and text input information, and receiving feedback results from the multimodal information processing center for visualization. The multimodal information processing center interacts with the airport integrated database through a data interface, performing semantic analysis, intent recognition, and multimodal information fusion processing on the received raw interaction data to generate structured user query instructions. The airport integrated database stores real-time and historical data on airport flight dynamics, facility layout, service resources, and emergency procedures, providing data support for the multimodal information processing center. The service response actuator, based on instructions output by the multimodal information processing center, drives physical service devices or invokes digital service interfaces to complete the specific operations of the information service.

[0017] The user interaction terminal integrates voice acquisition, gesture sensing, text input, and display output modules; the voice acquisition module captures voice signals through a microphone array, the gesture sensing module recognizes hand gestures through a depth camera, the text input module receives text queries, and the display output module presents the response content.

[0018] The multimodal information processing center includes a multi-source information fusion engine, a natural language understanding unit, and a decision control unit; the multi-source information fusion engine employs a feature-level fusion method, and its fused features are represented as follows:

[0019] F_fused=α·F_voice+β·F_gesture+γ·F_text;

[0020] Where F_voice, F_gesture, and F_text are the feature vectors of speech, gesture, and text, respectively, and α, β, and γ are weight coefficients calculated based on the attention mechanism; the natural language understanding unit performs intent classification based on the fused features, and the decision control unit generates service instructions based on the classification results.

[0021] In this embodiment, the airport integrated database includes a flight information database, a facility map database, a service rule database, and a user profile database. The flight information database stores flight dynamic data, the facility map database stores terminal facility location information, the service rule database stores standard response templates, and the user profile database stores user history records and preference settings under authorization.

[0022] The service response executor includes a digital interface calling module and a physical device driving module; the digital interface calling module sends information to the user's mobile terminal via API, and the physical device driving module controls the self-service check-in equipment, signage display screen or robot guide device to perform service operations.

[0023] A method for providing intelligent airport information services based on multimodal interaction includes: collecting multimodal information requests input by users via voice, gestures, or text through a user interaction terminal; transmitting the multimodal information requests to a multimodal information processing center, where the center performs fusion analysis and semantic understanding on the multi-source interactive data to generate a structured user query intent; querying relevant real-time data and service resource information in the airport's comprehensive database based on the user query intent; generating service instructions and response content based on the query results, and sending them to a service response executor; and driving a digital interface or physical device through the service response executor to provide feedback on the information request results or perform corresponding service operations to the user.

[0024] During system operation, users first submit inquiries through a user interaction terminal integrating a microphone array, depth camera, and touchscreen. The voice acquisition module uses beamforming technology for directional sound pickup and performs noise reduction through spectral subtraction; the gesture sensing module extracts the three-dimensional coordinates of key points using the OpenPose algorithm and recognizes preset commands such as pointing and waving; the text input module supports virtual keyboard and handwriting input. Data from each modality is synchronized via timestamps and then transmitted to the multimodal information processing center.

[0025] The core of the multimodal information processing center is a multi-source information fusion engine, which employs a feature fusion method based on an attention mechanism. Specifically, it first extracts the MFCC feature vector F_voice from the speech signal, the spatiotemporal feature vector F_gesture from the gesture skeleton sequence, and the BERT embedding vector F_text from the text using a pre-trained model. The attention weights are calculated using the following formula:

[0026] α=softmax(W_a·[F_voice;F_gesture;F_text]);

[0027] Where W_a is the trainable parameter matrix, and [;] denotes vector concatenation. The weighted and fused features F_fused are input to the natural language understanding unit for intent classification, outputting structured instructions such as "query the boarding gate" or "navigate to the restaurant". The decision control unit invokes the corresponding service strategy according to the intent type; for example, navigation requests require the generation of the optimal path based on real-time location data.

[0028] The airport's integrated database employs a distributed architecture to ensure data consistency. The flight information database updates flight status in real time via ADS-B signal receivers; the facility map database stores BIM models containing topological relationships; the service rule database defines response logic for different scenarios (such as automatically pushing rebooking instructions in case of delays); and the user profile database stores personalized settings after passing GDPR compliance certification. When querying "nearest restroom," the system returns the optimal result by considering the user's current location, facility usage frequency, and accessibility requirements.

[0029] The service response executor triggers corresponding operations based on the instruction type. The digital interface call module pushes navigation routes including real-time distances to passengers' mobile phones via RESTful API; the physical device drive module controls the robotic guide device to lead passengers to the target area via the Modbus protocol, or drives electronic signs to update guidance information. In emergency scenarios, the system can simultaneously activate multiple terminal devices to achieve full-area broadcast guidance.

[0030] The system optimizes service quality through a continuous learning mechanism. After each interaction, the system records multimodal data and the final service result to form training samples, and incrementally updates the intent classification model weekly. Through A / B testing, after three months of operation, the accuracy of the first-round parsing of complex queries improved from 72% initially to 89%, and the average response time was reduced to 1.8 seconds.

[0031] The basic principles and main features of the present invention have been described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are only illustrative of the principles of the present invention. Various changes and modifications can be made to the present invention without departing from the spirit and scope of the present invention. All such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the invention is defined by the appended claims and their equivalents.

Claims

1. An airport intelligent inquiry service system based on multi-modal interaction, characterized in that, The application relates to an airport inquiry service system, which comprises a user interaction terminal, a multi-modal information processing center, an airport comprehensive database and a service response executor; the user interaction terminal is connected with the multi-modal information processing center through an airport internal communication network to collect user voice, gesture and text input information and receive feedback results from the multi-modal information processing center for visual presentation; the multi-modal information processing center interacts with the airport comprehensive database through a data interface to perform semantic analysis, intention recognition and multi-modal information fusion processing on received original interaction data to generate a structured user query instruction; The airport comprehensive database stores real-time and historical data of airport flight dynamics, facility layout, service resources and emergency processes to provide data support for the multi-modal information processing center; the service response executor drives physical service equipment or calls a digital service interface to complete specific operations of inquiry services according to the instruction output by the multi-modal information processing center. 2.The airport intelligent inquiry service system based on multi-modal interaction of claim 1, wherein, The user interaction terminal integrates a voice collection module, a gesture sensing module, a text input module and a display output module; the voice collection module captures voice signals through a microphone array, the gesture sensing module identifies gesture actions through a depth camera, the text input module receives a text query, and the display output module is used for presenting response content. 3.The airport intelligent inquiry service system based on multi-modal interaction of claim 1, wherein, The multi-modal information processing center comprises a multi-source information fusion engine, a natural language understanding unit and a decision control unit; the multi-source information fusion engine adopts a feature-level fusion method, and a fused feature is expressed as: F_fused = alpha * F_voice + beta * F_gesture + gamma * F_text; wherein F_voice, F_gesture and F_text are feature vectors of voice, gesture and text respectively, alpha, beta and gamma are weight coefficients calculated based on an attention mechanism; the natural language understanding unit performs intention classification based on the fused feature, and the decision control unit generates a service instruction according to a classification result.

4. The airport intelligent inquiry service system based on multi-modal interaction according to claim 1, characterized in that, The airport comprehensive database comprises a flight information library, a facility map library, a service rule library and a user archive library; the flight information library stores flight dynamic data, the facility map library stores terminal facility position information, the service rule library stores standard response templates, and the user archive library stores user historical records and preference settings under authorization.

5. The airport intelligent inquiry service system based on multi-modal interaction according to claim 1, characterized in that, The service response executor comprises a digital interface calling module and a physical device driving module; The digital interface calling module sends information to a user mobile terminal through an API, and the physical device driving module controls a self-service check-in device, an indicator display screen or a robot guide device to perform a service operation.

6. An airport intelligent inquiry service method based on multi-modal interaction, characterized in that, The application relates to an airport inquiry service system, which comprises a user interaction terminal, a multi-modal information processing center, an airport comprehensive database and a service response executor; the user interaction terminal is connected with the multi-modal information processing center through an airport internal communication network to collect user voice, gesture and text input information and receive feedback results from the multi-modal information processing center for visual presentation; the multi-modal information processing center interacts with the airport comprehensive database through a data interface to perform semantic analysis, intention recognition and multi-modal information fusion processing on received original interaction data to generate a structured user query instruction; The airport comprehensive database stores real-time and historical data of airport flight dynamics, facility layout, service resources and emergency processes to provide data support for the multi-modal information processing center; the service response executor drives physical service equipment or calls a digital service interface to complete specific operations of inquiry services according to the instruction output by the multi-modal information processing center. The user interaction terminal integrates a voice collection module, a gesture sensing module, a text input module and a display output module; the voice collection module captures voice signals through a microphone array, the gesture sensing module identifies gesture actions through a depth camera, the text input module receives a text query, and the display output module is used for presenting response content. The multi-modal information processing center comprises a multi-source information fusion engine, a natural language understanding unit and a decision control unit; the multi-source information fusion engine adopts a feature-level fusion method, and a fused feature is expressed as: F_fused = alpha * F_voice + beta * F_gesture + gamma * F_text; wherein F_voice, F_gesture and F_text are feature vectors of voice, gesture and text respectively, alpha, beta and gamma are weight coefficients calculated based on an attention mechanism; the natural language understanding unit performs intention classification based on the fused feature, and the decision control unit generates a service instruction according to a classification result. The airport comprehensive database comprises a flight information library, a facility map library, a service rule library and a user archive library; the flight information library stores flight dynamic data, the facility map library stores terminal facility position information, the service rule library stores standard response templates, and the user archive library stores user historical records and preference settings under authorization. The service response executor comprises a digital interface calling module and a physical device driving module; The digital interface calling module sends information to a user mobile terminal through an API, and the physical device driving module controls a self-service check-in device, an indicator display screen or a robot guide device to perform a service operation. The service response executor drives the digital interface or physical device to feedback the interrogation result to the user or execute the corresponding service operation.

Images