Cushion work data standardization analysis system based on multi-modal model

Abstract

The invention discloses a crew work data standardization analysis system based on a multi-modal model, relates to the technical field of communication monitoring, and solves the problem that the crew work quality is not easy to improve due to the fact that video data volume is large and coverage analysis is difficult to realize. The crew work data standardization analysis system comprises a multi-mode data acquisition layer, a multi-mode data processing layer, a multi-mode fusion analysis layer and an intelligent decision optimization layer. Performing intelligent coverage analysis on the operation standard of each steward by combining all branches of the multi-modal fusion inference model with the multi-modal data of each compartment of the vehicle; the crew work quality of the crew is improved, and the reasoning speed of the multi-modal data is improved.

Description

Technical Field

[0001] The present invention relates to the field of communication monitoring technology, and in particular to a crew operation data standardization analysis system based on a multimodal model. Background Art

[0002] With the expansion of high-speed rail and other transportation sectors, crew operations are becoming increasingly complex, involving multi-dimensional data such as personnel scheduling, equipment operation, safety inspections, and service interactions. Traditional single-dimensional analysis models struggle to fully reflect operational quality. Crew data is often heterogeneous and multi-source: unstructured data such as voice recordings, video surveillance, text logs, and sensor signals accounts for over 70%, resulting in less than 30% data utilization. Upgraded industry safety regulations (such as ISO 45001) and rising service quality requirements have necessitated the development of a cross-modal data fusion analysis framework.

[0003] The breakthrough development of artificial intelligence technology has made it possible to solve this pain point. Multimodal learning technology can effectively integrate multi-dimensional information such as vision, voice, and text, and through feature alignment and cross-modal association mining, it can achieve intelligent evaluation of core indicators such as the standardization of crew operations and the timeliness of service response.

[0004] Patent No. CN202410284468.1 discloses a one-button emergency escape data analysis system and method based on the Internet of Things. First, real-time data collection is performed on the vehicle, the collected data is preprocessed, and the processed data is analyzed to obtain a safety status assessment of the vehicle, a passenger health status assessment model is constructed, and the health of the vehicle passengers is assessed; then, based on the preliminary safety status assessment report, an escape strategy is formulated, and the escape strategy is encrypted and anonymized to obtain an encrypted and anonymized escape strategy recommendation; the preprocessed data is deeply integrated with the encrypted and anonymized escape strategy recommendation to generate a comprehensive emergency assessment; finally, based on the comprehensive emergency assessment report, a priority classifier is used to prioritize the information. This solves the technical problems in the existing technology of inaccurate and timely analysis of emergency escape data, as well as lack of flexibility.

[0005] Patent No. CN202310478421.4 discloses a method and system for identifying, judging and guiding visual work specifications for security inspectors. The method includes the following steps: acquiring images of inspected items in real time, and capturing the visual behavior data of security inspectors on the images of inspected items in real time; generating key area attention prediction information for the images of inspected items using a key area attention prediction model; comprehensively evaluating the images of inspected items, key area attention prediction information, and the visual behavior data of security inspectors on the images of inspected items to determine the quality of visual work behavior; and conducting work guidance and review processing based on the determined visual work behavior. The above invention can solve the problem of real-time perception and autonomous control of personnel risks during security inspection operations.

[0006] However, the above patents and the current vehicle crew work process require the use of work recorders to conduct work record video analysis, and then the workshop analyzes the video. However, due to the large amount of video data, it is difficult to cover the analysis well, resulting in difficulty in improving the quality of crew work. Summary of the Invention

[0007] The purpose of the present invention is to provide a standardized analysis system for crew operation data based on a multimodal model, which can combine the multimodal data of each vehicle compartment through all branches of the multimodal fusion reasoning model to perform intelligent coverage analysis on the operating standards of each crew member; so as to improve the quality of the crew's service operations and increase the reasoning speed of multimodal data.

[0008] The present invention utilizes the following technical solutions: A standardized analysis system for crew operation data based on a multimodal model, including a multimodal data acquisition layer, a multimodal data processing layer, a multimodal fusion analysis layer and an intelligent decision optimization layer; wherein, The multimodal data collection layer is used to obtain multimodal operational data of train crew members, including video data, voice data, text data, and environmental data. The multimodal data processing layer is used to clean and anomaly-annotate multimodal operation data to obtain a data label sequence table; The multimodal fusion analysis layer uses a multimodal fusion inference model to jointly analyze the cleaned multimodal operation data based on the data label sequence table, obtain a full-scene crew operation map, and determine the crew operation rating; The intelligent decision-making optimization layer is used to globally verify the full-scenario crew operation map based on the train crew operation standard manual and generate crew operation disposal plans.

[0009] Preferably, the multi-mode data acquisition layer uses a monitoring device to record the video stream of crew operations, passenger status and equipment dashboards in the railway train to obtain video data; at the same time, a microphone array is used to collect the audio stream of crew broadcasts, passenger conversations and emergency calls in the railway train to obtain voice data; at the same time, an electronic log is used to record the text files of crew schedules, event record logs, fault repair orders and service texts in the railway train to obtain text data; at the same time, a sensor network is used to collect the time series data streams of temperature, humidity, oxygen content, door status and seat pressure in the railway train to obtain environmental data; the multi-mode data acquisition layer associates and fuses the video data, voice data, text data and environmental data according to the crew operation scenario to obtain the multi-modal operation data of the crew.

[0010] Preferably, the multi-mode data processing layer uses a decomposition conversion algorithm to slice and convert the video data to obtain several frames of images; and uses an image filtering algorithm to denoise and deblur each frame of the image to obtain a noise-free image; at the same time, a facial recognition algorithm is used to identify and mark the faces in the noise-free image, and then a random projection hash algorithm is used to desensitize the marked faces to obtain a non-sensitive image; The multi-mode data processing layer uses a wavelet domain noise reduction algorithm to reduce the noise of the voice data to obtain a noise-free frequency spectrum. It also uses a voiceprint segmentation and clustering algorithm based on the crew voice library to distinguish between crew members and passengers in the noise-free frequency spectrum, obtaining a crew audio set and a passenger audio set. The multi-mode data processing layer uses a sensitive word extraction algorithm to extract keywords from the clean text content based on the sensitive word list, obtaining key information about the train event: crew recording time, crew member ID, event time, and the carriage where the event occurred. The multi-mode data processing layer uses a data classification algorithm to classify environmental data according to sensor type to obtain an initial sensor data table. It then uses a filtering detection algorithm to filter the initial sensor data table to obtain a uniform sensor data table. It then uses a time series alignment algorithm to correlate and fuse the uniform sensor data tables based on timestamps and carriage numbers to obtain a full-state environmental data table. The multimodal data processing layer uses the data annotation network to annotate the non-sensitive images, crew audio sets, passenger audio sets and full-state environment data tables according to the key information of train events, and obtains a data label sequence table.

[0011] Preferably, the operation process of the multimodal fusion reasoning model includes the following steps: The visual parsing branch combines a spatiotemporal dual-frame network to extract the scene state of each frame of the non-sensitive image and capture the crew equipment operation motion to obtain the global matrix of the car; The facial analysis branch uses a micro-expression encoding network combined with a posture estimation network to quantify the flight attendant's micro-expressions and identify micro-motion deviations in each frame of the non-sensitive image to obtain a flight attendant local matrix. The semantic decoupling branch uses a speech representation network to separate the speech content and paralinguistic features of the crew audio set to generate semantic vectors and emotion vectors. The voiceprint mapping branch uses a voiceprint recognition network to identify and cluster the passenger audio set and perform voice mapping to construct a passenger-voice mapping matrix; The text extraction branch uses a pattern knowledge extraction network combined with a time series understanding network to convert key train event information into event tuples and obtain an event time series matrix. The signal fusion branch uses a multi-scale convolutional network combined with timestamps to synchronize the global environment data table to obtain a heterogeneous data synchronization matrix.

[0012] Preferably, the operation process of the multimodal fusion reasoning model further includes the following steps: The abnormality judgment branch uses a variational autoencoder to learn the full-state environmental data distribution of the train's normal operating conditions and calculate the reconstruction error matrix; The hierarchical association branch uses a spatiotemporal cross attention network to calculate the association weights of the car global matrix and the crew local matrix combined with the heterogeneous data synchronization matrix according to the timestamp to obtain a dynamic spatiotemporal weight matrix. The semantic alignment branch uses a self-supervised contrastive learning network to verify the phenomena of the car global matrix and the crew local matrix combined with the semantic vector according to the car number, and construct a semantic-physical alignment space; The graph construction branch uses an information mapping algorithm to define nodes and edges by combining the car global matrix, crew local matrix, and event time series matrix with the reconstruction error matrix according to the crew number, car number, and passenger identification, and then constructs a full-scene crew operation graph.

[0013] Preferably, the causal reasoning branch of the multi-modal fusion analysis layer adopts a linear non-Gaussian acyclic network to perform causal mining on the full-scene crew operation map combined with the reconstructed error matrix to obtain the event causal reasoning chain; The simulation back-calculation branch uses a digital twin algorithm to rehearse crew operations on the event time series matrix based on the semantic-physical alignment space and the dynamic spatiotemporal weight matrix to predict passenger satisfaction levels. The crew evaluation branch quantifies the operating process of each crew member in the crew local matrix based on the event causal reasoning chain and the event time series matrix, combined with passenger satisfaction, to determine the crew operation level.

[0014] Preferably, the intelligent decision optimization layer uses a named entity recognition network to extract entities from the train crew operating standard manual, construct atomic rules, and convert fuzzy descriptions into time-quantized instructions; at the same time, it converts railway safety regulations into logical execution rules; A rule mapping algorithm is used to map the multimodal data from historical excellent flight attendant operation cases with atomic rules and logical execution rules to construct a case standard mapping table; Traverse all graph nodes in the full-scenario crew operation graph, and match the crew operation level with the case standard mapping table to obtain the crew standard operation graph; At the same time, a graph pattern matching algorithm is used to detect violations of the full-scene crew operation graph combined with the passenger satisfaction level to obtain the operation violation subgraph; The execution time thresholds in the atomization rules and logic execution rules are modified according to the train speed to obtain a dynamic time threshold; Based on the standard crew operation diagram and operation violation sub-diagram combined with dynamic time thresholds, a compliance score report is generated, and combined with the heterogeneous data synchronization matrix, a real-time risk heat map in each carriage is marked.

[0015] Preferably, the intelligent decision optimization layer constructs a full-scenario operation state space based on the compliance score report and the real-time status of the train, according to the preset execution time threshold and service coverage combined with the safety operation time and crew working hours; In the full-scenario operation state space, event simulation is performed based on the real-time risk heat map of all carriages and the dispatch task type, generating dynamic decision-making plans and resource conflict warnings; Based on the event type, combined with the crew portrait library and real-time train positioning data, the critical path algorithm is used to optimize the dynamic decision-making plan and generate a crew operation disposal plan.

[0016] The present invention combines all branches of the multimodal fusion reasoning model with the multimodal data of each vehicle compartment to perform intelligent coverage analysis on the operating standards of each crew member, thereby improving the quality of the crew's service operations and increasing the reasoning speed of multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0018] Figure 1 Schematic diagram of the crew operation data standardization analysis system; Figure 2 Schematic diagram of the principle of multimodal fusion reasoning model. DETAILED DESCRIPTION

[0019] The present invention is described in detail below with reference to the accompanying drawings and embodiments: like Figures 1 to 2 As shown, the crew operation data standardization analysis system based on the multimodal model of the present invention includes a multimodal data acquisition layer, a multimodal data processing layer, a multimodal fusion analysis layer and an intelligent decision optimization layer; wherein, The multimodal data collection layer is used to obtain multimodal operational data of train crew members, including video data, voice data, text data, and environmental data. The multimodal data processing layer is used to clean and anomaly-annotate multimodal operation data to obtain a data label sequence table; The multimodal fusion analysis layer uses a multimodal fusion inference model to jointly analyze the cleaned multimodal operation data based on the data label sequence table, obtain a full-scene crew operation map, and determine the crew operation rating; The intelligent decision-making optimization layer is used to globally verify the full-scenario crew operation map based on the train crew operation standard manual and generate crew operation disposal plans.

[0020] In the present invention, the multi-mode data acquisition layer uses a monitoring device to record the video streams of crew operations, passenger status and equipment dashboards in the railway train to obtain video data; at the same time, a microphone array is used to collect the audio streams of crew broadcasts, passenger conversations and emergency calls in the railway train to obtain voice data; at the same time, an electronic log is used to record the text files of crew schedules, event record logs, fault repair orders and service texts in the railway train to obtain text data; at the same time, a sensor network is used to collect the time-series data streams of temperature, humidity, oxygen content, door status and seat pressure in the railway train to obtain environmental data; the multi-mode data acquisition layer associates and fuses the video data, voice data, text data and environmental data according to the crew operation scenario to obtain the multi-modal operation data of the crew.

[0021] In this embodiment, the sensor network includes a temperature and humidity composite sensor, a gas sensor, a door switch sensor, and a pressure sensor; Crew operation scenarios include pre-departure inspection, catering supply (meal cart route planning, response to special dietary needs), environmental maintenance (temperature and humidity adjustment, lighting mode switching, toilet cleaning inspections), information broadcasting (arrival reminders, weather warnings, transfer information), equipment operation monitoring (real-time monitoring of air-conditioning system and braking system sensor data), passenger status observation (through video analysis of abnormal passenger behavior, such as prolonged stay in the toilet, passenger fainting, heart attack, sudden infectious disease symptoms), equipment failure (doors / cabin doors cannot be closed, air-conditioning system shutdown, lighting system failure), natural disasters (heavy rain causing water accumulation on the track, strong winds causing turbulence, earthquake warnings), man-made threats (terrorist attack threats, illegal carrying of dangerous goods, malicious interference with driving) and internal collaboration of the crew (equipment status during shift rotation, unfinished events, handover of key passenger information, dynamic deployment of crew members during peak hours).

[0022] In the present invention, the multi-mode data processing layer uses a decomposition conversion algorithm to slice and convert the video data to obtain several frames of images; and uses an image filtering algorithm to denoise and deblur each frame of the image to obtain a noise-free image; at the same time, a facial recognition algorithm is used to identify and mark the faces in the noise-free image, and then a random projection hash algorithm is used to desensitize the marked faces to obtain a desensitized image; The multi-mode data processing layer uses a wavelet domain noise reduction algorithm to reduce the noise of the voice data to obtain a noise-free frequency spectrum. It also uses a voiceprint segmentation and clustering algorithm based on the crew voice library to distinguish between crew members and passengers in the noise-free frequency spectrum, obtaining a crew audio set and a passenger audio set. The multi-mode data processing layer uses a sensitive word extraction algorithm to extract keywords from the clean text content based on the sensitive word list, obtaining key information about the train event: crew recording time, crew member ID, event time, and the carriage where the event occurred. The multi-mode data processing layer uses a data classification algorithm to classify environmental data according to sensor type to obtain an initial sensor data table. It then uses a filtering detection algorithm to filter the initial sensor data table to obtain a uniform sensor data table. It then uses a time series alignment algorithm to correlate and fuse the uniform sensor data tables based on timestamps and carriage numbers to obtain a full-state environmental data table. The multimodal data processing layer uses the data annotation network to annotate the non-sensitive images, crew audio sets, passenger audio sets and full-state environment data tables according to the key information of train events, and obtains a data label sequence table.

[0023] In this embodiment, the decomposition and transformation algorithm, image filtering algorithm, facial recognition algorithm, random projection hash algorithm, wavelet domain noise reduction algorithm, voiceprint segmentation and clustering algorithm, sensitive word extraction algorithm, data classification algorithm, filtering detection algorithm, time series alignment algorithm, and data annotation network are all commonly used technical means in this field and will not be repeated here.

[0024] In the present invention, the operation process of the multimodal fusion reasoning model includes the following steps: Visual parsing branch combined with spatiotemporal dual-frame network No sensitive image for each frame Perform scene state extraction and crew equipment operation motion capture to obtain the car global matrix ; In this embodiment, the visual parsing branch includes four 3X3 convolutional layers with a stride of 2 and one batch normalization layer; ; in, express Step length The convolution kernel, Indicates cascade operation, For batch normalization, Indicates a timestamp; The SlowFast network is used to decompose the video stream. The slow path (low frame rate) extracts scene context (such as the overall state of the vehicle), and the fast path (high frame rate) captures fast action details (such as emergency braking gestures). The facial analysis branch uses a micro-expression encoding network Combined with pose estimation network The micro-expression quantification and micro-motion deviation identification of the crew members are performed on each frame of the non-sensitive image to obtain the crew local matrix ; In this embodiment, the facial analysis branch includes two 3X3 convolutional layers with a stride of 1, three 5X5 convolutional layers with a stride of 2, and one normalization layer. ; ; ; ; The Facial Action Coding System (FACS) is used to quantify flight attendants' micro-expressions (such as frown frequency), and OpenPose is used to identify gesture standardization (such as the angle deviation of the finger's oral movement).

[0025] The semantic decoupling branch uses a speech representation network Audio collection for crew members Separate the speech content and paralinguistic features to generate semantic vectors and sentiment vector ; In this embodiment, the semantic decoupling branch includes 1 bidirectional LSTM layer, 4 hidden state layers and 2 fully connected layers; the bidirectional LSTM output is separated after 4 hidden layers; ; ; ; ; ; represents transpose; is the activation function; Use the HuBERT model to separate speech content ("Please fasten your seat belts") and paralinguistic features (speech rate, intonation), generating semantic vectors (768 dimensions) and sentiment vectors (128 dimensions) respectively. The voiceprint mapping branch uses the voiceprint recognition network Passenger Audio Collection Perform identity clustering and voice mapping to build a passenger-voice mapping matrix ; In this embodiment, the voiceprint mapping branch includes two unidirectional LSTM layers and three hidden state layers. and 1 fully connected layer; ; is the activation function; Based on the speaker recognition model (ECAPA-TDNN), passenger audio is clustered into independent IDs and a "passenger-voice" mapping relationship is constructed (for example, ID_25 corresponds to the passenger in row 1 of carriage 3).

[0026] The text extraction branch uses pattern knowledge extraction network Understanding Networks with Time Series , convert the key information of train events into event tuples and obtain the event time series matrix ; In this embodiment, the text extraction branch includes 2 Inception layers and 1 average pooling layer. and 2 fully connected layers ; 3 Inception layers followed by average pooling and full connection; ; Use the Schema-based BERT model to convert the key information of the train event ("water leakage was found in seat 3A at 10:05") into structured event triples: <Event Type: Equipment Failure, Location: Carriage 3, Seat A, Time: 10:05>; Use the TimeBERT model to identify time expressions in the text ("unprocessed for 15 minutes") and construct an event timeline; The signal fusion branch uses a multi-scale convolutional network Combined with timestamp Synchronize the full-state environment data table to obtain the heterogeneous data synchronization matrix ; In this embodiment, the signal fusion branch includes two residual blocks , 5 max pooling layers , 2 fully connected layers and LeakyReLU activation function; residual blocks and pooling layers are processed alternately; ; A multi-scale convolutional network (MS-CNN) is used to synchronously process heterogeneous sensor signals such as vibration (high frequency), temperature (low frequency), and pressure (pulse).

[0027] The anomaly judgment branch uses a variational autoencoder Learning the full environmental data distribution of normal train operation , calculate the reconstruction error matrix ; In this embodiment, the anomaly determination branch includes 2 CBL blocks, 3 residual blocks, 3 InceptionV2 blocks and Silu activation function; the CBL block includes 2 5X5 convolutional layers with a stride of 1, 1 batch standard layer and LeakyRelu activation function; ; represents the square of L2 norm; A variational autoencoder (VAE) is used to learn the distribution of sensor data under normal operating conditions and calculate the reconstruction error as an anomaly score (for example, a warning is triggered when the door sensor error is greater than 0.8).

[0028] The hierarchical association branch uses a spatiotemporal cross attention network According to the timestamp, the global matrix of the carriage is and crew local matrix Combine heterogeneous data synchronization matrix to calculate association weights , and obtain the dynamic spatiotemporal weight matrix ; In this embodiment, the hierarchical association branch includes two inverted residual blocks, three maximum pooling layers, three fully connected layers, and two RELU activation functions; the spatiotemporal cross attention network is implemented by the inverted residual blocks; ; Design a spatiotemporal cross-attention network (ST-CrossAttn) to calculate the association weights between video clips and contemporaneous sensor data. For example, when "smoke" is detected in a video, the attention weight of the smoke sensor within 10 seconds is automatically increased to 0.93. The semantic alignment branch uses a self-supervised contrastive learning network According to the carriage number, the carriage global matrix and the crew local matrix are combined with the semantic vector to verify the phenomenon and construct the semantic-physical alignment space. ; In this embodiment, the semantic alignment branch includes three Transformer encoding blocks , 2 batch standard layers, fusion attention mechanism and 2 MISH functions; ; The graph construction branch uses information mapping algorithm to calculate the crew number , car number and passenger ID, and then combine the car global matrix, crew local matrix and event time series matrix to reconstruct the error matrix for node Definition and Edge Definition, and then build a full-scenario crew operation map .

[0029] In this embodiment, the graph construction branch includes 2 CBS blocks, 3 inverted residual blocks , 3 mixed cross layers , 1 InceptionV4 block and 1 Silu activation function; CBS block It includes 2 5X5 convolutional layers with a step size of 2, 2 3X3 convolutional layers with a step size of 1, and 2 batch standard layers. And 2 Softmax activation functions.

[0030] ; ; ; ; Node types: flight attendant (features: operation history and voice features), equipment (features: sensor status and maintenance records), passenger (features: voiceprint ID and complaint records); Edge types: perform an action (crew member to device), trigger an alarm (device to system), and trigger a feedback (device to passenger). When a new event is detected (e.g., “a passenger presses the emergency call button”), a node is inserted and associated edges are connected (passenger → button → flight attendant) in real time.

[0031] In the present invention, the causal reasoning branch of the multi-mode fusion analysis layer adopts a linear non-Gaussian acyclic network , conduct causal mining on the full-scene crew operation graph combined with the reconstructed error matrix to obtain the event causal reasoning chain ; In this embodiment, ; The simulation back-stepping branch uses a digital twin algorithm Based on the semantic-physical alignment space and dynamic spatiotemporal weight matrix, the flight crew operation is previewed on the event time series matrix to predict the passenger satisfaction level. ; In this embodiment, ; represents a composite mapping function; Applying the LiNGAM algorithm to analyze multimodal time series data, we can identify causal relationships: voice command delay (cause) to operation timeout (effect) to passenger complaints (effect); Build a hypothetical scenario: If the flight attendant announces the delay 2 minutes earlier (modifying the voice modal data), predict the change in passenger satisfaction (the text modal sentiment analysis value will increase by 15%). The crew evaluation branch quantifies the crew's operating process and passenger satisfaction in the crew local matrix based on the event causal reasoning chain and the event time series matrix to determine the crew's operating level. .

[0032] In this embodiment, ; represents the normalization function; The classification of cabin crew work levels is usually based on dimensions such as scope of responsibilities, skill requirements, and management authority. The job grade system is shown in Table 1: Table 1 Job grade system In this invention, the intelligent decision optimization layer uses a named entity recognition network to extract entities from the train crew operating standard manual, construct atomized rules, and convert fuzzy descriptions into time-quantized instructions; at the same time, it converts railway safety regulations into logical execution rules; In this embodiment, railway safety regulations are converted into executable logic rules: <Rule ID: SOP_202> When the vehicle speed is >120km / h, the crew member must not leave the cockpit (video positioning + door sensor joint verification)>; A rule mapping algorithm is used to map the multimodal data from historical excellent flight attendant operation cases with atomic rules and logical execution rules to construct a case standard mapping table; In this embodiment, multimodal data packets of historical events (such as voice recordings, video clips, and sensor curves of the "brake failure event in XXXX") are stored to support similar case matching.

[0033] Traverse all graph nodes in the full-scenario crew operation graph, and match the crew operation level with the case standard mapping table to obtain the crew standard operation graph; At the same time, a graph pattern matching algorithm is used to detect violations of the full-scene crew operation graph combined with the passenger satisfaction level to obtain the operation violation subgraph; The execution time thresholds in the atomization rules and logic execution rules are modified according to the train speed to obtain a dynamic time threshold; Based on the standard crew operation diagram and operation violation sub-diagram combined with dynamic time thresholds, a compliance score report is generated, and combined with the heterogeneous data synchronization matrix, a real-time risk heat map in each carriage is marked.

[0034] In this invention, the intelligent decision optimization layer constructs a full-scenario operation state space based on the compliance score report and the real-time status of the train, according to the preset execution time threshold and service coverage combined with the safety operation time and crew working hours; In this embodiment, the full-scene state space includes: cabin environment state (sensor data) and crew state (busyness monitoring); In the full-scenario operation state space, event simulation is performed based on the real-time risk heat map of all carriages and the dispatch task type, generating dynamic decision-making plans and resource conflict warnings; In this embodiment, task types (equipment inspection, passenger service, emergency response) are assigned; resource conflict warnings are provided, such as when carriage 2 needs to handle both equipment failure and passenger first aid. Based on the event type, combined with the crew portrait library and real-time train positioning data, a critical path algorithm is used to optimize dynamic decision-making plans and generate crew operation response plans. In this embodiment, a dynamic decision-making plan (task list, execution priority); a crew portrait library (skill level, historical operation accuracy, response speed); based on the real-time positioning data of the train, the contact information of the hospital at the nearest stop is dynamically inserted; event type (equipment failure / passenger first aid / weather emergency); based on the real-time positioning data of the train, the contact information of the hospital at the nearest stop is dynamically inserted.

[0035] Use the critical path algorithm to calculate the shortest path. For example, "simultaneously executing equipment reset (15 minutes) and passenger comfort (10 minutes)" saves 5 minutes compared to "serial execution"; Trigger contextual voice prompts ("Open the first aid kit 2 meters to the left front") based on the flight attendant's location (UWB positioning).

[0036] Example: The multi-mode data acquisition layer uses monitoring devices to record video streams of train crew operations, passenger status, and equipment dashboards to obtain video data. It also uses a microphone array to collect audio streams of train crew announcements, passenger conversations, and emergency calls to obtain voice data. It also uses electronic logs to record text files such as train crew schedules, event logs, fault repair orders, and service texts to obtain text data. It also uses a sensor network (temperature and humidity composite sensors, gas sensors, door switch sensors, and pressure sensors) to collect time-series data streams of temperature, humidity, oxygen content, door status, and seat pressure within the train to obtain environmental data. The multi-mode data acquisition layer correlates and fuses video data, voice data, text data, and environmental data based on the crew operation scenario to obtain multi-modal operation data for the crew. The multi-mode data processing layer uses a decomposition and conversion algorithm to slice and convert the video data to obtain several frames of images; and uses an image filtering algorithm to denoise and deblur each frame of the image to obtain a noise-free image; at the same time, a facial recognition algorithm is used to identify and mark the faces in the noise-free image, and then a random projection hash algorithm is used to desensitize the marked faces to obtain a non-sensitive image; The multi-mode data processing layer uses a wavelet domain noise reduction algorithm to reduce the noise of the voice data to obtain a noise-free frequency spectrum. It also uses a voiceprint segmentation and clustering algorithm based on the crew voice library to distinguish between crew members and passengers in the noise-free frequency spectrum, obtaining a crew audio set and a passenger audio set. The multi-mode data processing layer uses a sensitive word extraction algorithm to extract keywords from the clean text content based on the sensitive word list, obtaining key information about the train event: crew recording time, crew member ID, event time, and the carriage where the event occurred. The multi-mode data processing layer uses a data classification algorithm to classify environmental data according to sensor type to obtain an initial sensor data table. It then uses a filtering detection algorithm to filter the initial sensor data table to obtain a uniform sensor data table. It then uses a time series alignment algorithm to correlate and fuse the uniform sensor data tables based on timestamps and carriage numbers to obtain a full-state environmental data table. The multimodal data processing layer uses the data annotation network to annotate the non-sensitive images, crew audio sets, passenger audio sets and full-state environment data tables according to the key information of train events, and obtains a data label sequence table.

[0037] The visual parsing branch of the multimodal fusion reasoning model combines a spatiotemporal dual-frame network to extract the scene state of each frame of the non-sensitive image and capture the crew equipment operation motion to obtain the global matrix of the car; The facial analysis branch uses a micro-expression encoding network combined with a posture estimation network to quantify the flight attendant's micro-expressions and identify micro-motion deviations in each frame of the non-sensitive image to obtain a flight attendant local matrix. The semantic decoupling branch uses a speech representation network to separate the speech content and paralinguistic features of the crew audio set to generate semantic vectors and emotion vectors. The voiceprint mapping branch uses a voiceprint recognition network to identify and cluster the passenger audio set and perform voice mapping to construct a passenger-voice mapping matrix; The text extraction branch uses a pattern knowledge extraction network combined with a time series understanding network to convert key train event information into event tuples and obtain an event time series matrix. The signal fusion branch uses a multi-scale convolutional network combined with timestamps to synchronize the global environment data table to obtain a heterogeneous data synchronization matrix; The abnormality judgment branch uses a variational autoencoder to learn the full-state environmental data distribution of the train's normal operating conditions and calculate the reconstruction error matrix; The hierarchical association branch uses a spatiotemporal cross attention network to calculate the association weights of the car global matrix and the crew local matrix combined with the heterogeneous data synchronization matrix according to the timestamp to obtain a dynamic spatiotemporal weight matrix. The semantic alignment branch uses a self-supervised contrastive learning network to verify the phenomena of the car global matrix and the crew local matrix combined with the semantic vector according to the car number, and construct a semantic-physical alignment space; The graph construction branch uses an information mapping algorithm to define nodes and edges by combining the car global matrix, crew local matrix, and event time series matrix with the reconstruction error matrix according to the crew number, car number, and passenger identification, and then constructs a full-scene crew operation graph.

[0038] The causal reasoning branch of the multi-modal fusion analysis layer uses a linear non-Gaussian acyclic network to perform causal mining on the full-scenario crew operation map combined with the reconstructed error matrix to obtain the event causal reasoning chain; The simulation back-calculation branch uses a digital twin algorithm to rehearse crew operations on the event time series matrix based on the semantic-physical alignment space and the dynamic spatiotemporal weight matrix to predict passenger satisfaction levels. The crew evaluation branch quantifies the operating process of each crew member in the crew local matrix based on the event causal reasoning chain and the event time series matrix, combined with passenger satisfaction, to determine the crew operation level.

[0039] The intelligent decision-making optimization layer extracts entities from the train attendant operating standard manual, constructs atomic rules, and converts fuzzy descriptions into time-quantified instructions; at the same time, it converts railway safety regulations into logical execution rules; maps the multimodal data in historical excellent attendant operation cases with atomic rules and logical execution rules to construct a case standard mapping table; traverses all graph nodes in the full-scene attendant operation map, and matches the attendant operation level with the case standard mapping table to obtain the attendant standard operation map; at the same time, the full-scene attendant operation map is combined with the passenger satisfaction level to detect violations and obtain an operation violation subgraph; the execution time threshold in the atomic rules and logical execution rules is corrected according to the train speed to obtain a dynamic time threshold; based on the attendant standard operation map and the operation violation subgraph combined with the dynamic time threshold, a compliance score report is generated, and combined with the heterogeneous data synchronization matrix, a real-time risk heat map in each carriage is annotated.

[0040] The intelligent decision-making optimization layer constructs a full-scenario operation state space based on compliance score reports and real-time train status, according to preset execution time thresholds and service coverage, combined with safety-related operation time and crew working hours; In the full-scenario operation state space, event simulation is performed based on the real-time risk heat map of all carriages and the dispatch task type, generating dynamic decision-making plans and resource conflict warnings; Based on the event type, combined with the crew portrait library and real-time train positioning data, the critical path algorithm is used to optimize the dynamic decision-making plan and generate a crew operation disposal plan.

Claims

1. A standardized analysis system for crew operation data based on a multimodal model, characterized by: It includes multi-mode data acquisition layer, multi-mode data processing layer, multi-mode fusion analysis layer and intelligent decision optimization layer; among them, The multimodal data collection layer is used to obtain multimodal operational data of train crew members, including video data, voice data, text data, and environmental data. The multimodal data processing layer is used to clean and anomaly-annotate multimodal operation data to obtain a data label sequence table; The multimodal fusion analysis layer uses a multimodal fusion inference model to jointly analyze the cleaned multimodal operation data based on the data label sequence table, obtain a full-scene crew operation map, and determine the crew operation rating; The intelligent decision-making optimization layer is used to globally verify the full-scenario crew operation map based on the train crew operation standard manual and generate crew operation disposal plans.

2. The multimodal model-based standardized analysis system for crew operation data according to claim 1 is characterized by: The multi-mode data acquisition layer records the video stream in the railway train to obtain video data; at the same time, it collects the audio stream in the railway train to obtain voice data; at the same time, it records the text files uploaded by the crew in the railway train to obtain text data; at the same time, it collects the time-series data stream of the environment of each carriage in the railway train to obtain environmental data; the multi-mode data acquisition layer associates and fuses the video data, voice data, text data and environmental data according to the crew operation scenario to obtain the crew's multi-modal operation data.

3. The multimodal model-based standardized analysis system for crew operation data according to claim 1 is characterized by: The multi-mode data processing layer slices and converts the video data to obtain several frames of images; and performs denoising and deblurring on each frame of image to obtain a noise-free image; at the same time, the face in the noise-free image is identified and marked, and then the marked face is desensitized to obtain a non-sensitive image; The multi-mode data processing layer reduces the noise of the voice data to obtain a noise-free frequency. And based on the crew voice library, the crew and passengers in the noise-free frequency are distinguished to obtain the crew audio set and the passenger audio set; The multi-mode data processing layer extracts keywords from the clean text content based on the sensitive word list to obtain key information about the train event; The multi-mode data processing layer classifies environmental data according to sensor type and obtains the initial sensor data table; And filter the initial sensor data table to obtain the same sensor data table; And according to the timestamp and carriage number, the sensor data tables with no difference are associated and fused to obtain the full-state environment data table; The multi-mode data processing layer annotates the non-sensitive images, crew audio sets, passenger audio sets and full-state environment data tables with events according to the key information of train events to obtain a data label sequence table.

4. The multimodal model-based standardized analysis system for crew operation data according to claim 1 is characterized by: The operation process of the multimodal fusion reasoning model includes the following steps: The visual parsing branch combines a spatiotemporal dual-frame network to extract the scene state of each frame of the non-sensitive image and capture the crew equipment operation motion to obtain the global matrix of the car; The facial analysis branch uses a micro-expression encoding network combined with a posture estimation network to quantify the flight attendant's micro-expressions and identify micro-motion deviations in each frame of the non-sensitive image to obtain a flight attendant local matrix. The semantic decoupling branch uses a speech representation network to separate the speech content and paralinguistic features of the crew audio set to generate semantic vectors and emotion vectors. The voiceprint mapping branch uses a voiceprint recognition network to identify and cluster the passenger audio set and perform voice mapping to construct a passenger-voice mapping matrix; The text extraction branch uses a pattern knowledge extraction network combined with a time series understanding network to convert key train event information into event tuples and obtain an event time series matrix. The signal fusion branch uses a multi-scale convolutional network combined with timestamps to synchronize the global environment data table to obtain a heterogeneous data synchronization matrix.

5. The standardized analysis system for crew operation data based on a multimodal model according to claim 4 is characterized by: The operation process of the multimodal fusion reasoning model also includes the following steps: The abnormality judgment branch uses a variational autoencoder to learn the full-state environmental data distribution of the train's normal operating conditions and calculate the reconstruction error matrix; The hierarchical association branch uses a spatiotemporal cross attention network to calculate the association weights of the car global matrix and the crew local matrix combined with the heterogeneous data synchronization matrix according to the timestamp to obtain a dynamic spatiotemporal weight matrix. The semantic alignment branch uses a self-supervised contrastive learning network to verify the phenomena of the car global matrix and the crew local matrix combined with the semantic vector according to the car number, and construct a semantic-physical alignment space; The graph construction branch uses an information mapping algorithm to define nodes and edges by combining the car global matrix, crew local matrix, and event time series matrix with the reconstruction error matrix according to the crew number, car number, and passenger identification, and then constructs a full-scene crew operation graph.

6. The standardized analysis system for crew operation data based on a multimodal model according to claim 1 is characterized by: The causal reasoning branch of the multi-modal fusion analysis layer uses a linear non-Gaussian acyclic network to perform causal mining on the full-scene crew operation map combined with the reconstruction error matrix to obtain the event causal reasoning chain; The simulation back-calculation branch uses a digital twin algorithm to rehearse crew operations on the event time series matrix based on the semantic-physical alignment space and the dynamic spatiotemporal weight matrix to predict passenger satisfaction levels. The crew evaluation branch quantifies the operating process of each crew member in the crew local matrix based on the event causal reasoning chain and the event time series matrix, combined with passenger satisfaction, to determine the crew operation level.

7. The multimodal model-based standardized analysis system for crew operation data according to claim 1 is characterized by: The intelligent decision optimization layer extracts entities from the train crew operating standard manual, constructs atomic rules, and converts fuzzy descriptions into time-quantified instructions; at the same time, it converts railway safety regulations into logical execution rules; Map the multimodal data from historical excellent flight attendant operation cases with atomization rules and logical execution rules to build a case standard mapping table; Traverse all graph nodes in the full-scenario crew operation graph, and match the crew operation level with the case standard mapping table to obtain the crew standard operation graph; At the same time, the full-scene crew operation graph is combined with the passenger satisfaction level to detect violations and obtain the operation violation subgraph; The execution time thresholds in the atomization rules and logic execution rules are modified according to the train speed to obtain a dynamic time threshold; Based on the standard crew operation diagram and operation violation sub-diagram combined with dynamic time thresholds, a compliance score report is generated, and combined with the heterogeneous data synchronization matrix, a real-time risk heat map in each carriage is marked.

8. The standardized analysis system for crew operation data based on a multimodal model according to claim 1 is characterized by: The intelligent decision optimization layer constructs a full-scenario operation state space based on the compliance score report and the real-time status of the train, according to the preset execution time threshold and service coverage combined with the safety operation time and crew working hours; In the full-scenario operation state space, event simulation is performed based on the real-time risk heat map of all carriages and the dispatch task type, generating dynamic decision-making plans and resource conflict warnings; Based on the event type, combined with the crew portrait library and real-time train positioning data, the critical path algorithm is used to optimize the dynamic decision-making plan and generate a crew operation disposal plan.

Images