Hydropower station emergency command intelligent question and answer decision-making method
By constructing a smart question-and-answer command system for hydropower stations based on a large language model fine-tuned by LoRA and multimodal data fusion technology, the problems of slow response and decision-making errors in traditional hydropower station emergency management systems have been solved, and intelligent, automated and precise emergency command and decision-making have been achieved.
Patent Information
- Application Number
- CN202511116298.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-11
- Publication Date
- 2025-11-21
AI Technical Summary
Traditional hydropower station emergency management systems are slow to respond and information is delayed. They cannot effectively integrate multimodal data and historical experience, leading to decision-making errors and untimely responses.
By employing a large language model based on LoRA fine-tuning and multimodal data fusion technology, an intelligent question-and-answer command model for hydropower stations is constructed. Through real-time collection and analysis of multimodal data, emergency command suggestions are output.
It has enabled intelligent, automated, and precise emergency command for hydropower stations, improving the speed of emergency response and the accuracy of decision-making, and enhancing digital management capabilities.
Smart Images
Figure CN120994787A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of emergency management technology for hydropower stations, and in particular to an intelligent question-and-answer decision-making method for emergency command in hydropower stations. Background Technology
[0002] With the continuous advancement of technology, especially the rapid development of cutting-edge artificial intelligence (AI) technology, traditional emergency management and decision-making systems for hydropower stations are facing enormous challenges. During operation, hydropower stations must cope with various emergencies such as earthquakes, floods, and equipment failures. However, traditional manual decision-making and emergency response methods often suffer from slow response, information lag, and decision-making errors, making it difficult to cope with complex and ever-changing emergency environments.
[0003] Emergency command at hydropower stations is highly complex, especially when multiple natural disasters or equipment failures occur simultaneously. Traditional emergency management systems rely on human judgment and experience, which is not only slow but also easily limited by the capabilities and experience of the command personnel, failing to provide a comprehensive and timely emergency response. Therefore, there is an urgent need for a more intelligent and automated emergency command system to respond quickly to emergencies.
[0004] Secondly, hydropower stations typically generate a large amount of real-time data, including equipment monitoring, sensor data, weather forecasts, and water level and flow information. This data comes from different monitoring systems and sensors, exhibiting multimodal characteristics. Traditional systems often can only process single data sources and cannot fully integrate multimodal data for in-depth analysis. This results in emergency command systems lacking comprehensive evaluation of multifaceted information, thus affecting the accuracy and timeliness of decision-making.
[0005] Furthermore, the role of historical knowledge and case studies in emergency management cannot be ignored. Summarizing and analyzing past incidents can provide valuable experience for future emergency responses. However, traditional emergency systems often fail to effectively integrate historical data and operational experience into the decision-making process, and also fail to update their knowledge bases in a timely manner. This can result in a lack of flexibility in the decision-making process, hindering rapid adaptation to new technologies and emergency needs. Intelligent emergency systems based on AI and Natural Language Processing (NLP) technologies can extract useful information from historical data, providing commanders with more timely and intelligent decision-making suggestions. Summary of the Invention
[0006] This application provides an intelligent question-and-answer decision-making method for emergency command in hydropower stations. It utilizes a large language model and multimodal data fusion technology to achieve intelligent, automated, and precise emergency command decision-making in hydropower stations.
[0007] To address the aforementioned technical issues, this application provides an intelligent question-and-answer decision-making method for emergency command in hydropower stations, comprising a model training phase and a model application phase. The model training phase involves constructing an emergency command model based on LoRA fine-tuning and training the model to obtain an intelligent question-and-answer command model for the hydropower station. The model application phase involves real-time collection and intelligent analysis of multimodal data from the hydropower station based on the intelligent question-and-answer command model, outputting corresponding emergency command suggestions to enhance the digital management capabilities of the hydropower station.
[0008] In some exemplary embodiments, the model training phase includes the following steps: acquiring a dataset; constructing a sample set based on the dataset; fine-tuning a large language model using LoRA technology to construct an emergency command model based on LoRA fine-tuning; and training the emergency command model based on LoRA fine-tuning to obtain a hydropower station intelligent question-and-answer command model.
[0009] In some exemplary embodiments, the dataset includes the hydropower station's operation manual, operation guidelines, emergency response plan, historical accidents and solutions, as well as the hydropower station's laws, regulations and safety standards.
[0010] In some exemplary embodiments, the Trainer class is used to train an emergency command model based on LoRA fine-tuning, and training parameters are set. After training is completed, a smart question-and-answer command model for hydropower stations is generated.
[0011] In some exemplary embodiments, the model application stage includes the following steps: real-time acquisition of multimodal data from the hydropower station based on the data acquisition module; the multimodal data includes video data, sensor data, audio data, and environmental data; feature extraction is performed on the data of each modality, and the extracted features are fused to obtain fused features; pose analysis is performed on the fused features to output prediction results with pose assessment level information; object recognition, scene understanding, and personnel activity recognition are performed on each frame of video to generate image description information; the image description information and the prediction results with pose assessment level information are sent to the hydropower station's intelligent question-and-answer command model to output corresponding emergency command suggestions.
[0012] In some exemplary embodiments, multimodal data of a hydropower station is collected in real time based on a data acquisition module, including: collecting multimodal data of the hydropower station by installing various sensor devices inside the hydropower station.
[0013] In some exemplary embodiments, the video data in the multimodal data of the hydropower station includes park monitoring video, surveillance video, and recorder video; sensor data includes personnel information, including personnel positioning data, personnel vital sign data, and personnel trajectory data; audio data includes equipment operation information, including the equipment's acoustic signature signal and vibration signal; environmental data includes environmental monitoring information and operational safety information, including water level data, meteorological data, and landslide monitoring data; and operational safety information includes work permit data, major hazard source monitoring data, and contractor operation status.
[0014] In some exemplary embodiments, feature extraction is performed on the data of each modality using a convolutional neural network or a recurrent neural network; wherein, video data is used to extract spatial-temporal features through a 3D-CNN; sensor data is used to extract temporal information through an LSTM; audio data is used to extract audio features through a combination of CNN-RNN; and environmental data is used to extract environmental features through a fully connected layer.
[0015] In some exemplary embodiments, pose analysis is performed on the fused features through multiple fully connected layers to output prediction results with pose assessment level information; the pose assessment level information includes: level information of personnel risk, environmental risk and equipment risk.
[0016] In some exemplary embodiments, a large video model is used to perform object recognition, scene understanding, and human activity recognition on each frame of video to generate image description information; the large video model includes a deep learning model based on a convolutional neural network or Transformer architecture.
[0017] The technical solution provided in this application has at least the following advantages:
[0018] This application provides an intelligent question-and-answer decision-making method for emergency command in hydropower stations, including a model training stage and a model application stage. The model training stage is used to construct an emergency command model based on LoRA fine-tuning and train the model to obtain an intelligent question-and-answer command model for hydropower stations. The model application stage is used to collect and intelligently analyze multimodal data of hydropower stations in real time based on the intelligent question-and-answer command model for hydropower stations, and output corresponding emergency command suggestions to enhance the digital management capabilities of hydropower stations.
[0019] This application aims to improve the efficiency and intelligence of hydropower station emergency management and decision support systems by introducing advanced technologies such as artificial intelligence, big data, the Internet of Things, and intelligent question-answering large language models. Through multimodal data fusion, real-time monitoring, and deep learning analysis, the system can quickly identify potential risks and provide accurate decision support in the face of complex emergencies. The intelligent question-answering large language system can generate reasonable emergency plans based on historical cases and real-time data, automatically providing solutions for emergency command personnel, thereby improving the safety and emergency response capabilities of hydropower stations. Attached Figure Description
[0020] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments, and unless otherwise stated, the figures in the drawings are not to be limited by scale.
[0021] Figure 1 This is a flowchart illustrating an intelligent question-and-answer decision-making method for emergency command of a hydropower station, provided as an embodiment of this application. Detailed Implementation
[0022] As can be seen from the background technology, traditional emergency systems suffer from technical problems such as the inability to effectively integrate historical data and operational experience into the decision-making process, and the failure to update the knowledge base in a timely manner.
[0023] Related technologies propose a decision-making method for emergency command at hydropower stations. This method inputs several sets of emergency data influencing factors into a prediction model and uses coupled predictions of emergency event risk levels to determine auxiliary decision-making for hydropower station emergency command. However, this method does not fully consider comprehensive data analysis, including video information, personnel information, equipment operation information, environmental monitoring information, and operational safety information. Furthermore, it does not consider learning from prior knowledge such as hydropower station-specific operation manuals and guidelines, emergency response plans, and relevant laws and standards, thus failing to achieve deep intelligent decision-making for hydropower station emergencies.
[0024] Another related technology proposes a Bayesian network-based intelligent analysis system and method for deep-sea emergency situations. This system includes an emergency data acquisition module, an emergency data processing module, an emergency situation analysis module, and an emergency situation display module. However, this system and method only quantifies the potential risks in deep-sea emergency processes and improves the ability to discover, identify, and analyze emergency operation risks from a global perspective, but it has not been applied to hydropower station emergencies.
[0025] In addition, another related technology proposes a language-based large-scale model question-answering system and method for the political and legal field. While this method considers learning from relevant political and legal case text data, including cases involving legal interpretations, judicial judgments, and court rulings, it does not provide corresponding decision-making application outputs. In summary, existing emergency decision-making methods have many shortcomings, potentially leading to a lack of flexibility in the decision-making process and an inability to quickly adapt to new technologies and emergency needs.
[0026] Intelligent emergency response systems based on AI and Natural Language Processing (NLP) technologies can extract useful information from historical data, providing commanders with more timely and intelligent decision-making suggestions. To address the aforementioned technical issues, this application provides an intelligent question-and-answer decision-making method for hydropower station emergency command. This method includes a model training phase and a model application phase. The model training phase involves constructing and training an emergency command model based on LoRA fine-tuning to obtain an intelligent question-and-answer command model for the hydropower station. The model application phase involves real-time collection and intelligent analysis of multimodal data from the hydropower station based on the intelligent question-and-answer command model, outputting corresponding emergency command suggestions to enhance the digital management capabilities of the hydropower station. This application provides an intelligent question-and-answer decision-making method for hydropower station emergency command, utilizing large-scale language models and multimodal data fusion technology to achieve intelligent, automated, and precise emergency command decision-making for hydropower stations.
[0027] The embodiments of this application will now be described in detail with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the embodiments of this application to facilitate a better understanding of the application. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.
[0028] See Figure 1 This application provides an intelligent question-and-answer decision-making method for emergency command of hydropower stations, including a model training stage and a model application stage. The model training stage is used to construct an emergency command model based on LoRA fine-tuning and train the model to obtain an intelligent question-and-answer command model for hydropower stations. The model application stage is used to collect and intelligently analyze multimodal data of hydropower stations in real time based on the intelligent question-and-answer command model of hydropower stations, and output corresponding emergency command suggestions to enhance the digital management capabilities of hydropower stations.
[0029] In some embodiments, the model training phase includes the following steps:
[0030] Step 1: Obtain the dataset.
[0031] Step 2: Construct a sample set based on the dataset.
[0032] Step 3: Use LoRA technology to fine-tune the large language model and build an emergency command model based on LoRA fine-tuning.
[0033] Step 4: Train the emergency command model based on LoRA fine-tuning to obtain the intelligent question-and-answer command model for hydropower stations.
[0034] In step one, a dataset is obtained by collecting data. In some embodiments, the dataset includes the hydropower station's operation manual, operation guidelines, emergency response plan, historical accidents and solutions, as well as the hydropower station's laws, regulations, and safety standards.
[0035] Specifically, operation manuals and guidelines include documents such as hydropower station operation manuals, equipment maintenance manuals, and system operation manuals. Emergency response plans include various fault handling procedures, emergency drill plans, accident reporting, and handling measures. Historical accidents and solutions can be obtained by collecting past accident cases of hydropower stations, their causes, solutions, and lessons learned. Relevant laws and standards include laws and regulations governing hydropower stations, national and local safety standards, etc.
[0036] In step two, a sample set is constructed by extracting a large number of frequently asked questions and answer pairs from the collected hydropower station data. The format of the sample set is as follows:
[0037] {"question":"What to do when the water pump malfunctions?","answer":"First check the power supply. After confirming the power supply is normal, check if the pump body is blocked by debris. If there is no problem, then check for faults in the control system..."}
[0038] In step three, the Trainer class is used to train the LoRA-based emergency command model and the training parameters are set. After training is completed, the intelligent question-and-answer command model for the hydropower station is generated.
[0039] The fine-tuning process of the LoRA-based GPT model is as follows:
[0040] First, select and load a basic large language model (such as GPT, Lamma), and use LoRA technology to fine-tune the model to reduce the resources required for training.
[0041] Then, configure the LoRA fine-tuning parameters as follows:
[0042] lora_config = LoraConfig(
[0043] r=8, # Rank of Low-rank approximation
[0044] lora_alpha = 16, # Scaling factor for training LoRA parameters
[0045] lora_dropout = 0.1, #dropout of LoRA layer
[0046] bias="none", # Bias setting for LoRA layer )
[0048] Construct an emergency command model based on LoRA fine-tuning.
[0049] model=get_peft_model(model, lora_config)
[0050] Use the Trainer class to train the model, specify the training parameters, and generate a smart question-and-answer command model for hydropower stations after training.
[0051] model.trainer.train()
[0052] In some embodiments, the model application phase includes the following steps:
[0053] Step 1: Based on the data acquisition module, real-time acquisition of multimodal data of the hydropower station is carried out; multimodal data includes video data, sensor data, audio data and environmental data.
[0054] Step 2: Extract features from the data of each modality, and fuse the extracted features to obtain the fused features.
[0055] Step 3: Perform pose analysis on the fused features and output prediction results with pose assessment level information.
[0056] Step 4: Perform object recognition, scene understanding, and personnel activity recognition on each frame of video to generate image description information.
[0057] Step 5: Send the image description information and the prediction results with attitude assessment level information to the hydropower station's intelligent question-and-answer command model, and output the corresponding emergency command suggestions.
[0058] In some embodiments, in step one of the model application phase, multimodal data of the hydropower station is collected in real time based on the data acquisition module, including: collecting multimodal data of the hydropower station by installing various sensor devices inside the hydropower station.
[0059] In some embodiments, the video data in the multimodal data of the hydropower station includes park monitoring video, surveillance video, and recorder video; sensor data includes personnel information, including personnel positioning data, personnel vital sign data, and personnel trajectory data; audio data includes equipment operation information, including the equipment's acoustic signature signal and vibration signal; environmental data includes environmental monitoring information and operational safety information, including water level data, meteorological data, and landslide monitoring data; and operational safety information includes work permit data, major hazard source monitoring data, and contractor operation status.
[0060] In some embodiments, in step two of the model application stage, feature extraction is performed on the data of each modality using a convolutional neural network or a recurrent neural network; wherein, video data is extracted for spatial-temporal features using a 3D-CNN; sensor data is extracted for temporal information using an LSTM; audio data is extracted for audio features using a combination of CNN-RNN; and environmental data is extracted for environmental features using a fully connected layer.
[0061] In some embodiments, in step three of the model application phase, pose analysis is performed on the fused features through multiple fully connected layers to output prediction results with pose assessment level information; the pose assessment level information includes: level information of personnel risk, environmental risk and equipment risk.
[0062] In some embodiments, in step four of the model application phase, a large video model is used to perform object recognition, scene understanding, and personnel activity recognition on each frame of video to generate image description information; the large video model includes a deep learning model based on a convolutional neural network or Transformer architecture.
[0063] The following detailed explanation of the model application phase is provided through specific examples.
[0064] 1. Data Collection:
[0065] Data acquisition module: By installing various sensor devices inside the hydropower station, the following data is collected:
[0066] Video data includes park surveillance video, PTZ camera video, and recorder video. The video data undergoes preliminary analysis using image processing algorithms (such as target detection and motion recognition).
[0067] Personnel information (personnel_data): This includes personnel location data, personnel vital signs data (such as heart rate, body temperature, etc.), and personnel trajectory data. Personnel location data is obtained through indoor and outdoor positioning systems, and vital signs data is collected through smart wearable devices.
[0068] Equipment operation information (sensor_data): This includes the equipment's acoustic signature and vibration signals. The equipment's acoustic signature is extracted using acoustic signature recognition technology to detect abnormal behavior of the equipment (such as malfunctions, overloads, etc.).
[0069] Environmental monitoring information (environmental_data): This includes water level data, meteorological data, landslide monitoring data, etc. Environmental data is collected by sensors and transmitted in real time.
[0070] Work safety information (task_data): includes work order data, monitoring data of major hazard sources, contractor work status, etc.
[0071] Data output: data_collection = [video_data,personnel_data,sensor_data,environmental_data,task_data].
[0072] 2. Feature fusion:
[0073] Feature extraction module: Features of each modality's data are extracted using an independent convolutional neural network (CNN) or recurrent neural network (RNN).
[0074] Video: Extracting spatial-temporal features using 3D-CNN.
[0075] Sensors: Timing information is extracted using LSTM.
[0076] Audio: Audio features are extracted using a combination of CNN and RNN.
[0077] Environmental data: Environmental features are extracted using a simple fully connected layer.
[0078] Feature output: feature_collection=[video_feature,personnel_feature,sensor_feature,environmental_feature,task_feature].
[0079] 3. Situation Analysis:
[0080] Deep fully connected layers: Multiple fully connected layers are used to further process the fused features for classification or regression analysis.
[0081] Output layer: Outputs the prediction results
[0082] Result type: Situation assessment level, including: people risk level, environmental risk level, and equipment risk level.
[0083] 4. Emergency Command Question and Answer Model:
[0084] Use a large video model (such as a deep learning model based on convolutional neural networks or Transformer architecture) to perform object recognition, scene understanding, and human activity recognition on each frame of video, and generate descriptive information: video_descriptions = generate_video_description(video_data).
[0085] Send the image description information, along with information on personnel risk, environmental risk, and equipment risk levels, to the question-and-answer command model, and have it output corresponding emergency command suggestions.
[0086] Large model prompt: prompt = [video_descriptions,people_risk_level,environmental_risk_level,equipment_risk_level].
[0087] Input prompt words: inputs = tokenizer.encode(prompt).
[0088] Output command suggestion: outputs = model.generate(inputs).
[0089] The generated emergency command recommendations are sent to the command center or relevant personnel to support real-time decision-making.
[0090] 5. Process Management:
[0091] Repeat steps 1-4 above until the emergency ends.
[0092] Compared with existing technologies, the intelligent question-and-answer decision-making method for emergency command of hydropower stations provided in this application has the following advantages:
[0093] This application deeply integrates visual large model technology with the HAZOP safety risk analysis method, providing an intelligent and high-precision method for the quantitative identification of behavioral risks in emergency and daily operations of large hydropower stations, which significantly improves the efficiency and reliability of operational safety management.
[0094] (1) With the help of environmental data collected by cameras, the visual big model can accurately identify the actions, postures and dangerous situations of workers and the surrounding environment. Through multimodal feature extraction and fusion algorithms, unstructured image information is transformed into structured risk descriptions.
[0095] (2) Combining the HAZOP analysis framework, the system can dynamically assess whether there are safety risks in the current behavior of the workers and quantify and classify the degree of risk, realizing the transformation from traditional passive protection to proactive intelligent early warning.
[0096] (3) Through a multi-level risk perception model that integrates spatiotemporal features and a real-time deviation reasoning engine based on knowledge graphs, this application can not only capture local details of the work site, but also understand long-distance semantic dependencies, thereby achieving comprehensive and accurate risk identification in the complex and ever-changing hydropower station operation environment.
[0097] (4) The adaptive weight adjustment mechanism and edge computing architecture ensure the system's adaptability and real-time performance under different operating scenarios, significantly reducing the false alarm rate and missed alarm rate. Operators and managers can quickly locate risk points and take targeted measures to effectively prevent safety accidents and provide strong technical support for the safe production of hydropower stations.
[0098] Based on the above technical solutions, this application provides an intelligent question-and-answer decision-making method for emergency command of hydropower stations, including a model training stage and a model application stage. The model training stage is used to construct an emergency command model based on LoRA fine-tuning and train the model to obtain an intelligent question-and-answer command model for hydropower stations. The model application stage is used to collect and intelligently analyze multimodal data of hydropower stations in real time according to the intelligent question-and-answer command model of hydropower stations, and output corresponding emergency command suggestions to enhance the digital management capabilities of hydropower stations.
[0099] This application aims to improve the efficiency and intelligence of hydropower station emergency management and decision support systems by introducing advanced technologies such as artificial intelligence, big data, the Internet of Things, and intelligent question-answering large language models. Through multimodal data fusion, real-time monitoring, and deep learning analysis, the system can quickly identify potential risks and provide accurate decision support in the face of complex emergencies. The intelligent question-answering large language system can generate reasonable emergency plans based on historical cases and real-time data, automatically providing solutions for emergency command personnel, thereby improving the safety and emergency response capabilities of hydropower stations.
[0100] Those skilled in the art will understand that the above-described embodiments are specific examples of implementing this application, and in practical applications, various changes in form and detail may be made without departing from the spirit and scope of this application. Any person skilled in the art can make their own modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application should be determined by the scope defined in the claims.
Claims
1. A smart question-and-answer decision-making method for emergency command in hydropower stations, characterized in that, It includes the model training phase and the model application phase; among which, The model training phase is used to construct an emergency command model based on LoRA fine-tuning and to train the model to obtain a hydropower station intelligent question-and-answer command model. The model application phase is used to collect and intelligently analyze multimodal data of the hydropower station in real time based on the intelligent question-and-answer command model of the hydropower station, and output corresponding emergency command suggestions to enhance the digital management capabilities of the hydropower station.
2. The intelligent question-and-answer decision-making method for emergency command of hydropower stations according to claim 1, characterized in that, The model training phase includes the following steps: Obtain the dataset; Based on the dataset, construct a sample set; LoRA technology was used to fine-tune the large language model, and an emergency command model based on LoRA fine-tuning was constructed. By training an emergency command model based on LoRA fine-tuning, an intelligent question-and-answer command model for hydropower stations is obtained.
3. The intelligent question-and-answer decision-making method for emergency command of hydropower stations according to claim 2, characterized in that, The dataset includes hydropower station operation manuals, operation guidelines, emergency response plans, historical accidents and solutions, as well as laws, regulations and safety standards for hydropower stations.
4. The intelligent question-and-answer decision-making method for emergency command of hydropower stations according to claim 2, characterized in that, The Trainer class is used to train an emergency command model based on LoRA fine-tuning, and training parameters are set. After training, a smart question-and-answer command model for hydropower stations is generated.
5. The intelligent question-and-answer decision-making method for emergency command of hydropower stations according to claim 1, characterized in that, The model application phase includes the following steps: The data acquisition module is used to collect multimodal data from the hydropower station in real time; the multimodal data includes video data, sensor data, audio data, and environmental data. Feature extraction is performed on the data of each modality, and the extracted features are then fused to obtain the fused features. Perform pose analysis on the fused features and output prediction results with pose evaluation level information; Perform object recognition, scene understanding, and personnel activity recognition on each frame of video to generate image description information; The image description information and the prediction results with attitude assessment level information are sent to the hydropower station's intelligent question-and-answer command model, which then outputs corresponding emergency command suggestions.
6. The intelligent question-and-answer decision-making method for emergency command of hydropower stations according to claim 5, characterized in that, Based on the data acquisition module, multimodal data of the hydropower station are collected in real time, including: By installing various sensor devices inside the hydropower station, multimodal data of the hydropower station can be collected.
7. The intelligent question-and-answer decision-making method for emergency command of hydropower stations according to claim 6, characterized in that, The video data in the multimodal data of the hydropower station includes park surveillance video, surveillance camera video, and recorder video; The sensor data includes personnel information; the personnel information includes personnel location data, personnel vital sign data, and personnel trajectory data. The audio data includes device operation information, which includes the device's acoustic signature and vibration signal. The environmental data includes environmental monitoring information and operational safety information; the environmental monitoring information includes water level data, meteorological data, and landslide monitoring data; the operational safety information includes work permit data, major hazard source monitoring data, and contractor operational status.
8. The intelligent question-and-answer decision-making method for emergency command of hydropower stations according to claim 5, characterized in that, For each modality of data, feature extraction is performed using a convolutional neural network or a recurrent neural network; wherein... The video data is used to extract spatial-temporal features through 3D-CNN; The sensor data is used to extract timing information through LSTM; The audio data is used to extract audio features through a combination of CNN-RNN; The environmental data is processed through a fully connected layer to extract environmental features.
9. The intelligent question-and-answer decision-making method for emergency command of hydropower stations according to claim 5, characterized in that, The fused features are subjected to pose analysis through multiple fully connected layers, and the prediction results with pose evaluation level information are output. The attitude assessment level information includes the level information of personnel risk, environmental risk, and equipment risk.
10. The intelligent question-and-answer decision-making method for emergency command of hydropower stations according to claim 5, characterized in that, A large video model is used to perform object recognition, scene understanding, and human activity recognition on each frame of video, generating image description information; The large video model includes deep learning models based on convolutional neural networks or Transformer architectures.