Intelligent highway weather and disaster detection system based on ResNet-ViT collaborative reasoning

Abstract

The invention discloses an intelligent highway weather and disaster detection system based on ResNet-ViT collaborative reasoning, and belongs to the technical field of computer vision and intelligent traffic cross.The system construction steps are that a sequence with weather labels is constructed based on multi-source data parallel collection operation, disaster labels are expanded for the sequence, and a weather-disaster sequence is obtained; the weather-disaster sequence is input into an end side platform through the highway line end side platform, and a preliminary detection result is obtained; inputting the preliminary detection result into a side platform for secondary analysis; the knowledge distillation optimization operation is executed by taking the side platform as a teacher model and the end platform as a student model, and meanwhile, the detection result is pushed in real time through the cloud platform to complete the construction of the system. According to the method, the detection efficiency and the recognition precision are both considered, the generalization ability and the recognition stability of the model in an unstructured disaster scene are remarkably improved, the system comprehensively adapts to domestic software and hardware, the national secret algorithm and intranet deployment are supported, and the overall security level of the system is effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and intelligent transportation technology, specifically relating to an intelligent detection system for highway weather and disasters based on ResNet-ViT collaborative reasoning. Background Technology Currently, meteorological and disaster monitoring on highways mainly relies on three types of existing technologies: detection systems based on traditional sensors, video image analysis methods, and detection models based on deep learning. Among them, detection systems based on traditional sensors collect physical data such as temperature, humidity, visibility, and road surface conditions by deploying equipment such as weather stations and water level gauges, and combine them with geological disaster sensors to monitor slope displacement or debris flow triggering conditions. Video image analysis methods utilize road surveillance cameras and use traditional image processing algorithms (such as edge detection and threshold segmentation) to identify water accumulation, snow accumulation, or abnormal line-of-sight. Detection models based on deep learning use CNNs (such as YOLO and ResNet) to perform weather classification or target detection on surveillance videos, and some studies attempt to combine time series models (such as LSTM) to improve the adaptability to dynamic scenes.

[0002] However, existing technologies have significant shortcomings, such as insufficient sensor coverage: traditional equipment has a low deployment density, making it difficult to achieve continuous perception across the entire road section, and it cannot identify visual semantic information (such as slope collapse patterns); poor model generalization ability: existing deep learning models are mostly optimized for urban roads, resulting in low recognition rates for small-sample disasters on mountainous highways; high response latency: sensor data and video analysis results require manual verification, leading to poor timeliness of sudden disaster warnings; and conflicting computing power requirements: high-precision ViT models have high computational complexity, making it difficult to meet the real-time requirements of edge devices. Therefore, we now propose a collaborative detection method for highway meteorological disasters that can achieve high precision and low latency under limited computing power. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides an intelligent detection system for highway weather and disasters based on ResNet-ViT collaborative reasoning.

[0004] To achieve the above technology, the following steps are included: S1. Based on the monitoring camera system deployed by the meteorological bureau and along the highway, perform multi-source data parallel acquisition operations to construct sequences with weather tags; The method for parallel acquisition of multi-source data is as follows: The system uses the meteorological bureau's application programming interface (API) to obtain structured precipitation intensity and visibility data for the target area. The obtained structured data includes timestamps and weather types, which include sunny, rainy, and snowy weather. By utilizing the surveillance camera system deployed along the highway, and aligning the surveillance video with the weather type using the Network Time Protocol (NTP), a spatiotemporal correlation sequence is constructed. The method for constructing sequences with weather labels is as follows: By manually labeling meteorological features in monitoring videos, a Level 1 weather labeling system is defined, including: 0 - sunny, 1 - rainy, 2 - snowy.

[0005] S2. Based on the sequences with weather labels, disaster labels are expanded to obtain weather-disaster sequences; To address the problem of scarce samples for disaster events such as slope collapse and debris flow, this invention constructs a disaster image enhancement mechanism; The methods for expanding disaster labels are as follows: Using the image-driven approach of word prompting, we designed disaster-related text prompts with geographical, weather, and scene description capabilities (such as vehicles trapped in landslides and rainy scenes after highway slope collapses). These prompts were then input into the StableDiffusion text-to-image model to generate disaster images, thereby improving the model's ability to identify and generalize rare events. The weather-disaster sequence is constructed as follows: based on the disaster images obtained by the Stable Diffusion model, a two-level disaster label system is expanded by manual annotation, including 3-slope collapse and 4-debris flow, and the disaster images are randomly mixed with sequences with weather labels.

[0006] S3. Define each camera node of the highway-side surveillance camera system as an end-side platform, input the weather-disaster sequence into the end-side platform, and obtain preliminary detection results, including the following steps: S3.1. Configure the edge platform as a Huawei Atlas 300 inference card and deploy it on the Huawei Atlas 300 inference card. An improved dual-branch ResNet50 network; The ResNet50 convolutional neural network model was chosen as the initial detection model for the edge platform. This model has the characteristics of relatively simple structure and high computational efficiency, and can quickly extract features from images and perform preliminary weather classification. Each node is equipped with a Huawei Atlas 300 inference card, which can efficiently support the operation of ResNet models and achieve parallel processing of 8 video streams. At the same time, to ensure the accuracy of time synchronization, all edge platform devices are equipped with NTP time synchronization modules for precise time alignment with subsequent meteorological data and other systems. Leveraging the NPU support of the Huawei Atlas 300 inference card, each card can process multiple simultaneous video inference streams, with the total throughput expressed as: In the formula, Indicates the overall inference throughput; Indicates the number of parallel video streams; This indicates the frame processing rate per channel; Indicates the number of reasoning cards; S3.2 Perform frame extraction on the weather-disaster sequence to obtain a weather-disaster image dataset, and perform data partitioning according to a preset ratio to obtain a training set, a validation set, and a test set; the training set is used for model training and model weight adjustment, the validation set is used for model hyperparameter tuning, and the test set is used for performance evaluation; In this invention, the frame extraction operation is performed as follows: the sequence is divided into segments based on a preset time, a fixed number of seconds is extracted from the segmented sequence, and a fixed number of frames are extracted from the fixed number of seconds. S3.3 Perform preprocessing operations on the training and validation sets; Preprocessing operations include: performing image normalization and standardization operations on the training and validation sets; The image normalization operation is performed by uniformly scaling the image to a fixed size (224×224 pixels) to adapt to the input requirements of ResNet50. The expression for the standardized operation is as follows: In the formula, This represents the result of standardized operations; Represents the images in the input training and validation sets; This represents the RGB mean of the image; Represents the RGB standard deviation of the image; S3.4. Input the preprocessed training and validation sets into the improved dual-branch ResNet50 network to obtain disaster suspected detection results, including the following steps: S3.4.1 Perform random resized crop and random horizontal flip operations on the training set to enhance data diversity; S3.4.2 Construct an improved dual-branch ResNet50 network; Improvements include: multi-scale feature fusion operations, the introduction of channel attention mechanisms, and the introduction of adaptive pooling layers; The multi-scale feature fusion operation involves introducing a feature pyramid between each residual block to enhance the capture of small-scale weather features, as shown in the following expression: In the formula, This represents the fused output feature map; Indicates the higher-level features of the previous layer; This represents the lower-level feature map of the next layer; This represents a 1×1 convolution operation; This indicates an upsampling operation; by fusing features across layers, it solves the problem of missed detection of small-scale targets such as rain and snow in a single-resolution feature map; The channel attention mechanism is introduced as follows: an SE module is introduced into each residual block to automatically learn the importance weights of each channel feature, as expressed below: In the formula, Represents the Sigmoid function; This represents the weight matrix of the second fully connected layer; Represents the ReLU activation function; This represents the weight matrix of the first fully connected layer; and These represent the height and width of the feature map, respectively. and These are indices representing the height and width of the image, respectively. Indicates the feature map at location The value; The introduction of an adaptive pooling layer is to replace the original fixed-size global average pooling with adaptive pooling, which is compatible with different input resolutions. S3.4.3 Input the obtained training set and validation set into the improved dual-branch ResNet50 network to obtain the disaster suspected detection results; The data is input into an improved dual-branch ResNet50 network for forward inference, which sequentially passes through initial convolutional operations, residual connections, SE modules, feature pyramid operations, adaptive pooling layers, and fully connected layers, before outputting the feature extraction results. The improved dual-branch ResNet50 network includes a model training phase and a model validation phase. During the model training phase, the training set is input into the improved dual-branch ResNet50 network. The cross-entropy loss function drives the model to focus on distinguishing between disaster and weather features, and the model weights are updated through backpropagation. The Adam optimizer is used to minimize the cross-entropy loss function. After every fixed number of training epochs, an intermediate model is saved. The expression for the cross-entropy loss function is as follows: In the formula, M represents the number of categories; c represents the index of the category number; Represents the actual value; Indicates the prediction result; The Adam optimizer is used to minimize the cross-entropy loss function. An intermediate model is saved after every 1000 training iterations, and the initial learning rate is set to 0.0001. The expression is as follows: In the formula, This represents the model weights at iteration step t; This represents the model weights at iteration step t-1; Indicates the initial learning rate; This represents the momentum term (a first-order estimate of the gradient). This represents the adaptive learning rate term (the second moment estimate of the gradient). This is to prevent the denominator from being zero. >0; During the model validation phase, the improved two-branch ResNet50 network is input using the validation set. Eval mode is enabled, and the randomness of Dropout and BN is disabled. argmax is used to obtain the predicted classes and calculate the top-1 accuracy. The model parameters with the highest accuracy are saved. The expression for argmax is as follows: In the formula, This represents the model's original prediction score for category c; Indicates the predicted category label; The expression for top-1 accuracy is as follows: In the formula, Indicates the total number of validation sets. An index representing the total number of validation sets; Indicates an indicator function; Indicates the first The prediction results for each argmax; Indicates the first One true value; S3.5 Calculate the confidence score of the edge platform and select those greater than the threshold as the preliminary detection results; The confidence threshold for this invention is set to 0.9.

[0007] S4. Input the preliminary test results into the side platform for secondary analysis; The Ascend 910B and Kylin OS platforms are built on the edge platform, and the ViT analysis module is deployed on the edge platform, enabling it to receive preliminary detection results from the edge platform and perform subsequent in-depth analysis and processing. Among them, the Ascend 910B chip provides powerful computing capabilities for the operation of the ViT model, while the Kylin OS operating system provides stable and secure support for the entire operating environment. The steps of secondary analysis include: S4.1. Call CLIPProcessor to convert the preliminary detection results into the ViT model input format. For each converted image sample, after conversion, the preliminary detection results are divided into a preset number of fixed-size patches, and the patches are flattened and then linearly mapped (patch embedding) is performed, as shown in the following expression: In the formula, Represents the embedding vector after linear mapping; Represents the linear mapping weight matrix; The image represents the first piece; Represents the linear mapping bias term; S4.2 Input the linearly mapped tiles into the ViT model, use a multi-head self-attention mechanism, and obtain the disaster scene recognition result through the classification head; The core of the ViT model is the self-attention mechanism, which assigns attention weights to each tile and automatically learns the relationships between different parts of the tile. The expression for the self-attention mechanism is as follows: In the formula, These represent the query matrix, key matrix, and value matrix, respectively. Indicates dimension; ViT uses a multi-head attention mechanism, which can compute multiple attention heads in parallel, further enhancing the model's expressive power. The expression is as follows: In the formula, Indicates the number of heads of attention; S4.3 Perform training and optimization operations on the ViT model; The training and optimization operations are performed as follows: the cross-entropy loss function is used to calculate the difference between the model's output and the true label; simultaneously, the AdamW optimizer is used to train the model, with a learning rate of (5e) for the classifier. -5 ); To improve training efficiency and performance, an accelerator is used for multi-device acceleration (such as GPU). Accelerator can automatically manage operations such as data parallelism and gradient accumulation, enabling the code to run efficiently in different hardware environments. Specifically, during training, Accelerator includes: automatically handling backpropagation (accelerator.backward(loss)), automatically synchronizing parameters and gradients during distributed training, and dynamically adjusting the batch size to adapt to the GPU's memory limitations. After training is complete, the model's state dictionary (state_dict) is saved in .pth and .safetensors formats for later loading and use; in addition, the mapping relationship between categories and labels is also saved to facilitate subsequent reasoning and analysis.

[0008] S5. Using the edge platform as the teacher model and the terminal platform as the student model, perform knowledge distillation optimization to reduce model parameters and computational load on the terminal platform while maintaining accuracy. At the same time, push the detection results in real time through the cloud platform to complete the system construction. The optimization method for knowledge distillation is as follows: The teacher model is set as the optimized ViT model, and the student model is set as the improved ResNet model. A temperature parameter is introduced to perform a smoothing operation. The loss of knowledge distillation is defined as the KL divergence between the output distribution of the student model and the output distribution of the teacher model. The student model achieves efficient inference on the edge platform by mimicking the output distribution of the teacher model, reducing the amount of computation and memory usage. The final loss function of the student model is the weighted sum of the cross-entropy loss of the edge platform and the KL divergence, as expressed below: In the formula, Represents the balance coefficient. ; This represents the loss value obtained from the KL divergence.

[0009] Beneficial effects of the present invention This invention employs a collaborative inference mechanism combining ResNet and ViT models, integrating lightweight front-end screening with high-precision edge detection, thus balancing detection efficiency and recognition accuracy. The ResNet model rapidly processes video frames sampled every 15 seconds, while the ViT model performs contextual depth modeling on suspected disaster images, enhancing the detection capabilities for complex semantic scenes such as slope collapses and debris flows.

[0010] This invention automatically synthesizes realistic disaster images through a text-based image model guided by prompt word engineering, expands the small sample category dataset, effectively alleviates the problem of data scarcity for rare disasters such as debris flows and slope collapses, and significantly improves the model's generalization ability and recognition stability in unstructured disaster scenarios.

[0011] This invention integrates historical meteorological data and video image data to establish a time-series semantic enhancement mechanism, enabling the model to possess cross-modal environmental perception capabilities. For example, by using meteorological conditions (such as rainfall probability) to assist in disaster-triggered reasoning logic, the accuracy and rationality of scene understanding are improved.

[0012] The system of this invention adopts a multi-channel video polling sampling mechanism, which performs frame sampling every 15 seconds for each channel. Combined with the edge-end collaborative architecture, it achieves distributed deployment and load balancing, and can maintain high throughput and high stability even in resource-constrained edge devices.

[0013] This invention is fully compatible with the Atlas 300 inference card, Ascend 910B acceleration chip, and Kylin operating system—a domestic software and hardware platform—and has the following advantages: Full-stack localization: From algorithm framework and operating system to hardware chips, all components are domestically produced and controllable. Network security auditability: Supports national cryptographic algorithms and intranet deployment, improving the overall security level of the system; Government and enterprise friendly: Suitable for deployment in units with strict information security requirements, such as the Ministry of Transport, provincial highway administration bureaus, and emergency management systems. Attached Figure Description

[0014] Figure 1 This is a flowchart of the steps of the present invention; Figure 2 This is a schematic diagram illustrating the process of expanding disaster labeling in this invention; Figure 3 This is a schematic diagram of the end-side-side structure of the present invention. Detailed Implementation

[0015] The present invention will be further described in detail below with reference to specific embodiments.

[0016] This embodiment uses 500 hours of surveillance video from a section of highway in Yunnan Province, collected over multiple time periods. The video format is H.264, 1080p, 25fps. The PyTorch framework is used, and the graphics card is an NVIDIA RTX 4090.

[0017] like Figure 1 As shown, a highway meteorological and disaster intelligent detection system based on ResNet-ViT collaborative reasoning includes the following steps: S1. Based on the monitoring camera system deployed by the meteorological bureau and along the highway, perform multi-source data parallel acquisition operations to construct sequences with weather tags; The method for parallel acquisition of multi-source data is as follows: The system uses the meteorological bureau's application programming interface (API) to obtain structured precipitation intensity and visibility data for the target area. The obtained structured data includes timestamps and weather types, which include sunny, rainy, and snowy weather. By utilizing the surveillance camera system deployed along the highway, and aligning the surveillance video with the weather type using the Network Time Protocol (NTP), a spatiotemporal correlation sequence is constructed. The method for constructing sequences with weather labels is as follows: By manually labeling meteorological features in monitoring videos, a Level 1 weather labeling system is defined, including: 0 - sunny, 1 - rainy, 2 - snowy.

[0018] S2. Based on the sequences with weather labels, disaster labels are expanded to obtain weather-disaster sequences; To address the problem of scarce samples for disaster events such as slope collapse and debris flow, this invention constructs a disaster image enhancement mechanism; like Figure 2 As shown, the method for expanding disaster labels is as follows: Using the image-driven approach of word prompting, we designed disaster-related text prompts with geographical, weather, and scene description capabilities (such as vehicles trapped in landslides and rainy scenes after highway slope collapses). These prompts were then input into the StableDiffusion text-to-image model to generate disaster images, thereby improving the model's ability to identify and generalize rare events. The weather-disaster sequence is constructed as follows: based on the disaster images obtained by the Stable Diffusion model, a two-level disaster label system is expanded by manual annotation, including 3-slope collapse and 4-debris flow, and the disaster images are randomly mixed with sequences with weather labels.

[0019] S3. Define each camera node of the highway-side surveillance camera system as an end-side platform, input the weather-disaster sequence into the end-side platform, and obtain preliminary detection results, including the following steps: S3.1 Set the edge platform to Huawei Atlas 300 inference card and deploy an improved dual-branch ResNet50 network on Huawei Atlas 300 inference card; The ResNet50 convolutional neural network model was chosen as the initial detection model for the edge platform. This model has the characteristics of relatively simple structure and high computational efficiency, and can quickly extract features from images and perform preliminary weather classification. Each node is equipped with a Huawei Atlas 300 inference card, which can efficiently support the operation of ResNet models and achieve parallel processing of 8 video streams. At the same time, to ensure the accuracy of time synchronization, all edge platform devices are equipped with NTP time synchronization modules for precise time alignment with subsequent meteorological data and other systems. Leveraging the NPU support of the Huawei Atlas 300 inference card, each card can process multiple simultaneous video inference streams, with the total throughput expressed as: In the formula, Indicates the overall inference throughput; This represents the number of parallel video streams; in this embodiment, it is set to 8. This represents the frame processing rate for each channel; in this embodiment, it is taken as 2fps. This indicates the number of reasoning cards; in this embodiment, it is set to 1. The theoretical inference rate is: 8 × 2 = 16 fps; S3.2 Perform frame extraction on the weather-disaster sequence to obtain a weather-disaster image dataset, and perform data partitioning according to a preset ratio to obtain a training set, a validation set, and a test set; the training set is used for model training and model weight adjustment, the validation set is used for model hyperparameter tuning, and the test set is used for performance evaluation; In this invention, the frame extraction operation is performed as follows: the sequence is divided into segments based on a preset time, a fixed number of seconds is extracted from the segmented sequence, and a fixed number of frames are extracted from the fixed number of seconds. In this embodiment, the sequence is divided into 3-minute segments, and a 15-second video sequence is extracted from the divided sequence. The frame extraction operation is performed at 2fps. In this embodiment, the preset ratio is 8:1:1; the validation set has 16,000 images, the test set has 1,000 images; S3.3 Perform preprocessing operations on the training and validation sets; Preprocessing operations include: performing image normalization and standardization operations on the training and validation sets; The image normalization operation is performed by uniformly scaling the image to a fixed size (224×224 pixels) to adapt to the input requirements of ResNet50. The expression for the standardized operation is as follows: In the formula, This represents the result of standardized operations; Represents the images in the input training and validation sets; This represents the RGB mean of the image. ; This represents the RGB standard deviation of the image. ; S3.4. Input the preprocessed training and validation sets into the improved dual-branch ResNet50 network to obtain disaster suspected detection results, including the following steps: S3.4.1 Perform random resized crop and random horizontal flip operations on the training set to enhance data diversity; S3.4.2 Construct an improved dual-branch ResNet50 network; Improvements include: multi-scale feature fusion operations, the introduction of channel attention mechanisms, and the introduction of adaptive pooling layers; The multi-scale feature fusion operation involves introducing a feature pyramid between each residual block to enhance the capture of small-scale weather features, as shown in the following expression: In the formula, This represents the fused output feature map; Indicates the higher-level features of the previous layer; This represents the lower-level feature map of the next layer; This represents a 1×1 convolution operation; This indicates an upsampling operation; by fusing features across layers, it solves the problem of missed detection of small-scale targets such as rain and snow in a single-resolution feature map; The channel attention mechanism is introduced as follows: an SE module is introduced into each residual block to automatically learn the importance weights of each channel feature, as expressed below: In the formula, Represents the Sigmoid function; This represents the weight matrix of the second fully connected layer; Represents the ReLU activation function; This represents the weight matrix of the first fully connected layer; and These represent the height and width of the feature map, respectively. and These are indices representing the height and width of the image, respectively. Indicates the feature map at location The value; The introduction of an adaptive pooling layer is to replace the original fixed-size global average pooling with adaptive pooling, which is compatible with different input resolutions. S3.4.3 Input the obtained training set and validation set into the improved dual-branch ResNet50 network to obtain the disaster suspected detection results; The data is input into an improved dual-branch ResNet50 network for forward inference, which sequentially passes through initial convolutional operations, residual connections, SE modules, feature pyramid operations, adaptive pooling layers, and fully connected layers, before outputting the feature extraction results. The improved dual-branch ResNet50 network includes a model training phase and a model validation phase. During the model training phase, the training set is input into the improved dual-branch ResNet50 network. The cross-entropy loss function drives the model to focus on distinguishing between disaster and weather features, and the model weights are updated through backpropagation. The Adam optimizer is used to minimize the cross-entropy loss function. After every fixed number of training epochs, an intermediate model is saved. The expression for the cross-entropy loss function is as follows: In the formula, M represents the number of categories, and in this invention, M is 5; c represents the index of the number of categories. Represents the actual value; Indicates the prediction result; The Adam optimizer is used to minimize the cross-entropy loss function. An intermediate model is saved after every 1000 training iterations, and the initial learning rate is set to 0.0001. The expression is as follows: In the formula, This represents the model weights at iteration step t; This represents the model weights at iteration step t-1; Indicates the initial learning rate; This represents the momentum term (a first-order estimate of the gradient). This represents the adaptive learning rate term (the second moment estimate of the gradient). This is to prevent the denominator from being zero. >0; During the model validation phase, the improved two-branch ResNet50 network is input using the validation set. Eval mode is enabled, and the randomness of Dropout and BN is disabled. argmax is used to obtain the predicted classes and calculate the top-1 accuracy. The model parameters with the highest accuracy are saved. The expression for argmax is as follows: In the formula, This represents the model's original prediction score for category c; Indicates the predicted category label; The expression for top-1 accuracy is as follows: In the formula, Indicates the total number of validation sets. An index representing the total number of validation sets; Indicates an indicator function; Indicates the first The prediction results for each argmax; Indicates the first One true value; S3.5 Calculate the confidence score of the edge platform and select those greater than the threshold as the preliminary detection results; The confidence threshold for this invention is set to 0.9.

[0020] S4. Input the preliminary test results into the side platform for secondary analysis, such as... Figure 3 As shown; The Ascend 910B and Kylin OS platforms are built on the edge platform, and the ViT analysis module is deployed on the edge platform, enabling it to receive preliminary detection results from the edge platform and perform subsequent in-depth analysis and processing. Among them, the Ascend 910B chip provides powerful computing capabilities for the operation of the ViT model, while the Kylin OS operating system provides stable and secure support for the entire operating environment. The steps of secondary analysis include: S4.1. Call CLIPProcessor to convert the preliminary detection results into the format of the ViT model input. For each converted image sample, after transformation (such as scaling, cropping, etc.), the preliminary detection results are divided into a preset number of fixed-size patches, and the patches are flattened and then linearly mapped (patch embedding) is performed. The expression is as follows: In the formula, Represents the embedding vector after linear mapping; Represents the linear mapping weight matrix; The image represents the first piece; Represents the linear mapping bias term; S4.2 Input the linearly mapped tiles into the ViT model, use a multi-head self-attention mechanism, and obtain the disaster scene recognition result through the classification head; The core of the ViT model is the self-attention mechanism, which assigns attention weights to each tile and automatically learns the relationships between different parts of the tile. The expression for the self-attention mechanism is as follows: In the formula, These represent the query matrix, key matrix, and value matrix, respectively. Indicates dimension; ViT uses a multi-head attention mechanism, which can compute multiple attention heads in parallel, further enhancing the model's expressive power. The expression is as follows: In the formula, Indicates the number of heads of attention; S4.3 Perform training and optimization operations on the ViT model; The training and optimization operations are performed as follows: the cross-entropy loss function is used to calculate the difference between the model's output and the true label; simultaneously, the AdamW optimizer is used to train the model, with a learning rate of (5e) for the classifier. -5 ); To improve training efficiency and performance, an accelerator is used for multi-device acceleration (such as GPU). Accelerator can automatically manage operations such as data parallelism and gradient accumulation, enabling the code to run efficiently in different hardware environments. Specifically, during training, Accelerator includes: automatically handling backpropagation (accelerator.backward(loss)), automatically synchronizing parameters and gradients during distributed training, and dynamically adjusting the batch size to adapt to the GPU's memory limitations. After training is complete, the model's state dictionary (state_dict) is saved in .pth and .safetensors formats for later loading and use; in addition, the mapping relationship between categories and labels is also saved to facilitate subsequent reasoning and analysis.

[0021] S5. Using the edge platform as the teacher model and the terminal platform as the student model, perform knowledge distillation optimization to reduce model parameters and computational load on the terminal platform while maintaining accuracy. At the same time, push the detection results in real time through the cloud platform to complete the system construction. The optimization method for knowledge distillation is as follows: The teacher model is set as the optimized ViT model, and the student model is set as the improved ResNet model. A temperature parameter is introduced to perform a smoothing operation. The loss of knowledge distillation is defined as the KL divergence between the output distribution of the student model and the output distribution of the teacher model. The student model achieves efficient inference on the edge platform by mimicking the output distribution of the teacher model, reducing the amount of computation and memory usage. The final loss function of the student model is the weighted sum of the cross-entropy loss of the edge platform and the KL divergence, as expressed below: In the formula, Represents the balance coefficient. ; This represents the loss value obtained from the KL divergence. The expression for incorporating temperature parameters to perform smoothing operations is as follows: In the formula, This represents the smoothed teacher-student model; This represents a teacher-student model. Indicates temperature parameter; The cloud platform is equipped with data storage, processing, and display capabilities. It features a secure, encrypted data transmission interface for receiving detection results pushed from the edge. Simultaneously, an early warning data push module and a platform early warning display module are deployed on the cloud platform to visualize detection results and deliver timely early warning information. The detection results are uniformly formatted into structured information and pushed to the upper-level management platform via a secure, encrypted interface, providing the following functions: Real-time alarm information pop-up display; Disaster event image visualization; Historical records can be traced and exported. Connect to the traffic emergency dispatch system to achieve closed-loop disaster response.

[0022] To verify the present invention, its performance was compared with other models, including ResNet50, ViT-Base, and Swin Transformer. The performance metrics included accuracy, precision, recall, and F1 score. The results are shown in Table 1. Table 1: Model Comparison Results As can be seen from Table 1, the model of this invention combines the local feature extraction advantages of ResNet with the global modeling capabilities of ViT, and outperforms traditional convolutional networks and pure Transformer architectures in all aspects.

[0023] To verify the performance of this invention in disaster-related small sample identification tasks, ResNet50 and ViT-Base were used as comparison models to verify the F1 score. The results are shown in Table 2. Table 2: Comparison of small sample validation results for disasters As shown in Table 2, traditional models perform poorly in the two disaster categories of slope collapse and debris flow due to the scarcity of training samples. However, the method of this invention effectively improves the model's ability to identify low-frequency classes by integrating multi-layer features and attention mechanisms.

[0024] To verify the inference efficiency of this invention, ResNet50 and ViT-Base were used as comparison models for efficiency verification, and the results are shown in Table 3. Table 3: Model Efficiency Comparison As can be seen from Table 3, the collaborative reasoning structure proposed in this invention successfully compresses the model size and improves the reasoning speed without significantly increasing the computational load, thus meeting the application requirements of real-time online identification of highways.

[0025] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A highway meteorological and disaster intelligent detection system based on ResNet-ViT collaborative reasoning, characterized in that, Includes the following steps: S1. Based on the monitoring camera system deployed by the meteorological bureau and along the highway, perform multi-source data parallel acquisition operations to construct sequences with weather tags; S2. Based on the sequences with weather labels, disaster labels are expanded to obtain weather-disaster sequences; The expansion method is as follows: design disaster-related text prompts with geographical, weather and scene description capabilities, input them into the text-generated image model Stable Diffusion, generate disaster images, and randomly mix the disaster images into sequences with weather tags; S3. Define each camera node of the highway-side surveillance camera system as an end-side platform, input the weather-disaster sequence into the end-side platform, and obtain preliminary detection results; The edge platform is a Huawei Atlas 300 inference card, and an improved dual-branch ResNet50 network is deployed on the Huawei Atlas 300 inference card; The improvements to the improved dual-branch ResNet50 network include: introducing multi-scale feature fusion operations, introducing a channel attention mechanism, and introducing an adaptive pooling layer. S4. Input the preliminary test results into the side platform for secondary analysis; The Ascend 910B and Kylin OS platforms are built on the edge platform, and the ViT network is deployed on the edge platform to receive preliminary detection results from the end platform. S5. Using the edge platform as the teacher model and the terminal platform as the student model, perform knowledge distillation optimization to reduce model parameters and computational load on the terminal platform while maintaining accuracy. At the same time, push the detection results in real time through the cloud platform to complete the system construction.

2. The intelligent highway meteorological and disaster detection system based on ResNet-ViT collaborative reasoning according to claim 1, characterized in that, The steps for defining each camera node of the highway-side surveillance camera system as an end-side platform and inputting weather-disaster sequences into the end-side platform to obtain preliminary detection results include: S3.1 Set the edge platform to Huawei Atlas 300 inference card and deploy an improved dual-branch ResNet50 network on Huawei Atlas 300 inference card; S3.2 Perform frame extraction on the weather-disaster sequence to obtain a weather-disaster image dataset, and perform data partitioning according to a preset ratio to obtain a training set, a validation set, and a test set; The training set is used for model training and model weight adjustment; The validation set is used for model hyperparameter tuning; The test set is used for performance evaluation; S3.3 Perform preprocessing operations on the training and validation sets; S3.4 Input the preprocessed training set and validation set into the improved dual-branch ResNet50 network to obtain disaster suspected detection results; S3.5 Calculate the confidence score of the edge platform and select those greater than the threshold as the preliminary detection results.

3. The intelligent detection system for highway meteorology and disasters based on ResNet-ViT collaborative reasoning according to claim 2, characterized in that, The step of inputting the preprocessed training set and validation set into the improved dual-branch ResNet50 network to obtain disaster suspected detection results includes: S3.4.1 Perform random cropping, scaling, and horizontal flipping operations on the training set to enhance data diversity; S3.4.2 Construct an improved dual-branch ResNet50 network; Improvements include: multi-scale feature fusion operations, the introduction of channel attention mechanisms, and the introduction of adaptive pooling layers; S3.4.3 Input the obtained training set and validation set into the improved dual-branch ResNet50 network to obtain the disaster suspected detection results; The data is input into an improved dual-branch ResNet50 network for forward inference, which sequentially passes through initial convolutional operations, residual connections, SE modules, feature pyramid operations, adaptive pooling layers, and fully connected layers, before outputting the feature extraction results. The improved dual-branch ResNet50 network includes a model training phase and a model validation phase. During the model training phase, the training set is input into the improved dual-branch ResNet50 network, and the cross-entropy loss function is used to drive the model to focus on distinguishing disaster and weather features. The model weights are updated through the backpropagation algorithm. The Adam optimizer is used to minimize the cross-entropy loss function, and the intermediate model is saved once after every fixed number of training epochs. During the model validation phase, the improved dual-branch ResNet50 network is input with the validation set, the eval mode is enabled, the randomness of Dropout and BN is disabled, the predicted class is obtained using argmax and the top-1 accuracy is calculated, and the model parameters with the highest accuracy are saved.

4. The intelligent detection system for highway meteorology and disasters based on ResNet-ViT collaborative reasoning according to claim 1, characterized in that, The steps for inputting the preliminary detection results into the side-side platform for secondary analysis are as follows: S4.1 Call CLIPProcessor to convert the preliminary detection results into the ViT model input format. For each converted image sample, after conversion, the preliminary detection results are divided into a preset number of fixed-size patches, and the patches are flattened and linearly mapped. S4.2 Input the linearly mapped tiles into the ViT model, use a multi-head self-attention mechanism, and obtain the disaster scene recognition result through the classification head; S4.3 Perform training and optimization operations on the ViT model; The training and optimization operations are performed as follows: the cross-entropy loss function is used to calculate the difference between the model's output and the true label; at the same time, the AdamW optimizer is used to train the model.

5. The intelligent detection system for highway meteorology and disasters based on ResNet-ViT collaborative reasoning according to claim 1, characterized in that, The system employs a side-side platform as the teacher model and an edge-side platform as the student model to perform knowledge distillation optimization. This reduces model parameters and computational load on the edge-side platform while maintaining accuracy. Simultaneously, the system pushes detection results in real time through a cloud platform. The knowledge distillation optimization operation is performed as follows: The teacher model is set as the optimized ViT model, and the student model is set as the improved ResNet model. A temperature parameter is introduced to perform a smoothing operation. The loss of knowledge distillation is defined as the KL divergence between the output distribution of the student model and the output distribution of the teacher model. The student model achieves efficient inference on the edge platform by mimicking the output distribution of the teacher model, reducing the amount of computation and memory usage. The final loss function of the student model is the weighted sum of the cross-entropy loss of the edge platform and the KL divergence, as expressed below: In the formula, Represents the balance coefficient. ; This represents the loss value obtained from the KL divergence.

Images