Multi-source data fusion monitoring video sea fog identification and situation prediction method and system

Abstract

The invention belongs to the technical field of sea fog identification, and discloses a multi-source data fusion monitoring video sea fog identification and situation prediction method and system. According to the sea fog forecasting method based on the video data of the monitoring camera, image processing and deep learning are combined, multi-source observation information is fused, and accurate identification and forecasting of sea fog are achieved. The core of the technical scheme is that the formation, change and dissipation processes of the sea fog are analyzed and forecasted in real time by using monitoring video data and combining image feature extraction, deep learning model and meteorological data fusion, and then the accuracy and real-time performance of sea fog early warning are improved. According to the invention, a dark channel method is adopted to preprocess monitoring video data, illumination interference is effectively removed, sea fog characteristics are enhanced, and thus sea fog detection and identification precision is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of sea fog recognition, and in particular relates to a monitoring video sea fog recognition and situation prediction method and system based on multi-source data fusion. Background Art

[0002] Monitoring and forecasting sea fog is an important task to ensure the safety of coastal cities, marine traffic and air transportation. Existing technologies mainly rely on meteorological stations, radars, satellite remote sensing data, and methods based on image processing and machine learning. Among them, meteorological stations determine the formation conditions of sea fog by measuring parameters such as temperature, humidity, and wind speed, radar is used to detect the distribution characteristics of fog, and satellite remote sensing technology monitors the distribution of sea fog over a large area. In addition, with the development of computer vision technology, fog recognition methods based on surveillance videos and machine learning have become an emerging means, providing new possibilities for sea fog monitoring. However, these technical means still have significant limitations and challenges in practical applications.

[0003] Technical Problem 1: Blurred and low contrast video data

[0004] Surveillance video is an important data source for capturing sea fog features. However, the features of sea fog images are complex and diverse, often manifested as reduced brightness, reduced contrast, and blurred edges, which reduces the amount of available information in the image and affects the accuracy of feature extraction. At the same time, sea fog of different concentrations and thicknesses has a large impact on the image, and is significantly affected by interference factors such as background lighting, waves, and ships. Existing image processing technologies are insufficient in extracting sea fog features in complex environments, resulting in low accuracy and stability in sea fog recognition in surveillance videos.

[0005] Technical Problem 2: Insufficient Modeling of Dynamically Changing Spatiotemporal Characteristics

[0006] The dynamic evolution of sea fog has significant spatiotemporal characteristics, and traditional static image methods cannot effectively describe its dynamic change process. Although some machine learning-based models introduce time series data, existing methods are usually limited to simple time-dependent modeling and have difficulty capturing complex dynamic changes. In addition, the formation and dissipation of sea fog are affected by a combination of meteorological factors such as temperature, humidity, and wind speed. Existing methods find it difficult to combine these dynamic physical characteristics with the spatiotemporal evolution process, resulting in insufficient accuracy and timeliness in sea fog situation prediction.

[0007] Technical Issue 3: Limitations of a Single Data Source

[0008] Traditional sea fog observation technologies mostly rely on a single data source. For example, the meteorological data provided by meteorological stations are mostly distributed in a point-like manner and cannot cover a large area; satellite remote sensing is affected by cloud cover and is difficult to capture subtle changes in fog; although surveillance videos provide rich image information, they lack the support of meteorological background data and are difficult to accurately describe the occurrence and development process of sea fog. Therefore, a single data source cannot meet the comprehensiveness and accuracy requirements of sea fog monitoring.

[0009] Technical Problem 4: Model processing is slow and cannot provide real-time forecasts

[0010] The dynamic changes of sea fog require the monitoring system to be able to capture the formation and expansion process of fog in real time. However, existing machine learning models have problems with low computational efficiency and slow processing speed when processing massive video data. For example, although deep learning models perform well in image feature extraction, they still have difficulty ensuring real-time performance when facing large-scale video streams. In addition, most existing methods are single-task models that cannot simultaneously achieve multi-task processing such as recognition, segmentation, and prediction, further affecting the system's real-time forecasting capabilities.

[0011] Technical Problem 5: Limited Capability of Capturing Complex Nonlinear Characteristics

[0012] The generation and dissipation of sea fog involves complex nonlinear physical mechanisms, such as the combined effects of meteorological factors, lighting conditions, and environmental interference. Although traditional machine learning methods (such as support vector machines) perform well on single tasks, they are unable to capture these complex nonlinear feature relationships. Even deep learning models have the problem of insufficient modeling of long-term dependencies and cannot effectively describe the subtle changes in the dynamic situation of sea fog.

[0013] Technical Problem 6: Lack of comprehensive prediction capabilities for multiple tasks and multiple targets

[0014] Most existing technologies focus on the single task of sea fog identification or judgment, but lack support for comprehensive tasks such as level quantification, expansion range prediction, direction and speed calculation, etc. In addition, different scenarios have different requirements for sea fog monitoring systems. For example, marine transportation requires the prediction of sea fog expansion direction and speed, while coastal cities may be more concerned with the quantification of visibility levels. Single-function technical means are difficult to meet the diverse needs of practical applications, limiting the practicality of sea fog monitoring systems.

[0015] In summary, the existing technologies have the following key problems in sea fog monitoring: the ambiguity and complexity of surveillance videos make it difficult to accurately extract features through image processing; the dynamically changing spatiotemporal features are difficult to accurately model through traditional methods; the limitations of a single data source weaken the comprehensiveness and stability of the system; the model processing speed is insufficient to meet the needs of real-time forecasting; the ability to capture complex nonlinear features is limited; and there is a lack of comprehensive prediction capabilities for multiple tasks and multiple targets. To solve these problems, it is necessary to develop an innovative sea fog monitoring and prediction system that can integrate multi-source data, capture dynamic changes, and have real-time and multi-task processing capabilities. Summary of the invention

[0016] In view of the problems existing in the prior art, the present invention provides a method and system for monitoring video sea fog recognition and situation prediction by multi-source data fusion.

[0017] The present invention is implemented in this way: a monitoring video sea fog recognition and situation prediction method based on multi-source data fusion includes:

[0018] Step 1, video data preprocessing;

[0019] The dark channel method is used to pre-process the surveillance video; the contrast of the image is enhanced, the low-brightness background is removed, and the details of the haze are highlighted;

[0020] Image color space conversion: First, the video frame image is converted from the RGB color space to the Lab color space. In the Lab space, brightness and chromaticity are separated to process haze-related features.

[0021] Then, the dark channel value I of each pixel in each frame image is calculated by the dark channel method. dark , using the formula:

[0022]

[0023] Among them, I c Represents the pixel value of the image in color channel c, and min represents the minimum value;

[0024] Step 2: video and multi-source observation data registration;

[0025] Step 3: Time series modeling and sea fog situation prediction.

[0026] Furthermore, the video and multi-source observation data registration method:

[0027] 1) Temporal Registration Model:

[0028] Timestamp alignment: Compare the timestamps of all data sources. If there are time differences in the data, use interpolation methods to adjust the inconsistent multi-source data. For video data, segment the video based on the frame rate to ensure that the timestamp of each video frame corresponds to the time point of other data sources.

[0029] Temporal interpolation methods: If the sensor data is based on interval time, linear interpolation or more complex interpolation methods are used to fill in the missing parts between the data to match the timestamps of the video data;

[0030] 2) Spatial registration model:

[0031] Multi-source data registration: unify the spatial information of different sensors or data sources, and use GIS tools or OpenCV-based registration methods to perform spatial registration;

[0032] Low-resolution preliminary registration: Starting from the lowest resolution image, calculate the transformation of the two images to obtain a rough registration result;

[0033] Layer-by-layer refinement: Continue to refine the registration results on higher resolution image layers until the resolution of the original image is reached;

[0034] Final transformation application: Apply the registration result to the original image for spatial alignment, so that the surveillance video and multi-source data can be fused and analyzed in the same coordinate system.

[0035] Furthermore, the time series modeling and sea fog situation prediction:

[0036] The calculation process of the LSTM unit is as follows:

[0037] Forget gate: decides how much past memory to forget; based on the current input x t and the output h at the previous moment t-1 , the output f of the forget gate t for:

[0038] f t =σ(W f [h t-1 ,x t ]+b f )

[0039] Among them, W f is the weight matrix of the forget gate, b f is the bias term, σ is the Sigmoid activation function;

[0040] Input gate: determines how much new information to update; the output i of the input gate t for:

[0041] i t=σ(W i [h t-1 ,x t ]+b i )

[0042]

[0043] in, is the candidate state, tanh is the hyperbolic tangent activation function;

[0044] Unit state update: Combine the forget gate and the input gate to update the unit state C t :

[0045]

[0046] Among them, C t-1 is the cell state at the previous moment, C t is the current cell state

[0047] Output gate: According to the current cell state and the state of the output gate o t Generate the output h at the current time t :

[0048] o t =σ(Wo[h t-1 ,x t ]+b0)

[0049] h t =o t *tanh(C t )

[0050] Among them, W o is the weight of the output gate, b o is the bias term;

[0051] Through the above process, LSTM can capture the temporal changes of sea fog based on the meteorological information of past image data;

[0052] Transformer enhances time series modeling;

[0053] LSTM and Transformer combined with sea fog recognition and prediction model.

[0054] Furthermore, the Transformer enhances temporal modeling:

[0055] The Transformer model is introduced to capture long-range dependencies in the sequence through the self-attention mechanism;

[0056] The self-attention mechanism of the Transformer model allows each time step to make a weighted combination of the inputs of all other time steps, effectively modeling the dependencies of long time series; calculation of the self-attention mechanism.

[0057] Furthermore, the LSTM and Transformer are combined with the sea fog recognition and prediction model:

[0058] The structure of the hybrid model is as follows:

[0059] LSTM part: used to capture short-term temporal dependencies in sea fog video data;

[0060] Transformer part: Use the self-attention mechanism to enhance the modeling of long-range dependencies in long time series and adapt to the long-term trend of sea fog changes; Fusion mechanism: Fusion the outputs of LSTM and Transformer through the fully connected layer to generate the final sea fog prediction result.

[0061] Further, the calculation of the self-attention mechanism:

[0062] Input representation: Input sequence X = [x1, x2, ..., x T ], where T is the sequence length, x t is the input at time step t;

[0063] Calculate attention weights: For each input x t It is mapped into three vectors: query, key, and value. The query, key, and value are obtained through linear transformation:

[0064] Q=XW Q ,K=XW K ,V=XW V

[0065] Among them, W Q , W K , W V is the weight matrix, corresponding to the transformation of query, key and value respectively;

[0066] Calculating self-attention: The output of self-attention is calculated as a weighted value, where the weight is calculated by the dot product of the query and the key:

[0067]

[0068] Among them, d k is the dimension of the key vector, and the softmax function ensures that the weight sums to 1;

[0069] MultiHead Attention: In order to capture different semantic information, Transformer usually uses a multi-head attention mechanism, that is, multiple independent attention calculations are performed in parallel and the results are merged:

[0070] MultiHead(Q,K,V)=Concat(head1,...,head h )W o

[0071] Among them, head is the attention head, h is the number of attention heads, and W o is the output linear transformation matrix;

[0072] Through the self-attention mechanism, Transformer can dynamically focus on the relationship between different time steps in the sequence, especially in the long-term evolution of sea fog, effectively capturing dependencies with a longer time span;

[0073] Multi-source data fusion:

[0074] Assume there are M source data Z1, Z2…, Z M , each source data is feature extracted through a shared neural network, and the obtained feature vectors are f1,f2,…,f M ; Then, the weighted summation method is used to fuse the features to obtain f fused :

[0075]

[0076] Among them, w i is the weight of each source data, and the weight value is adjusted according to the importance of the data; the fused features are input into the LSTM-Transformer hybrid model to implement the following tasks;

[0077] Sea fog recognition model:

[0078] Based on the above LSTM-Transformer hybrid model, a soft-max layer is added to the output of the model to realize the binary judgment of the presence or absence of sea fog, thereby realizing the recognition of sea fog;

[0079] Quantitative model of sea fog level:

[0080] This model is used to classify the identified sea fog into different levels. Based on the above LSTM-Transformer hybrid model, the output of the hybrid model is mapped to different sea fog level ranges using a fully connected layer.

[0081] Calculation model of sea fog movement direction:

[0082] First, the sea fog area in the video is detected based on the LSTM-Transformer hybrid model;

[0083] Then, the optical flow algorithm is used to calculate the motion vector field between consecutive video frames to detect the movement direction of the sea fog pixels;

[0084] Secondly, the image is divided into blocks to obtain the motion vector of each area, and then the overall direction is determined by calculating the mean;

[0085] Finally, based on the motion vector, the sea fog movement angle at each moment is calculated to obtain the main movement direction;

[0086] Sea fog movement speed calculation model:

[0087] First, based on the sea fog area in the video detected by the LSTM-Transformer hybrid model, the optical flow method is used to extract speed information:

[0088] Then, the optical flow method is used to calculate the motion distance between adjacent frames to obtain the movement speed of the sea fog;

[0089] Secondly, extract the velocity components in the horizontal and vertical directions, and calculate the average velocity of each block area;

[0090] Finally, according to the area covered by the sea fog, the overall movement speed is obtained using weighted average;

[0091] Sea fog development trend prediction model:

[0092] Also based on the LSTM-Transformer hybrid model described above, the fully connected layer is used to map the target prediction value to obtain the sea fog level or extension range prediction for each time step.

[0093] Another object of the present invention is to provide a multi-source data fusion monitoring video sea fog recognition and situation prediction system comprising:

[0094] The preprocessing module is used for video data preprocessing; the dark channel method is used to preprocess the surveillance video; the contrast of the image is enhanced, the low-brightness background is removed, and the details of the haze are highlighted;

[0095] Registration module, used for video and multi-source observation data registration;

[0096] Prediction module, used for time series modeling and sea fog situation prediction.

[0097] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the surveillance video sea fog recognition and situation prediction method based on multi-source data fusion.

[0098] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to execute the steps of the surveillance video sea fog recognition and situation prediction method based on multi-source data fusion.

[0099] Another object of the present invention is to provide an information data processing terminal, which is used to implement the monitoring video sea fog recognition and situation prediction system with multi-source data fusion.

[0100] In combination with the above technical solutions and the technical problems solved, please analyze the advantages and positive effects of the technical solutions to be protected by the present invention from the following aspects:

[0101] First, the present invention proposes a method for predicting sea fog situation based on the fusion of video image data and multi-source data. This method uses a dark channel prior defogging algorithm to pre-process the video image to eliminate the influence of ambient lighting and enhance the characteristics of fog; then, through a time series modeling method combining a long short-term memory network (LSTM) and a Transformer, the feature fusion of video data and multi-source data is realized, thereby accurately predicting the sea fog situation. This method can not only track the dynamics of sea fog in time and space, but also use a deep learning model to capture complex nonlinear feature changes, providing new technical support for low-cost, real-time, high-precision sea fog monitoring.

[0102] The present invention proposes a sea fog forecasting method based on surveillance camera video data, which combines image processing with deep learning and integrates multi-source observation information to achieve accurate identification and forecasting of sea fog. The core of this technical solution is to use surveillance video data, combined with image feature extraction, deep learning models and the fusion of meteorological data, to analyze and forecast the formation, change and dissipation of sea fog in real time, thereby improving the accuracy and real-time performance of sea fog warning.

[0103] 1. Efficient sea fog detection and identification: The present invention adopts the dark channel method to pre-process the monitoring video data, effectively removes the light interference, enhances the sea fog characteristics, and thus improves the detection and identification accuracy of sea fog.

[0104] 2. Combining time series modeling with multi-source data fusion: Through the combination of LSTM and Transformer models, it is possible to capture the time series characteristics of sea fog changes and integrate multi-source observation information (such as meteorological data) to improve prediction accuracy. This comprehensive approach helps to more fully understand the dynamic changes of sea fog.

[0105] 3. Strong adaptability: The present invention can adapt to different environmental conditions and diverse sea fog characteristics, and is suitable for sea fog monitoring and prediction in different regions and under different weather conditions.

[0106] 4. Real-time and high efficiency: Using the real-time video data from surveillance cameras, rapid monitoring and early warning of sea fog situations can be achieved, providing timely decision-making support for relevant departments.

[0107] 5. Low-cost solution: Compared with traditional meteorological monitoring equipment, the present invention utilizes existing surveillance camera resources to build a low-cost, high-efficiency sea fog monitoring system, reducing the economic burden of technology implementation.

[0108] 6. Adaptability to complex environments: Through multi-source data fusion, the model can effectively resist interference in complex environments, improve the reliability of sea fog monitoring, and ensure that high prediction accuracy is maintained under changing meteorological conditions.

[0109] 7. Scalability and wide application: This method can not only be used for sea fog monitoring, but can also be extended to the identification and prediction of other meteorological phenomena, and has good application prospects and market value.

[0110] 8. Technical support and scientific research value: It provides strong technical support for the fields of ocean, meteorology, environmental monitoring, etc., and promotes the in-depth development of related scientific research.

[0111] Protection point:

[0112] 1. Innovative use of surveillance camera video data to achieve intelligent monitoring of sea fog;

[0113] 2. Surveillance video color space conversion for sea fog recognition;

[0114] 3. Feature extraction of sea fog video based on dark channel method;

[0115] 4. Time series modeling of multi-source sea fog observation data based on LSTM;

[0116] 5. Transformer-based enhanced sea fog motion change prediction model;

[0117] 6. LSTM+Transformer sea fog recognition model;

[0118] 7. LSTM+Transformer sea fog level quantization model;

[0119] 8. LSTM+Transformer sea fog movement direction calculation model;

[0120] 9. LSTM+Transformer sea fog movement speed calculation model;

[0121] 10. LSTM+Transformer sea fog level extension range prediction model;

[0122] 11. Weighted fusion model of surveillance video + multi-source meteorological observation data;

[0123] 12. Temporal registration model of surveillance video + multi-source meteorological observation data;

[0124] 13.Spatial registration model of surveillance video + multi-source meteorological observation data;

[0125] 14. A method for unifying multi-scale spatial resolution of surveillance video + multi-source meteorological observation data based on pyramid

[0126] 15. Construction of sea fog monitoring video dataset;

[0127] Temporal interpolation model for multi-source observational data;

[0128] Second, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0129] (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are:

[0130] The present invention can provide sea fog observation data with high temporal and spatial resolution for relevant scientific research and business needs such as oceanography, meteorology, and environmental monitoring, which is of great significance for ensuring the normal operation of marine transportation and aviation, and the daily life and economic activities of coastal cities. The equipment involved in the patented technical solution is easy to operate and economical, with a simple and clear process, strong portability and replicability, and can not only be used for sea fog monitoring, but also can be extended to the identification and prediction of other meteorological phenomena. It has good application prospects and market value, and has the potential for large-scale promotion and use.

[0131] (2) The technical solution of the present invention fills the technical gap in the industry at home and abroad:

[0132] Traditional sea fog observation usually relies on weather stations, radar and satellite data, but the single observation method is difficult to meet the needs of real-time, high-precision, multi-time and space scale monitoring and prediction of sea fog. The patent of this invention applies widely distributed surveillance cameras to sea fog monitoring as one of the sources of sea fog data. First, multi-source sea fog data (such as video, wind speed, wind direction, temperature, humidity, etc.) are registered through time and space registration models. Then, the multi-source data is extracted and features are fused through a shared neural network. Secondly, the sea fog is identified and the sea fog level is quantified through the sea fog recognition model and the sea fog level quantification model. Finally, the direction and speed of the sea fog can be calculated through image processing algorithms such as the optical flow method, and the sea fog situation prediction model realizes the situation prediction of the sea fog.

[0133] (3) Whether the technical solution of the present invention solves the technical problems that people have been eager to solve but have not been able to solve successfully:

[0134] The existing technical problems are as follows:

[0135] 1. The image features of sea fog are complex and easily affected by background lighting, waves, ships, buildings and other factors, making it more difficult to extract fog features.

[0136] 2. The dynamic changes of sea fog are easily affected by various meteorological factors such as weather, temperature, humidity, wind speed, etc., and the accuracy of predicting its dynamic situation is still highly challenging.

[0137] 3. Currently, sea fog detection relies solely on a single observation method, which is difficult to meet the needs of real-time, high-precision, and multi-temporal and spatial scale monitoring and prediction of sea fog.

[0138] The patented invention directly observes sea fog through widely distributed surveillance cameras, captures the development trend of sea fog, uses deep learning technology combined with rich multi-source data to enhance the prediction effect, and achieves high-precision sea fog identification, level quantification and situation prediction, meeting the needs of high-precision, multi-temporal and spatial scale monitoring and prediction.

[0139] (4) Whether the technical solution of the present invention overcomes technical prejudice:

[0140] The technical solution of the present invention overcomes technical prejudice: the patent of the present invention uses multi-source data from multiple observation methods as data drive, breaking the current technical prejudice that sea fog monitoring relies on only a single observation method; the patent of the present invention uses surveillance camera video as one of the data, and uses deep learning technology to realize sea fog recognition, situation prediction and other technologies, breaking the traditional technical prejudice of realizing sea fog monitoring through meteorological data, satellite remote sensing, and radar. BRIEF DESCRIPTION OF THE DRAWINGS

[0141] Figure 1 It is a flow chart of a method for sea fog recognition and situation prediction of surveillance videos using multi-source data fusion provided by an embodiment of the present invention.

[0142] Figure 2 It is a flow chart of a video and multi-source observation data registration method provided by an embodiment of the present invention.

[0143] Figure 3 It is a structural block diagram of a surveillance video sea fog recognition and situation prediction system with multi-source data fusion provided by an embodiment of the present invention.

[0144] Figure 4 This is a diagram of the LSTM-Transformer hybrid model architecture provided by an embodiment of the present invention.

[0145] Figure 5 It is the image where the sea fog recognition model judges that there is no sea fog.

[0146] Figure 6 It is the image where the sea fog recognition model judges that there is sea fog.

[0147] Figure 7 This is the image predicted by the sea fog level extension model DETAILED DESCRIPTION

[0148] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0149] like Figure 1 As shown, a method for monitoring video sea fog recognition and situation prediction based on multi-source data fusion provided by an embodiment of the present invention includes the following steps:

[0150] S101, video data preprocessing;

[0151] The dark channel method is used to pre-process the surveillance video; the contrast of the image is enhanced, the low-brightness background is removed, and the details of the haze are highlighted;

[0152] Image color space conversion: First, the video frame image is converted from the RGB color space to the Lab color space. In the Lab space, brightness and chromaticity are separated to process haze-related features.

[0153] Then, the dark channel value I of each pixel in each frame image is calculated by the dark channel method. dark , using the formula:

[0154]

[0155] Among them, I c Represents the pixel value of the image in color channel c, and min represents the minimum value;

[0156] S102, video and multi-source observation data registration;

[0157] S103, Time series modeling and sea fog situation prediction.

[0158] like Figure 2 As shown, the video and multi-source observation data registration method provided by the embodiment of the present invention is:

[0159] S201, temporal registration model:

[0160] Timestamp alignment: Compare the timestamps of all data sources. If there are time differences in the data, use interpolation methods to adjust the inconsistent multi-source data. For video data, segment the video based on the frame rate to ensure that the timestamp of each video frame corresponds to the time point of other data sources.

[0161] Temporal interpolation methods: If the sensor data is based on interval time, linear interpolation or more complex interpolation methods are used to fill in the missing parts between the data to match the timestamps of the video data;

[0162] S202, spatial registration model:

[0163] Multi-source data registration: unify the spatial information of different sensors or data sources, and use GIS tools or OpenCV-based registration methods to perform spatial registration;

[0164] Low-resolution preliminary registration: Starting from the lowest resolution image, calculate the transformation of the two images to obtain a rough registration result;

[0165] Layer-by-layer refinement: Continue to refine the registration results on higher resolution image layers until the resolution of the original image is reached;

[0166] Final transformation application: Apply the registration result to the original image for spatial alignment, so that the surveillance video and multi-source data can be fused and analyzed in the same coordinate system.

[0167] Time series modeling and sea fog situation prediction provided by the embodiment of the present invention:

[0168] The calculation process of the LSTM unit is as follows:

[0169] Forget gate: decides how much past memory to forget; based on the current input x t and the output h at the previous moment t-1 , the output f of the forget gate t for:

[0170] f t =σ(W f [h t-1 ,x t ]+b f )

[0171] Among them, W f is the weight matrix of the forget gate, b f is the bias term, σ is the Sigmoid activation function;

[0172] Input gate: determines how much new information to update; the output i of the input gate t for:

[0173] i t =σ(W i [h t-1 ,x t ]+b i )

[0174]

[0175] in, is the candidate state, tanh is the hyperbolic tangent activation function;

[0176] Unit state update: Combine the forget gate and input gate to update the unit state:

[0177]

[0178] Among them, C t-1 is the cell state at the previous moment, C t is the current cell state

[0179] Output gate: According to the current cell state and the state of the output gate o t Generate the output h at the current time t :

[0180] o t =σ(Wo[h t-1 ,x t ]+b0)

[0181] h t =o t *tanh(C t )

[0182] Among them, W o is the weight of the output gate, b o is the bias term;

[0183] Through the above process, LSTM can capture the temporal changes of sea fog based on the meteorological information of past image data;

[0184] Transformer enhances time series modeling;

[0185] LSTM and Transformer combined with sea fog recognition and prediction model.

[0186] Transformer enhanced time series modeling provided by the embodiment of the present invention:

[0187] The Transformer model is introduced to capture long-range dependencies in the sequence through the self-attention mechanism;

[0188] The self-attention mechanism of the Transformer model allows each time step to make a weighted combination of the inputs of all other time steps, effectively modeling the dependencies of long time series; calculation of the self-attention mechanism.

[0189] The LSTM and Transformer combined with sea fog recognition and prediction model provided in the embodiment of the present invention is as follows:

[0190] The structure of the hybrid model is as follows:

[0191] LSTM part: used to capture short-term temporal dependencies in sea fog video data;

[0192] Transformer part: Use the self-attention mechanism to enhance the modeling of long-range dependencies in long time series and adapt to the long-term trend of sea fog changes; Fusion mechanism: Fusion the outputs of LSTM and Transformer through the fully connected layer to generate the final sea fog prediction result.

[0193] Calculation of the self-attention mechanism provided by the embodiment of the present invention:

[0194] Input representation: Input sequence X = [x1, x2, ..., x T ], where T is the sequence length, x t is the input at time step t;

[0195] Calculate attention weights: For each input x t It is mapped into three vectors: query, key, and value. The query, key, and value are obtained through linear transformation:

[0196] Q=XW Q ,K=XW K ,V=XW V

[0197] Among them, W Q , W K , W V is the weight matrix, corresponding to the transformation of query, key and value respectively;

[0198] Calculating self-attention: The output of self-attention is calculated as a weighted value, where the weight is calculated by the dot product of the query and the key:

[0199]

[0200] Among them, d k is the dimension of the key vector, and the softmax function ensures that the weight sums to 1;

[0201] Multi-head attention: In order to capture different semantic information, Transformer usually uses a multi-head attention mechanism, that is, multiple independent attention calculations are performed in parallel and the results are merged:

[0202] MultiHead(Q,K,V)=Concat(head1,…,head h )W o

[0203] Among them, head is the attention head, h is the number of attention heads, and W o is the output linear transformation matrix;

[0204] Through the self-attention mechanism, Transformer can dynamically focus on the relationship between different time steps in the sequence, especially in the long-term evolution of sea fog, effectively capturing dependencies with a longer time span;

[0205] Multi-source data fusion:

[0206] Assume there are M source data Z1, Z2…, Z M , each source data is feature extracted through a shared neural network, and the obtained feature vectors are f1,f2,…,f M ; Then, the features are fused by weighted summation:

[0207]

[0208] Among them, w i is the weight of each source data, and the weight value is adjusted according to the importance of the data; the fused features are input into the LSTM-Transformer hybrid model to implement the following tasks;

[0209] Sea fog recognition model:

[0210] Based on the above LSTM-Transformer hybrid model, a soft-max layer is added to the output of the model to realize the binary judgment of the presence or absence of sea fog, thereby realizing the recognition of sea fog;

[0211] Quantitative model of sea fog level:

[0212] This model is used to classify the identified sea fog into different levels. Based on the above LSTM-Transformer hybrid model, the output of the hybrid model is mapped to different sea fog level ranges using a fully connected layer.

[0213] Calculation model of sea fog movement direction:

[0214] First, the sea fog area in the video is detected based on the LSTM-Transformer hybrid model;

[0215] Then, the optical flow algorithm is used to calculate the motion vector field between consecutive video frames to detect the movement direction of the sea fog pixels;

[0216] Secondly, the image is divided into blocks to obtain the motion vector of each area, and then the overall direction is determined by calculating the mean;

[0217] Finally, based on the motion vector, the sea fog movement angle at each moment is calculated to obtain the main movement direction;

[0218] Sea fog movement speed calculation model:

[0219] First, based on the sea fog area in the video detected by the LSTM-Transformer hybrid model, the optical flow method is used to extract speed information:

[0220] Then, the optical flow method is used to calculate the motion distance between adjacent frames to obtain the movement speed of the sea fog;

[0221] Secondly, extract the velocity components in the horizontal and vertical directions, and calculate the average velocity of each block area;

[0222] Finally, according to the area covered by the sea fog, the overall movement speed is obtained using weighted average;

[0223] Sea fog development trend prediction model:

[0224] Also based on the LSTM-Transformer hybrid model described above, the fully connected layer is used to map the target prediction value to obtain the sea fog level or extension range prediction for each time step.

[0225] like Figure 3 As shown, a multi-source data fusion surveillance video sea fog recognition and situation prediction system provided by an embodiment of the present invention includes:

[0226] The preprocessing module is used for video data preprocessing; the dark channel method is used to preprocess the surveillance video; the contrast of the image is enhanced, the low-brightness background is removed, and the details of the haze are highlighted;

[0227] Registration module, used for video and multi-source observation data registration;

[0228] Prediction module, used for time series modeling and sea fog situation prediction.

[0229] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the surveillance video sea fog recognition and situation prediction method based on multi-source data fusion.

[0230] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to execute the steps of the surveillance video sea fog recognition and situation prediction method based on multi-source data fusion.

[0231] Another object of the present invention is to provide an information data processing terminal, which is used to implement the monitoring video sea fog recognition and situation prediction system with multi-source data fusion.

[0232] The present invention is specifically implemented:

[0233] The present invention proposes a sea fog forecasting method based on surveillance camera video data, which combines image processing with deep learning and integrates multi-source observation information to achieve accurate identification and forecasting of sea fog. The core of this technical solution is to use surveillance video data, combined with image feature extraction, deep learning models and the fusion of meteorological data, to analyze and forecast the formation, change and dissipation of sea fog in real time, thereby improving the accuracy and real-time performance of sea fog warning.

[0234] 1. Video data preprocessing

[0235] The quality of sea fog video data directly affects the effect of subsequent image analysis and recognition. Since sea fog images are usually low-contrast, blurry and fast-changing, the dark channel method is used to preprocess the surveillance video in view of this feature.

[0236] The dark channel method can effectively enhance the contrast of the image, remove the low-brightness background and highlight the details of the haze.

[0237] The steps of this method are as follows:

[0238] Image color space conversion: First, convert the video frame image from RGB color space to Lab color space. In Lab space, brightness and chromaticity are separated, making it easier to process haze-related features.

[0239] Then, the dark channel value I of each pixel in each frame image is calculated by the dark channel method. dark , using the formula:

[0240]

[0241] Among them, I cIt represents the pixel value of the image in the color channel c (such as R, G, B or L, a, b channels), and min represents the minimum value. This method enhances the haze part of the image and improves the visibility of the sea fog by extracting the dark channel information in the image. After processing with the dark channel method, the contrast of the image is greatly enhanced, which is helpful for subsequent feature extraction and training of deep learning models.

[0242] 2. Video + multi-source observation data registration method

[0243] In sea fog monitoring, achieving registration (i.e. alignment and synchronization) of surveillance video and multi-source data is a key step, especially when video data and other data sources (such as weather radar, satellite images, meteorological sensor data, etc.) come from different devices, locations, and times. The following are several key steps and methods for achieving registration of surveillance video and multi-source data:

[0244] I. Temporal Registration Model:

[0245] For sea fog monitoring, the time synchronization of video data and multi-source data is the primary issue of registration. Since video data is usually dynamic, and other data sources (such as meteorological data and radar data) may be based on fixed time points, it is necessary to ensure that their timestamps are aligned.

[0246] Timestamp alignment: Compare the timestamps of all data sources. If there are time differences in the data, interpolation methods can be used to adjust the inconsistent multi-source data. For video data, the video can be segmented based on the frame rate (for example, 30 frames per second) to ensure that the timestamp of each video frame corresponds to the time point of other data sources.

[0247] Temporal interpolation methods: If the sensor data is based on interval time, linear interpolation or more complex interpolation methods (such as spline interpolation) can be used to fill in the missing parts between the data to match the timestamps of the video data.

[0248] II. Spatial Registration Model:

[0249] Spatial registration refers to the alignment of spatial coordinates between different data sources (such as video and weather radar data). The coordinates of video data are usually in the image coordinate system, while other data sources may use different coordinate systems (such as geographic coordinate system, radar coordinate system, etc.). Therefore, these data must be converted into a common coordinate system.

[0250] Corresponding control points are selected from the surveillance video, and these points need to be aligned with other data sources (such as satellite images, radar images, or locations in sensor data).

[0251] Use these control points to calculate the transformation matrix and perform the geometric transformation of the image.

[0252] Multi-source data registration: Unify the spatial information of different sensors or data sources. You can use GIS (geographic information system) tools or OpenCV-based registration methods to perform spatial registration.

[0253] In the process of sea fog monitoring, there may be image data of multiple scales. For example, the image resolution of surveillance video is low, while other data sources (such as satellite images) may have higher resolution. In order to improve the accuracy of registration, the multi-scale registration method can be used to perform gradual registration at different scales. For surveillance video, a multi-level pyramid image can be generated for each frame; for other data sources (such as radar images), pyramids are generated in the same way.

[0254] Low-resolution preliminary registration: Starting from the lowest resolution image, calculate the transformation of the two images to obtain a rough registration result.

[0255] Layer-by-layer refinement: The registration result is continuously refined on higher resolution image layers until the resolution of the original image is reached.

[0256] Final transformation application: Apply the registration result to the original image for spatial alignment, so that the surveillance video and multi-source data can be fused and analyzed in the same coordinate system.

[0257] 3. Time series modeling and sea fog situation prediction (combination of LSTM and Transformer)

[0258] The dynamic changes of sea fog involve complex spatiotemporal processes, so time series modeling is crucial for sea fog forecasting. This technical solution uses a time series modeling method that combines long short-term memory networks (LSTM) with Transformer to integrate video image features with multi-source observation information to comprehensively analyze and predict the formation, development and dissipation of sea fog.

[0259] 3.1 LSTM Modeling Principle

[0260] LSTM is an effective method for processing time series data, which can capture long-term and short-term dependencies in the data. Compared with the traditional recurrent neural network (RNN), LSTM introduces a gating mechanism, which enables it to solve the problems of gradient vanishing and long-term dependency. When processing the temporal changes of sea fog, the LSTM network can remember historical information over a long period of time and predict the changing trend of sea fog based on the input at the current moment. The basic structure of the LSTM model consists of three gates (input gate, forget gate, and output gate) and a unit state. The calculation process of the LSTM unit is as follows:

[0261] 1. Forget gate: determines how much past memory to forget. Based on the current input x t and the output h at the previous momentt-1 , the output f of the forget gate t for:

[0262] f t =σ(W f [h t-1 ,x t ]+b f )

[0263] Among them, W f is the weight matrix of the forget gate, b f is the bias term, and σ is the Sigmoid activation function.

[0264] 2 Input gate: determines how much new information to update. The output i of the input gate t for:

[0265] i t =σ(W i [h t-1 ,x t ]+b i )

[0266]

[0267] in, is a candidate state, and tanh is the hyperbolic tangent activation function.

[0268] 3. Unit state update: Combine the forget gate and input gate to update the unit state:

[0269]

[0270] Among them, C t-1 is the cell state at the previous moment, C t is the current cell state

[0271] 4. Output gate: According to the current cell state and the state of the output gate t Generate the output h at the current time t :

[0272] o t =σ(Wo[h t-1 ,x t ]+b0)

[0273] h t =o t *tanh(C t )

[0274] Among them, W o is the weight of the output gate, b o is the bias term.

[0275] Through the above process, LSTM can capture the temporal changes of sea fog based on past image data meteorological information.

[0276] 3.2 Transformer Enhanced Time Series Modeling

[0277] Although LSTM has advantages in capturing short-term temporal dependencies, its performance in long time series may be affected, especially when processing data with a long time span. In order to enhance the modeling ability of sea fog time series data, this technical solution introduces the Transformer model, the core idea of ​​which is to capture long-range dependencies in the sequence through the self-attention mechanism.

[0278] The self-attention mechanism of the Transformer model allows each time step to perform a weighted combination of the inputs of all other time steps, effectively modeling the dependencies of long time series. The calculation process of the self-attention mechanism is as follows:

[0279] 3.3 LSTM and Transformer combined with sea fog recognition and prediction model

[0280] In order to give full play to the advantages of LSTM and Transformer, the present invention proposes a LSTM-Transformer hybrid model, that is, on the basis of LSTM model capturing short-term dependencies, the self-attention mechanism of Transformer model is added to capture long-term dependencies. Combining the advantages of both, the time series prediction of sea fog can be performed more accurately.

[0281] The structure of the hybrid model is as follows:

[0282] LSTM part: used to capture short-term temporal dependencies in sea fog video data (such as instantaneous changes in sea fog).

[0283] Transformer part: Use the self-attention mechanism to enhance the modeling of long-range dependencies in long time series and adapt to the long-term trend of sea fog changes. Fusion mechanism: Fusion the outputs of LSTM and Transformer through the fully connected layer to generate the final sea fog prediction result.

[0284] 1). Input representation: Input sequence X = [x1, x2, ..., x T ], where T is the sequence length, x t is the input at time step t.

[0285] 2). Calculate attention weight: for each input x t It is mapped into three vectors: query, key, and value. The query, key, and value are obtained through linear transformation:

[0286] Q=XW Q ,K=XW K ,V=XW V

[0287] Among them, W Q , W K , W V is a weight matrix corresponding to the transformation of query, key and value respectively.

[0288] 3) Calculate self-attention: The output of self-attention is calculated as a weighted value, where the weight is calculated by the dot product of the query and the key:

[0289]

[0290] Among them, d k is the dimension of the key vector, and the softmax function ensures that the weights sum to 1.

[0291] 4) Multi-Head Attention: In order to capture different semantic information, Transformer usually uses a multi-head attention mechanism, that is, multiple independent attention calculations are performed in parallel and then the results are merged:

[0292] MultiHead(Q,K,V)=Concat(head1,…,head h )W o

[0293] Among them, head is the attention head, h is the number of attention heads, and W o is the output linear transformation matrix.

[0294] Through the self-attention mechanism, Transformer can dynamically focus on the relationship between different time steps in the sequence, especially in the long-term evolution of sea fog, effectively capturing dependencies with longer time spans.

[0295] 5). Multi-source data fusion

[0296] In order to improve the accuracy of sea fog identification and prediction, this paper introduces a multi-source observation information fusion strategy, including meteorological data (such as wind speed, temperature, humidity, etc.) and video image features. The fusion of multi-source data can provide richer information for time series modeling and improve the prediction accuracy of sea fog. In the multi-source data fusion process, a weighted summation method is used to combine data features from different sources. Specifically, assuming that there are M source data Z1, Z2…, Z M , each source data is feature extracted through a shared neural network, and the obtained feature vectors are f1,f2,…,f MThen, the features are fused by weighted summation:

[0297]

[0298] Among them, wi is the weight of each source data, and the weight value is adjusted according to the importance of the data. The fused features are input into the LSTM-Transformer hybrid model to implement the following tasks.

[0299] 6). Sea fog recognition model:

[0300] Based on the above LSTM-Transformer hybrid model, a soft-max layer is added to the output of the model to realize the binary judgment of the presence or absence of sea fog, thereby realizing the recognition of sea fog;

[0301] 7). Quantitative model of sea fog level:

[0302] The model is used to classify the identified sea fog into different levels, such as light, moderate and severe. Based on the above LSTM-Transformer hybrid model, the output of the hybrid model is mapped to different sea fog level ranges (e.g., 0-3 levels) using a fully connected layer.

[0303] 8). Calculation model of sea fog movement direction:

[0304] The goal of the model is to analyze the changes in sea fog morphology in consecutive video frames.

[0305] First, the sea fog area in the video is detected based on the LSTM-Transformer hybrid model;

[0306] Then, an optical flow algorithm (such as Farneback or optical flow method) is used to calculate the motion vector field between consecutive video frames to detect the movement direction of the sea fog pixels;

[0307] Secondly, the image is divided into blocks to obtain the motion vector of each area, and then the overall direction is determined by calculating the mean.

[0308] Finally, based on the motion vector, the sea fog movement angle at each moment is calculated to obtain the main movement direction.

[0309] 9). Calculation model of sea fog movement speed:

[0310] By analyzing the movement of sea fog in the surveillance video frame sequence, the movement speed of sea fog is calculated, providing data support for the prediction of diffusion speed and coverage range.

[0311] First, based on the sea fog area in the video detected by the LSTM-Transformer hybrid model, the optical flow method is used to extract speed information:

[0312] Then, the optical flow method is used to calculate the motion distance between adjacent frames to obtain the movement speed of the sea fog.

[0313] Secondly, the velocity components in the horizontal and vertical directions are extracted, and the average velocity of each block area is calculated.

[0314] Finally, according to the area covered by the sea fog, the weighted average is used to obtain the overall movement speed.

[0315] 10). Sea fog development trend prediction model

[0316] Also based on the LSTM-Transformer hybrid model described above, the sea fog level or extension range prediction for each time step is obtained by mapping it to the target prediction value through the fully connected layer.

[0317] Finally, the model can output the following:

[0318] Sea fog concentration prediction: The model can predict the concentration of sea fog at a certain moment.

[0319] Sea fog movement path and expansion range: predict the movement direction, speed and expansion range of sea fog in the video monitoring area.

[0320] ·Sea fog dissipation time: predict the dissipation process of sea fog and provide support for emergency decision-making.

[0321] Through the above steps, this technical solution can achieve high-precision sea fog forecast, has strong real-time and adaptability, and meets the sea fog warning needs in complex environments.

[0322] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0323] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A multi-source data fusion surveillance video sea fog recognition and situation prediction method, characterized in that: The following steps are involved: Step 1, video data preprocessing; The dark channel method is used to pre-process the surveillance video; the contrast of the image is enhanced, the low-brightness background is removed, and the details of the haze are highlighted; Image color space conversion: First, the video frame image is converted from the RGB color space to the Lab color space. In the Lab space, brightness and chromaticity are separated to process haze-related features. Then, the dark channel value I of each pixel in each frame image is calculated by the dark channel method. dark , using the formula: Among them, I c Represents the pixel value of the image in color channel c, and min represents the minimum value; Step 2: video and multi-source observation data registration; Step 3: Time series modeling and sea fog situation prediction.

2. The method for sea fog recognition and situation prediction of surveillance video using multi-source data fusion as claimed in claim 1, characterized in that: The video and multi-source observation data registration method: 1) Temporal Registration Model: Timestamp alignment: Compare the timestamps of all data sources. If there are time differences in the data, use interpolation methods to adjust the inconsistent multi-source data. For video data, segment the video based on the frame rate to ensure that the timestamp of each video frame corresponds to the time point of other data sources. Temporal interpolation methods: If the sensor data is based on interval time, linear interpolation or more complex interpolation methods are used to fill in the missing parts between the data to match the timestamps of the video data; 2) Spatial registration model: Multi-source data registration: unify the spatial information of different sensors or data sources, and use GIS tools or OpenCV-based registration methods to perform spatial registration; Low-resolution preliminary registration: Starting from the lowest resolution image, calculate the transformation of the two images to obtain a rough registration result; Layer-by-layer refinement: Continue to refine the registration results on higher resolution image layers until the resolution of the original image is reached; Final transformation application: Apply the registration result to the original image for spatial alignment, so that the surveillance video and multi-source data can be fused and analyzed in the same coordinate system.

3. The method for sea fog recognition and situation prediction of surveillance video by multi-source data fusion as claimed in claim 1, characterized in that: The time series modeling and sea fog situation prediction: The calculation process of LSTM unit is as follows: Forget gate: decides how much past memory to forget; based on the current input x t and the output h at the previous moment t-1 , the output f of the forget gate t for: f t =σ(W f [h t-1 ,x t ]+b f ) Among them, W f is the weight matrix of the forget gate, b f is the bias term, σ is the Sigmoid activation function; Input gate: determines how much new information to update; the output i of the input gate t for: i t =σ(W i [h t-1 ,x t ]+b i ) in, is the candidate state, tanh is the hyperbolic tangent activation function; Unit state update: Combine the forget gate and input gate to update the unit state: Among them, C t-1 is the cell state at the previous moment, C t is the current cell state Output gate: According to the current cell state and the state of the output gate o t Generate the output h at the current time t : O t =σ(Wh[h t-1 ,x t ]+b0) h t =o t *tanh(C t ) Among them, W o is the weight of the output gate, b o is the bias term; Through the above process, LSTM can capture the temporal changes of sea fog based on the meteorological information of past image data; Transformer enhances time series modeling; LSTM and Transformer combined with sea fog recognition and prediction model.

4. The method for sea fog recognition and situation prediction of surveillance video by multi-source data fusion as claimed in claim 3 is characterized in that: The Transformer enhances temporal modeling: The Transformer model is introduced to capture long-range dependencies in the sequence through the self-attention mechanism; The self-attention mechanism of the Transformer model allows each time step to make a weighted combination of the inputs of all other time steps, effectively modeling the dependencies of long time series; calculation of the self-attention mechanism.

5. The method for sea fog recognition and situation prediction of surveillance video by multi-source data fusion as claimed in claim 3, characterized in that: The LSTM and Transformer combined with sea fog recognition and prediction model: The structure of the hybrid model is as follows: LSTM part: used to capture short-term temporal dependencies in sea fog video data; Transformer part: Use the self-attention mechanism to enhance the modeling of long-range dependencies in long time series and adapt to the long-term trend of sea fog changes; Fusion mechanism: Fusion the outputs of LSTM and Transformer through the fully connected layer to generate the final sea fog prediction result.

6. The method for sea fog recognition and situation prediction of surveillance video by multi-source data fusion as claimed in claim 4, characterized in that: Calculation of the self-attention mechanism: Input representation: Input sequence X = [x1, x2, ..., x T ], where T is the sequence length, x t is the input at time step t; Calculate attention weights: For each input x t It is mapped into three vectors: query, key, and value. The query, key, and value are obtained through linear transformation: Q=XW Q ,K=XW K ,V=XW V Among them, W Q , W K , W V is a weight matrix corresponding to the transformation of query, key and value respectively; Calculating self-attention: The output of self-attention is calculated as a weighted value, where the weight is calculated by the dot product of the query and the key: Among them, d k is the dimension of the key vector, and the softmax function ensures that the weight sums to 1; MultiHead Attention: In order to capture different semantic information, Transformer usually uses a multi-head attention mechanism, that is, multiple independent attention calculations are performed in parallel and the results are merged: MultiHead(Q,K,V)=Concat(head1,…,head h )W o Among them, head is the attention head, h is the number of attention heads, and W o is the output linear transformation matrix; Through the self-attention mechanism, Transformer can dynamically focus on the relationship between different time steps in the sequence, especially in the long-term evolution of sea fog, effectively capturing dependencies with a longer time span; Multi-source data fusion: Assume there are M source data Z1, Z2…, Z M , each source data is feature extracted through a shared neural network, and the obtained feature vectors are f1,f2,…,f M ; Then, the weighted summation method is used to fuse the features to obtain f fused : Among them, w i is the weight of each source data, and the weight value is adjusted according to the importance of the data; the fused features are input into the LSTM-Transformer hybrid model to implement the following tasks; Sea fog recognition model: Based on the above LSTM-Transformer hybrid model, a soft-max layer is added to the output of the model to realize the binary judgment of the presence or absence of sea fog, thereby realizing the recognition of sea fog; Quantitative model of sea fog level: This model is used to classify the identified sea fog into different levels. Based on the above LSTM-Transformer hybrid model, the output of the hybrid model is mapped to different sea fog level ranges using a fully connected layer. Calculation model of sea fog movement direction: First, the sea fog area in the video is detected based on the LSTM-Transformer hybrid model; Then, the optical flow algorithm is used to calculate the motion vector field between consecutive video frames to detect the movement direction of the sea fog pixels; Secondly, the image is divided into blocks to obtain the motion vector of each area, and then the overall direction is determined by calculating the mean; Finally, based on the motion vector, the sea fog movement angle at each moment is calculated to obtain the main movement direction; Sea fog movement speed calculation model: First, based on the sea fog area in the video detected by the LSTM-Transformer hybrid model, the optical flow method is used to extract speed information: Then, the optical flow method is used to calculate the motion distance between adjacent frames to obtain the movement speed of the sea fog; Secondly, extract the velocity components in the horizontal and vertical directions, and calculate the average velocity of each block area; Finally, according to the area covered by the sea fog, the overall movement speed is obtained using weighted average; Sea fog development trend prediction model: Also based on the LSTM-Transformer hybrid model described above, the fully connected layer is used to map the target prediction value to obtain the sea fog level or extension range prediction for each time step.

7. A surveillance video sea fog recognition and situation prediction system using multi-source data fusion that implements the surveillance video sea fog recognition and situation prediction method using multi-source data fusion as described in any one of claims 1 to 6, characterized in that: The multi-source data fusion monitoring video sea fog recognition and situation prediction system includes: Preprocessing module, used for video data preprocessing; using dark channel method to preprocess surveillance video; enhancing image contrast, removing low-brightness background and highlighting haze details; Registration module, used for video and multi-source observation data registration; Prediction module, used for time series modeling and sea fog situation prediction.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the surveillance video sea fog recognition and situation prediction method based on multi-source data fusion as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the method for sea fog recognition and situation prediction of surveillance videos by multi-source data fusion as claimed in any one of claims 1 to 6.

10. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the monitoring video sea fog recognition and situation prediction system with multi-source data fusion as described in claim 7.

Images