Deep well casting molten aluminum leakage detection method based on multi-modal fusion

Abstract

The application provides a deep-well casting molten aluminum leakage detection method based on multi-modal fusion, and relates to the technical field of non-ferrous metal casting safety monitoring, which comprises the following steps: obtaining a historical data set; the historical data set comprises visual sample data, infrared sample data, acoustic sample data and label data; constructing a molten aluminum leakage detection model in a deep-well casting process; training the molten aluminum leakage detection model based on the historical data set to obtain a trained molten aluminum leakage detection model; obtaining visual data, infrared data and acoustic data in the deep-well casting process; inputting the visual data, the infrared data and the acoustic data into the trained molten aluminum leakage detection model to obtain a molten aluminum leakage detection result. The application can code, fuse and cross-modal interact multi-modal data of vision, infrared and sound, complement different modal information, mine deep layer association of data and realize accurate detection of molten aluminum leakage in deep-well casting.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of non-ferrous metal casting safety monitoring, in particular to a deep-well casting molten aluminum leakage detection method based on multi-modal fusion. BACKGROUND

[0002] As the core safety barrier connecting molten aluminum and water cooling system, leakage detection directly determines the safety and continuity of the casting process. In the deep-well casting process, the molten aluminum is shaped by contacting the crystallizer with high-pressure cooling water. Once a leak occurs, the molten aluminum will instantly vaporize and produce high-temperature steam, triggering a chain explosion.

[0003] However, existing detection technologies face severe challenges: the dual constraints of environmental complexity and perceptual limitations. The high-humidity environment (humidity > 90%) of the casting site causes the visible light system to miss more than 40% of the detection rate; the infrared thermal imaging is affected by the thermal conduction lag, with a delay of 7-8 seconds, and the high-temperature background noise causes a 32% false alarm rate. The more core pain point is the perception blind area of underwater leakage - the 2000-8000Hz acoustic fingerprint generated by the aluminum-water reaction is ignored by traditional acoustic monitoring.

[0004] Existing technical solutions can be summarized into three categories: the first category is manual inspection and simple tool assistance, mainly relying on workers holding infrared temperature guns or metal listening rods, relying on naked eye observation and ear listening judgment in high temperature and high humidity environment, which is greatly disturbed by environment and has very low reliability; the second category is automatic monitoring based on traditional threshold algorithm, using fixed infrared array, visible light camera or air microphone, triggering alarm through single temperature threshold, frame difference threshold or sound pressure threshold, lacking feature extraction and modal fusion, false alarm frequently occurs, and the system is often forced to be closed on site; the third category is intelligent detection based on deep learning, including single modal CNN, LSTM and later appearing multi-modal splicing or weighted fusion network, although it performs well in experimental environment, but the model size is large, synchronization is difficult, training samples are scarce, and after landing, the precision drops sharply, the delay is high, and it is still difficult to meet the explosion-proof and real-time requirements of industrial site. SUMMARY

[0005] The purpose of the present application is to provide a deep-well casting molten aluminum leakage detection method based on multi-modal fusion, which can encode, fuse and cross-modal interact with visual, infrared and acoustic multi-modal data, complement different modal information, mine deep layer association of data, and realize accurate detection of deep-well casting molten aluminum leakage.

[0006] A deep-well casting molten aluminum leakage detection method based on multi-modal fusion, comprising:

[0007] Obtain a historical data set; the historical data set includes visual sample data, infrared sample data, acoustic sample data and label data;

[0008] constructing an aluminum liquid leakage detection model in a deep well casting process; training the aluminum liquid leakage detection model based on the historical data set to obtain a trained aluminum liquid leakage detection model;

[0009] The aluminum liquid leakage detection model comprises a first detection branch, a second detection branch, a third detection branch, a fusion module, a cross-modal interaction module and an output module.

[0010] The first detection branch is used for processing the visual sample data to obtain visual features; the second detection branch is used for processing the infrared sample data to obtain infrared features; and the fusion module is used for fusing the visual features and the infrared features to obtain initial fusion features.

[0011] The third detection branch is used for processing the acoustic sample data to obtain acoustic features.

[0012] The cross-modal interaction module is used for cross-modal and cross-attention fusion of the fusion features and the acoustic features to obtain final fusion features; and the output module is used for obtaining the aluminum liquid leakage detection result according to the final fusion features.

[0013] Obtaining visual data, infrared data and acoustic data in a deep well casting process; inputting the visual data, the infrared data and the acoustic data into the trained aluminum liquid leakage detection model to obtain an aluminum liquid leakage detection result.

[0014] Optionally, the first detection branch comprises a first encoder, a first convolutional layer and a second convolutional layer.

[0015] The first encoder is used for encoding processing of the visual sample data to obtain first encoding features.

[0016] The first convolutional layer is used for convolution processing of the first encoding features to obtain first convolutional features.

[0017] The second convolutional layer is used for convolution processing of the first convolutional features to obtain the visual features.

[0018] Optionally, the second detection branch comprises a second encoder, a third convolutional layer and a fourth convolutional layer.

[0019] The second encoder is used for encoding processing of the infrared sample data to obtain second encoding features.

[0020] The third convolutional layer is used for convolution processing of the second encoding features to obtain second convolutional features.

[0021] The fourth convolutional layer is configured to perform convolutional processing on the second convolutional feature to obtain the infrared feature.

[0022] Optionally, the fusion module comprises a first fusion unit, a second fusion unit and a third fusion unit.

[0023] The first fusion unit, the second fusion unit and the third fusion unit are of the same structure.

[0024] The first fusion unit comprises a convolutional structure, a fusion structure, a residual connection and layer normalization structure and a first feedforward neural network structure.

[0025] The convolutional structure comprises a first input convolutional layer and a second input convolutional layer; the first convolutional structure is configured to perform convolutional processing on the first encoded feature to obtain a third convolutional feature; and the second input convolutional layer is configured to perform convolutional processing on the second convolutional feature to obtain a fourth convolutional feature.

[0026] The fusion structure is configured to perform element-wise addition on the third convolutional feature and the fourth convolutional feature to obtain an initial first fusion feature; the residual connection and layer normalization structure is configured to perform residual connection and layer normalization processing on the initial first fusion feature to obtain a normalized feature.

[0027] The first feedforward neural network structure is configured to perform nonlinear transformation on the normalized feature to obtain a first fusion feature.

[0028] The second fusion unit is configured to fuse the first fusion feature, the first convolutional feature and the second convolutional feature to obtain a second fusion feature.

[0029] The third fusion unit is configured to fuse the second fusion feature, the visual feature and the infrared feature to obtain the initial fusion feature.

[0030] Optionally, the third detection branch comprises a third encoder, a fifth convolutional layer and a multi-scale feature extraction layer.

[0031] The third encoder is configured to perform encoding processing on the acoustic sample data to obtain a third encoded feature; and the fifth convolutional layer is configured to perform convolutional processing on the third encoded feature to obtain a fifth convolutional feature.

[0032] The multi-scale feature extraction layer is configured to perform multi-scale and multi-dimensional fusion on the fifth convolutional feature to obtain the acoustic feature.

[0033] Optionally, the multi-scale feature extraction layer comprises a sixth convolutional layer, a seventh convolutional layer, an eighth convolutional layer, a ninth convolutional layer, a tenth convolutional layer, an eleventh convolutional layer, a twelfth convolutional layer, a thirteenth convolutional layer and a merging layer.

[0034] The sixth convolutional layer is configured to perform dimension adjustment on the fifth convolutional feature to obtain a first dimension feature; the seventh convolutional layer is configured to perform feature extraction on the first dimension feature to obtain a first extracted feature; and the eighth convolutional layer is configured to perform dimension reduction on the first extracted feature to obtain a second dimension feature.

[0035] The ninth convolutional layer is configured to split the fifth convolutional feature to obtain a first branch feature, a second branch feature, a third branch feature and a fourth branch feature.

[0036] The tenth convolutional layer is configured to perform feature extraction on the second branch feature to obtain a second extracted feature; the eleventh convolutional layer is configured to perform feature extraction on the third branch feature to obtain a third extracted feature; and the twelfth convolutional layer is configured to perform feature extraction on the fourth branch feature to obtain a fourth extracted feature.

[0037] The thirteenth convolutional layer is configured to aggregate the first branch feature, the second extracted feature, the third extracted feature and the fourth extracted feature to obtain an aggregated feature.

[0038] The merging layer is configured to add and merge the aggregated feature and the second dimension feature to obtain the acoustic feature.

[0039] Optionally, the cross-modal interaction module comprises a preprocessing structure, a cross-attention structure, a second feedforward neural network structure and a fusion structure.

[0040] The preprocessing structure is configured to perform dimension matching and feature mapping on the acoustic feature and the initial fusion feature to obtain a first mapped feature and a second mapped feature.

[0041] The cross-attention structure is configured to interact the first mapped feature and the second mapped feature based on different attention weights to obtain a plurality of interaction features.

[0042] The second feedforward neural network structure is configured to perform nonlinear transformation and dimension adjustment on each of the interaction features to obtain a plurality of initial features.

[0043] The fusion structure is configured to perform residual connection and fusion on each of the initial features to obtain the fusion feature.

[0044] Optionally, the convolution kernels of the sixth convolutional layer, the eighth convolutional layer, the ninth convolutional layer and the thirteenth convolutional layer are all 1x1; and the convolution kernels of the seventh convolutional layer, the ninth convolutional layer, the tenth convolutional layer, the eleventh convolutional layer and the twelfth convolutional layer are all 3x3.

[0045] Effects of the present application are as follows:

[0046] The application is based on a deep-well casting molten aluminum leakage detection method based on multi-modal fusion, which can effectively extract and fuse the information of multi-channel signals through a molten aluminum leakage detection model, model the correlation between intrinsic factors and external factor data values, automatically extract high-level semantic features strongly related to the target task, and realize accurate deep-well casting molten aluminum leakage detection.

[0047] The application is based on a deep-well casting molten aluminum leakage detection method based on multi-modal fusion, which can effectively extract and fuse the information of multi-channel signals through a molten aluminum leakage detection model, model the correlation between intrinsic factors and external factor data values, automatically extract high-level semantic features strongly related to the target task, and realize accurate deep-well casting molten aluminum leakage detection. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is a flow chart of the deep-well casting molten aluminum leakage detection method based on multi-modal fusion of the application;

[0049] Figure 2 is a schematic diagram of the molten aluminum leakage detection model structure of the application;

[0050] Figure 3 is a schematic diagram of the first fusion unit structure of the application;

[0051] Figure 4 is a schematic diagram of the multi-scale feature extraction layer structure of the application.

[0052] In the figure: 1, first detection branch; 2, second detection branch; 3, third detection branch; 4, fusion module; 5, cross-modal interaction module; 6, output module; 11, first encoder; 12, first convolutional layer; 13, second convolutional layer; 21, second encoder; 22, third convolutional layer; 23, fourth convolutional layer; 31, third encoder; 32, fifth convolutional layer; 33, multi-scale feature extraction layer; 41, first fusion unit; 42, second fusion unit; 43, third fusion unit; 51, preprocessing structure; 52, cross-attention structure; 53, second feedforward neural network structure; 54, fusion structure; 331, sixth convolutional layer; 332, seventh convolutional layer; 333, eighth convolutional layer; 334, ninth convolutional layer; 335, tenth convolutional layer; 336, eleventh convolutional layer; 337, twelfth convolutional layer; 338, thirteenth convolutional layer; 339, merging layer; 411, convolutional structure; 412, fusion layer; 413, residual connection and layer normalization structure; 414, first feedforward neural network structure; 521, first cross-attention layer; 522, second cross-attention layer; 531, first feedforward neural network layer; 532, second feedforward neural network layer; 533, third feedforward neural network layer; 534, fourth feedforward neural network layer. Detailed Implementation

[0053] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0054] Figure 1 This is a flowchart of the deep well casting aluminum liquid leakage detection method based on multimodal fusion of the present invention, as shown below. Figure 1 As shown, this invention provides a method for detecting leakage in deep-well cast aluminum liquid based on multimodal fusion, which includes:

[0055] S1, Obtain the historical dataset. The historical dataset includes visual sample data, infrared sample data, acoustic sample data, and label data.

[0056] S2, Construct an aluminum liquid leakage detection model during the deep well casting process; Train the aluminum liquid leakage detection model based on historical datasets to obtain a trained aluminum liquid leakage detection model.

[0057] During the training phase, mean squared error loss is used as the optimization function. After forward propagation to obtain the output of each branch of the model, the gradient is calculated based on the mean squared error loss, and feedback training is performed until the model converges.

[0058] Specifically, such as Figure 2 As shown, the aluminum liquid leakage detection model includes a first detection branch 1, a second detection branch 2, a third detection branch 3, a fusion module 4, a cross-modal interaction module 5, and an output module 6.

[0059] The first detection branch 1 is used to process visual sample data to obtain visual features; the second detection branch 2 is used to process infrared sample data to obtain infrared features; the fusion module 4 is used to fuse visual features and infrared features to obtain initial fused features.

[0060] Furthermore, the first detection branch 1 includes a first encoder 11, a first convolutional layer 12, and a second convolutional layer 13.

[0061] The first encoder 11 is used to encode the visual sample data to obtain the first encoded feature.

[0062] The first convolutional layer 12 is used to perform convolution processing on the first encoded feature to obtain the first convolutional feature.

[0063] The second convolutional layer 13 is used to perform convolution processing on the first convolutional features to obtain visual features.

[0064] The second detection branch 2 includes a second encoder 21, a third convolutional layer 22, and a fourth convolutional layer 23.

[0065] The second encoder 21 is used to encode the infrared sample data to obtain the second encoded feature.

[0066] The third convolutional layer 22 is configured to perform convolutional processing on the second encoded feature to obtain a second convolutional feature.

[0067] The fourth convolutional layer 23 is configured to perform convolutional processing on the second convolutional feature to obtain an infrared feature.

[0068] The fusion module 4 includes a first fusion unit 41, a second fusion unit 42, and a third fusion unit 43.

[0069] The first fusion unit 41, the second fusion unit 42, and the third fusion unit 43 have the same structure.

[0070] As shown in FIG. 4, the first fusion unit 41 includes a convolutional structure 411, a fusion structure 412, a residual connection and layer normalization structure 413 Add&Norm, and a first feedforward neural network structure 414. FFNs in the figure represent feedforward neural networks. Figure 3 The convolutional structure 411 includes a first input convolutional layer and a second input convolutional layer; the first convolutional structure 411 is configured to perform convolutional processing on the first encoded feature to obtain a third convolutional feature; and the second input convolutional layer is configured to perform convolutional processing on the second convolutional feature to obtain a fourth convolutional feature.

[0071] The fusion layer 412 is configured to perform element-wise addition on the third convolutional feature and the fourth convolutional feature to obtain an initial first fusion feature; and the residual connection and layer normalization structure 413 is configured to perform residual connection and layer normalization processing on the initial first fusion feature to obtain a normalized feature. The differential convolution and multi-link processing allow the features in complex scenes to interact more fully and be expressed more accurately.

[0072] The first feedforward neural network structure 414 is configured to perform nonlinear transformation on the normalized feature to obtain a first fusion feature.

[0073] The second fusion unit 42 is configured to fuse the first fusion feature, the first convolutional feature, and the second convolutional feature to obtain a second fusion feature.

[0074] The third fusion unit 43 is configured to fuse the second fusion feature, the visual feature, and the infrared feature to obtain an initial fusion feature.

[0075] Specifically, the convolutional structure 411 in the second fusion unit 42 includes three input convolutional layers corresponding to the first convolutional feature, the second convolutional feature, and the first fusion feature, respectively; and the convolutional structure 411 in the third fusion unit 43 includes three input convolutional layers corresponding to the second fusion feature, the visual feature, and the infrared feature, respectively.

[0076] The third detection branch 3 is configured to process the acoustic sample data to obtain an acoustic feature.

[0077] The third detection branch 3 is configured to process the acoustic sample data to obtain an acoustic feature.

[0078] Furthermore, the third detection branch 3 includes a third encoder 31, a fifth convolutional layer 32, and a multi-scale feature extraction layer 33.

[0079] The third encoder 31 is used to encode the acoustic sample data to obtain the third encoded feature; the fifth convolutional layer 32 is used to convolve the third encoded feature to obtain the fifth convolutional feature.

[0080] The multi-scale feature extraction layer 33 is used to perform multi-scale and multi-dimensional fusion of the fifth convolutional features to obtain acoustic features.

[0081] like Figure 4 As shown, the multi-scale feature extraction layer 33 includes a sixth convolutional layer 331, a seventh convolutional layer 332, an eighth convolutional layer 333, a ninth convolutional layer 334, a tenth convolutional layer 335, an eleventh convolutional layer 336, a twelfth convolutional layer 337, a thirteenth convolutional layer 338, and a merging layer 339.

[0082] The sixth convolutional layer 331 is used to adjust the dimensions of the fifth convolutional features to obtain the first dimension features; the seventh convolutional layer 332 is used to extract features from the first dimension features to obtain the first extracted features; the eighth convolutional layer 333 is used to restore the dimensions of the first extracted features to obtain the second dimension features.

[0083] The ninth convolutional layer 334 is used to split the fifth convolutional features to obtain the first branch features, the second branch features, the third branch features, and the fourth branch features.

[0084] The tenth convolutional layer 335 is used to extract features from the second branch features to obtain the second extracted features; the eleventh convolutional layer 336 is used to extract features from the third branch features to obtain the third extracted features; and the twelfth convolutional layer 337 is used to extract features from the fourth branch features to obtain the fourth extracted features.

[0085] The thirteenth convolutional layer 338 is used to aggregate the first branch features, the second extracted features, the third extracted features, and the fourth extracted features to obtain aggregated features.

[0086] Preferably, the convolution kernels of the sixth convolutional layer 331, the eighth convolutional layer 333, the ninth convolutional layer 334, and the thirteenth convolutional layer 338 are all 1×1; the convolution kernels of the seventh convolutional layer 332, the ninth convolutional layer 334, the tenth convolutional layer 335, the eleventh convolutional layer 336, and the twelfth convolutional layer 337 are all 3×3.

[0087] The merging layer 339 is used to add and merge the aggregated features and the second-dimensional features to obtain the acoustic features.

[0088] The multi-scale feature extraction layer 33 mines acoustic data features at different scales (such as different receptive fields and resolutions), integrates multi-dimensional information, and finally outputs acoustic features to provide rich acoustic information for cross-modal fusion.

[0089] The seventh convolutional layer 332, the ninth convolutional layer 334, the tenth convolutional layer 335, the eleventh convolutional layer 336, and the twelfth convolutional layer 337 are used to capture core spatial features. By using small convolution kernels to superimpose equivalent large receptive fields, different scale spatial information is obtained.

[0090] Overall, the main branch quickly completes basic multi-scale extraction, and the parallel branch mines fine-grained and diversified multi-scale information. After parallel operation, the implicit features are merged through an addition mode, so that the output covers the composite features of "basic large scale + deep fine-grained multi-scale". This design can meet the demand for multi-scale features in complex scenarios (such as underwater acoustics and visual-infrared cross-modal fusion), and provide rich and differentiated inputs for lower-level tasks. Through the combination of 1x1 and 3x3 convolution, multi-scale information is efficiently captured, and the feature expression capability is strengthened to support accurate feature fusion and interaction.

[0091] The cross-modal interaction module 5 is used for cross-modal and cross-attention fusion of the fusion features and the acoustic features to obtain final fusion features; and the output module 6 is used for obtaining the molten aluminum leakage detection result according to the final fusion features.

[0092] Specifically, the cross-modal interaction module 5 includes a preprocessing structure 51, a cross-attention structure 52, a second feedforward neural network structure 53, and a fusion structure 54.

[0093] The preprocessing structure 51 is used for dimension matching and feature mapping of the acoustic features and the initial fusion features to obtain first mapping features and second mapping features. After adapting the two types of features, subsequent cross-attention operations are ensured to be smoothly carried out.

[0094] The cross-attention structure 52 is used for interaction of the first mapping features and the second mapping features based on different attention weights to obtain a plurality of interaction features. This mechanism calculates the attention weights between the first mapping features and the second mapping features, focuses on the key associated information between the modalities, strengthens the important feature interaction, deeply associates the visual-infrared features and the acoustic features, and outputs the cross-modal features after preliminary interaction.

[0095] The second feedforward neural network structure 53 is used for nonlinear transformation and dimension adjustment of each interaction feature to obtain a plurality of initial features.

[0096] The fusion structure 54 is used for residual connection and fusion of each initial feature to obtain fusion features.

[0097] Specifically, the cross-attention structure 52 comprises a first cross-attention layer 521 and a second cross-attention layer 522; the second feedforward neural network structure 53 comprises a first feedforward neural network layer 531, a second feedforward neural network layer 532, a third feedforward neural network layer 533 and a fourth feedforward neural network layer 534.

[0098] The first cross-attention layer 521 interacts the first mapped feature and the second mapped feature based on different attention weights to obtain a plurality of first interaction features; the second cross-attention layer 522 interacts the first mapped feature and the second mapped feature based on different attention weights to obtain a plurality of second interaction features.

[0099] The first feedforward neural network layer 531 and the second feedforward neural network layer 532 perform nonlinear transformation and dimension adjustment on each first interaction feature to obtain a plurality of first initial features; the third feedforward neural network layer 533 and the fourth feedforward neural network layer 534 perform nonlinear transformation and dimension adjustment on each second interaction feature to obtain a plurality of second initial features. The fusion structure 54 is used for residual connection and fusion of each first initial feature and each second initial feature to obtain a fusion feature.

[0100] S3, acquiring visual data, infrared data and acoustic data in the deep well casting process; inputting the visual data, the infrared data and the acoustic data into the trained molten aluminum leakage detection model to obtain a molten aluminum leakage detection result.

[0101] Preferably, the visual data is acquired based on a camera acquisition mold plate, also known as a distribution plate or a distribution plate; the infrared data is acquired based on an infrared thermal imager; the acoustic data is acquired based on an underwater sound pickup array, which comprises a plurality of sound pickups.

[0102] The above-described embodiments are only preferred embodiments of the present application and do not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements to the technical solutions of the present application made by those skilled in the art shall fall within the protection scope of the present application as defined by the claims.

Claims

1. A deep-well casting molten aluminum leakage detection method based on multi-modal fusion, characterized in that, It comprises: acquiring a historical data set; the historical data set comprises visual sample data, infrared sample data, acoustic sample data and label data; constructing an aluminum liquid leakage detection model in a deep well casting process; training the aluminum liquid leakage detection model based on the historical data set to obtain a trained aluminum liquid leakage detection model; The aluminum liquid leakage detection model comprises a first detection branch, a second detection branch, a third detection branch, a fusion module, a cross-modal interaction module and an output module; The first detection branch is used for processing the visual sample data to obtain visual features; The second detection branch is used for processing the infrared sample data to obtain infrared features; The fusion module is used for fusing the visual features and the infrared features to obtain initial fusion features; The third detection branch is used for processing the acoustic sample data to obtain acoustic features; The cross-modal interaction module is used for cross-modal and cross-attention fusion of the initial fusion features and the acoustic features to obtain final fusion features; The output module is used for obtaining aluminum liquid leakage detection results according to the final fusion features; acquiring visual data, infrared data and acoustic data in a deep well casting process; inputting the visual data, the infrared data and the acoustic data into the trained aluminum liquid leakage detection model to obtain aluminum liquid leakage detection results; The first detection branch comprises a first encoder, a first convolutional layer and a second convolutional layer; The first encoder is used for encoding processing of the visual sample data to obtain first encoding features; The first convolutional layer is used for convolution processing of the first encoding features to obtain first convolutional features; The second convolutional layer is used for convolution processing of the first convolutional features to obtain the visual features; The second detection branch comprises a second encoder, a third convolutional layer and a fourth convolutional layer; The second encoder is used for encoding processing of the infrared sample data to obtain second encoding features; The third convolutional layer is used for convolution processing of the second encoding features to obtain second convolutional features; The fourth convolutional layer is used for convolution processing of the second convolutional features to obtain the infrared features; The fusion module comprises a first fusion unit, a second fusion unit and a third fusion unit; The first fusion unit, the second fusion unit and the third fusion unit have the same structure; The first fusion unit comprises a convolution structure, a fusion structure, a residual connection and a layer normalization structure, and a first feedforward neural network structure; The convolution structure comprises a first input convolutional layer and a second input convolutional layer; the convolution structure is used for convolution processing of the first encoding features to obtain third convolutional features; the second input convolutional layer is used for convolution processing of the second convolutional features to obtain fourth convolutional features; The fusion structure is used for element-wise addition of the third convolutional features and the fourth convolutional features to obtain initial first fusion features; the residual connection and the layer normalization structure are used for residual connection and layer normalization processing of the initial first fusion features to obtain normalized features; The first feedforward neural network structure is configured to perform nonlinear transformation on the normalized features to obtain first fused features; The second fusion unit is configured to fuse the first fused features, the first convolutional features and the second convolutional features to obtain second fused features; The third fusion unit is configured to fuse the second fused features, the visual features and the infrared features to obtain the initial fused features.

2. The deep-well casting molten aluminum leakage detection method based on multi-modal fusion according to claim 1, characterized in that, The third detection branch includes a third encoder, a fifth convolutional layer and a multi-scale feature extraction layer; The third encoder is configured to perform encoding processing on the acoustic sample data to obtain third encoded features; and the fifth convolutional layer is configured to perform convolutional processing on the third encoded features to obtain fifth convolutional features; The multi-scale feature extraction layer is configured to perform multi-scale and multi-dimensional fusion on the fifth convolutional features to obtain the acoustic features.

3. The deep-well casting molten aluminum leakage detection method based on multi-modal fusion according to claim 2, characterized in that, The multi-scale feature extraction layer includes a sixth convolutional layer, a seventh convolutional layer, an eighth convolutional layer, a ninth convolutional layer, a tenth convolutional layer, an eleventh convolutional layer, a twelfth convolutional layer, a thirteenth convolutional layer and a merging layer; The sixth convolutional layer is configured to perform dimensional adjustment on the fifth convolutional features to obtain first dimensional features; the seventh convolutional layer is configured to perform feature extraction on the first dimensional features to obtain first extracted features; and the eighth convolutional layer is configured to perform dimensional reduction on the first extracted features to obtain second dimensional features; The ninth convolutional layer is configured to split the fifth convolutional features to obtain first branch features, second branch features, third branch features and fourth branch features; The tenth convolutional layer is configured to perform feature extraction on the second branch features to obtain second extracted features; the eleventh convolutional layer is configured to perform feature extraction on the third branch features to obtain third extracted features; and the twelfth convolutional layer is configured to perform feature extraction on the fourth branch features to obtain fourth extracted features; The thirteenth convolutional layer is configured to aggregate the first branch features, the second extracted features, the third extracted features and the fourth extracted features to obtain aggregated features; The merging layer is configured to add and merge the aggregated features and the second dimensional features to obtain the acoustic features.

4. The deep-well casting molten aluminum leakage detection method based on multi-modal fusion according to claim 1, characterized in that, The cross-modal interaction module includes a preprocessing structure, a cross-attention structure, a second feedforward neural network structure and a fusion structure; The preprocessing structure is configured to perform dimensional matching and feature mapping on the acoustic features and the initial fused features to obtain first mapped features and second mapped features; The cross-attention structure is configured to interact the first mapped features and the second mapped features based on different attention weights to obtain a plurality of interaction features; The second feedforward neural network structure is configured to perform nonlinear transformation and dimensional adjustment on each of the interaction features to obtain a plurality of initial features; The fusion structure is configured to perform residual connection and fusion on each of the initial features to obtain the fused features.

5. The deep-well casting molten aluminum leakage detection method based on multi-modal fusion according to claim 3, characterized in that, The convolution kernel of the sixth convolution layer, the eighth convolution layer, the ninth convolution layer and the thirteenth convolution layer is 1x1; the convolution kernel of the seventh convolution layer, the tenth convolution layer, the eleventh convolution layer and the twelfth convolution layer is 3x3.

Images