An audio-visual target segmentation method based on improved Swin Transformer

Abstract

The application discloses an audio-visual target segmentation method based on an improved Swin Transformer, and relates to the technical field of speech recognition. The improved Swin Transformer is a deep neural network based on a sliding window multi-head self-attention transformation mechanism, which enhances the global modeling capability of the feature extraction network and the acquisition efficiency of long-distance semantic information, and improves the bottom feature extraction capability and the abstract effect of high-level semantic information. The application applies a speech encoder and a visual encoder to map speech features and visual features to a unified semantic space, and then performs semantic association and feature alignment between modes through an AFI module. The application proposes multi-modal fusion, that is, the combination of speech recognition and video processing, which can realize multi-modal information fusion, thereby improving the overall system performance and robustness. Not only can rich context information be provided, but also the precision and real-time processing capability of the system can be improved, thereby providing a more optimal solution for various application scenarios.

Description

Technical Field

[0001] This invention relates to the field of audiovisual target segmentation methods, and in particular to an audiovisual target recognition method based on an improved SwinTransformer segmentation network. Background Technology

[0002] With the development of digital vision technology, images in various fields are evolving towards large data volumes and diverse types of information. Semantic segmentation tasks have great application value in many fields such as land and resources surveys, geographic information mapping, autonomous driving navigation, and medical image detection. Accurately and effectively identifying precise detailed features from images while extracting accurate semantic information and balancing information types is a practical need.

[0003] Video semantic segmentation is an important research direction in the fields of audiovisual object segmentation methods and computer vision. Its purpose is to classify each frame of a video at the pixel level in order to distinguish different objects from the background.

[0004] Speech recognition plays a crucial role in video semantic segmentation. By capturing and analyzing audio information in videos, speech recognition technology can provide additional contextual clues, improving segmentation accuracy. For example, in traffic monitoring scenarios, speech recognition can help detect sounds such as sirens and horns, assisting in object recognition in video images, thereby improving the speed and accuracy of emergency response. Furthermore, speech recognition can provide additional contextual information; by combining audio information from the video, it can provide more environmental background, helping to more accurately segment and identify objects in the video. It also enhances scene understanding because audio signals can reflect dynamic changes in the environment, such as conversations and background music, helping the system better understand the scene in the video. Moreover, it improves real-time processing capabilities; speech recognition technology can quickly capture and process audio information, providing immediate feedback and enhancing the system's real-time processing capabilities. Therefore, this invention proposes an audiovisual object segmentation method based on an improved Swing Transformer, applying Swing Transformer to extract video frame features to complete the video semantic segmentation task within the technical field of audiovisual object segmentation methods. Summary of the Invention

[0005] The purpose of this invention is to propose an audiovisual target segmentation method based on an improved Swin Transformer to solve the following problems existing in the prior art:

[0006] (1) Complex video scenes contain both relevant and irrelevant features. Extracting semantics from images based on a single modality requires a lot of computing power, making it difficult to achieve online real-time video semantic segmentation.

[0007] (2) Alignment of speech recognition and video frame image features based on multimodal fusion;

[0008] (3) Convolutional neural networks have the problem of balancing large-scale and small-scale features, which ultimately affects the accuracy of complex target recognition tasks. The imbalance between low-level features and high-level semantic information of images leads to inaccurate local localization and segmentation, and the problem of ignoring detailed features.

[0009] (4) The problem of imbalance in classification and recognition caused by insufficient exchange of contextual information and processing of large sample data in the global image.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] An audiovisual target recognition method based on an improved Swing Transformer segmentation network, wherein the improved Swing Transformer segmentation network includes:

[0012] Based on the end-to-end audiovisual pixel-level segmentation module, semantic association and feature alignment between modalities are performed through the AFI module;

[0013] A sliding window self-attention transformation module based on residual connections is used to enhance the network model's ability to recognize objects of different sizes in video frames and to enhance the correlation of global long-distance semantic information space.

[0014] A size adaptive module is used to address the fixed limitations of the sliding window self-attention mechanism;

[0015] Symmetrical transform encoder-decoder structure network based on hierarchical deep sliding window self-attention mechanism;

[0016] Multiple skip connections are used to directly connect corresponding layers of the encoder and decoder to preserve and pass high-resolution feature information;

[0017] A fusion loss function is used to simultaneously improve recognition accuracy and classification balance;

[0018] The method includes the following steps:

[0019] S1. Input the video frame feature information into the image size adaptive module to extract image feature information;

[0020] S2. Input the audio information into the convolution module to extract audio feature information;

[0021] S3. Input the image processed by S1 into a series of sliding window self-attention transformation modules to further extract and process features, and then use the image size adaptive module to restore the size of the output feature map.

[0022] S4. Use a hierarchical encoder to generate feature maps of different depths through downsampling at each stage, and record the features in each encoding stage as well as the original image resolution features.

[0023] S5. Input the audio feature information and image feature information obtained in S2 and S4 into the AFI module for feature alignment. Introduce audio feature information to guide segmentation in order to establish the relationship between temporal audio and video frame pixels. The video frame features after passing through the AFI module are expected to be enhanced in the region that matches the audio.

[0024] S6. Input the image feature information output in S4 into the decoder, upsample the image, and restore it to the feature map of the input size.

[0025] S7. Use feature fusion technology to fuse the audio sampling features fused by the AFI module in S5 and the features upsampled after decoding in S6;

[0026] S8. After the fusion is completed, the fused features are processed using the operations described in S1 and S3.

[0027] S9. Input the feature map output from S8 into the segmentation head, classify each pixel into a predetermined category, obtain the semantic segmentation and recognition result, calculate the recognition result using the fusion loss function, and improve the network parameters through backpropagation.

[0028] Preferably, S1 specifically includes the following:

[0029] S1.1 Input the video frames into the convolution module to initially extract image features;

[0030] S1.2 Input the image features into the image size adaptive module to make them conform to the fixed size requirements of the sliding window self-attention transformation module;

[0031] S1.3. Feature extraction and downsampling are performed using the sliding window self-attention transformation module.

[0032] Preferably, S2 specifically includes the following:

[0033] S2.1 Extract audio independently using an audio encoder, given an audio segment s1;

[0034] S2.2. Process the audio segment s1 given in S1 into a spectrum using short-time Fourier transform;

[0035] S2.3. Input the spectrogram into the convolutional neural network VGGish, and extract audio features A using pre-trained weights from the AudioSet dataset. The function is expressed as:

[0036] A=T×d

[0037] Where T represents the time in seconds, and d represents the feature dimension;

[0038] S2.4. The audio features A obtained in S2.3 are transformed into a feature space with the same dimension as the visual feature mapping Fi through a linear layer;

[0039] S2.5. Copy the converted audio features in the feature space. h i × w i Next, and reshape it to the same size as Fi, denoted as . ; h i × w i This indicates the size of the image embedding feature map.

[0040] Preferably, S3 specifically includes the following:

[0041] S3.1. Divide the image into multiple small blocks, flatten the small blocks and convert them into fixed-dimensional vectors through a linear layer, and add positional information to the feature vector of each small block to ensure that the model understands the spatial relationship between different blocks;

[0042] S3.2 After processing in S3.1, the sequence containing positional encoding information is retained as a residual and input into the normalization layer. LN For each element of the input sequence, multi-head self-attention is calculated and added to the residual. The specific formula is as follows:

[0043]

[0044] in, Indicates the output sequence. x l-1 Represents the input sequence;

[0045] S3.3. Retain the output obtained in S3.2 as a residual and input it sequentially into the normalization layer. LN Multilayer perceptron network MLP Then add it to the residual, the specific formula is as follows:

[0046]

[0047] in, x l Indicates the output sequence;

[0048] S3.4. Retain the output sequence from S3.3 as a residual and input it into the normalization layer. LNFor each element of the input sequence, a moving window self-attention calculation is performed, which is then added to the residual. The specific formula is as follows:

[0049]

[0050] in, The output sequence is represented;

[0051] S3.5. Using the output obtained in S3.4 as the residual and input, repeat the operation described in S2.3. The specific formula is as follows:

[0052]

[0053] in, x l+1 Indicates the output sequence;

[0054] S3.6 After a series of sliding window self-attention transformation modules, the video frame feature size is restored to the input size using the Resize adaptive module, which is beneficial for the design of the hierarchical frame feature extraction structure.

[0055] Preferably, S4 specifically includes the following:

[0056] Based on the video frame feature coding network, feature downsampling is implemented through convolutional layers with a stride of 2 and a kernel size of 3×3. For each deeper layer, the width and height are half of the previous layer, and the number of channels is twice that of the previous layer, forming feature maps of four different depths: 1 / 4, 1 / 8, 1 / 16, and 1 / 32. The shallow feature maps contain detailed feature information, while the deep feature maps contain high-level semantic information.

[0057] Preferably, S5 specifically includes the following:

[0058] Visual feature maps V i and the audio characteristics of the entire video A Input the AFI module to get the first i Phase update feature map Z i Its function is expressed as:

[0059]

[0060] in, θ , ϕ , g and μ This represents a 1×1×1 convolution operation. N Represents a normalization factor. α iIndicates audio-visual similarity, where each visual pixel interacts with all audio elements through the AFI module; superscript T This indicates the transpose operation.

[0061] Preferably, step S6 specifically includes the following:

[0062] The image feature information output from S4 is upsampled through a convolutional layer with a stride of 2 and a kernel size of 3×3. Each shallower layer has twice the width and height of the previous layer and half the number of channels, thus restoring feature maps at depths of 1 / 8, 1 / 4, and 1 / 2. The restored feature maps contain both detailed feature information and high-level semantic information extracted from the deeper feature maps.

[0063] Preferably, step S7 specifically includes the following:

[0064] S7.1. Using feature fusion, the deepest 1 / 16 of the encoded feature map is upsampled to restore it to 1 / 8 of the size of the previous layer.

[0065] S7.2. The feature map output in S7.1 is fused with the corresponding 1 / 8 feature map output in S5 after alignment with audio features by the AFI module, in order to ensure the integrity and richness of information during feature transmission and improve the accuracy of segmentation and edge details.

[0066] Preferably, S9 specifically includes the following:

[0067] The semantic segmentation and recognition results are fused with Soft Cross Entrophy Loss and Lovza Loss at a weight ratio of 1:1, and backpropagation is performed to iterate the network parameters, thereby improving the accuracy of video frame segmentation and classification.

[0068] The loss function employs a 1:1 weighted fusion of Soft Cross Entropy Loss and Lovasz Loss. This fusion improves model robustness and generalization ability, reduces overfitting, provides more accurate gradient signals, and facilitates better model learning and parameter tuning. Furthermore, because Lovasz Loss is mathematically differentiable, it is easier to combine with other loss functions to further enhance model performance. Additionally, by adjusting the weights of the two loss functions during the fusion calculation, the weights of different objectives can be balanced, allowing for flexible control over the optimization level of the model across various task metrics to adapt to practical needs.

[0069] The specific formula for calculating Soft Cross Entropy Loss is as follows:

[0070]

[0071] in, L SCE This represents the value of the Soft Cross Entropy Loss function; y i The label representing the i-th category in the real labels is set as a soft label. y i ∈(0,1); P ( x i () represents the probability of the i-th class predicted by the model;

[0072] Lovasz Loss not only focuses on correctly classified samples but also considers boundary samples and misclassified samples. Therefore, it can handle problems of class imbalance and imbalance between easy and difficult samples. The specific calculation formula is as follows:

[0073]

[0074]

[0075]

[0076]

[0077] in, This represents the loss function to be optimized. Represents the actual value; c This represents the set of pixels with prediction errors. M c This represents the set of segments where the network segmentation results and labels do not match. M c The domain is {0,1} p , p Indicates the number of pixels;

[0078] against We utilize the lovasz extension for smooth extension and implement it in multi-class segmentation. f i ( c ) indicates the first c The probability value after a softmax-like operation; using a scoring function. f i ( c To construct a pixel error m i ( c )vector;

[0079] against Using errors m ( c Vector construction to replace Δ Jc The loss function;

[0080] against To optimize the evaluation of mIoU metrics across all categories, the average of the above... loss ( f ( c )).

[0081] Compared with existing technologies, this invention provides an audiovisual target recognition method based on an improved Swing Transformer segmentation network, which has the following advantages:

[0082] This invention applies a speech encoder and a visual encoder to map speech features and visual features to a unified semantic space, and then uses an AFI module to perform semantic association and feature alignment between modalities. This invention proposes multimodal fusion, which combines speech recognition with video processing, to achieve multimodal information fusion, thereby improving the overall system performance and robustness. It can not only provide rich contextual information, but also improve the system's accuracy and real-time processing capabilities, providing better solutions for various application scenarios. Attached Figure Description

[0083] Figure 1 This is a schematic diagram of an important module of the network structure mentioned in Embodiment 1 of the present invention;

[0084] Figure 2 This is the mesh structure diagram mentioned in Embodiment 1 of the present invention;

[0085] Figure 3 This is a schematic diagram of the self-attention mechanism calculation mentioned in Embodiment 1 of the present invention;

[0086] Figure 4 This is a schematic diagram of the segmentation result mentioned in Embodiment 2 of the present invention. Detailed Implementation

[0087] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0088] Example 1:

[0089] Please see Figure 1-3 This invention proposes an audiovisual target recognition method based on an improved Swing Transformer segmentation network. The improved Swing Transformer segmentation network includes:

[0090] Based on the end-to-end audiovisual pixel-level segmentation module, semantic association and feature alignment between modalities are performed through the AFI module;

[0091] A sliding window self-attention transformation module based on residual connections is used to enhance the network model's ability to recognize objects of different sizes in video frames and to enhance the correlation of global long-distance semantic information space.

[0092] A size adaptive module is used to address the fixed limitations of the sliding window self-attention mechanism;

[0093] Symmetrical transform encoder-decoder structure network based on hierarchical deep sliding window self-attention mechanism;

[0094] Multiple skip connections are used to directly connect corresponding layers of the encoder and decoder to preserve and pass high-resolution feature information;

[0095] A fusion loss function is used to simultaneously improve recognition accuracy and classification balance;

[0096] The method includes the following steps:

[0097] S1. Input the video frame feature information into the image size adaptive module to extract image feature information; specifically, this includes the following:

[0098] S1.1 Input the video frames into the convolution module to initially extract image features;

[0099] S1.2 Input the image features into the image size adaptive module to make them conform to the fixed size requirements of the sliding window self-attention transformation module;

[0100] S1.3, Feature extraction and downsampling are performed using the sliding window self-attention transformation module;

[0101] S2. Input the audio information into the convolution module to extract audio feature information; specifically, this includes the following:

[0102] S2.1 Extract audio independently using an audio encoder, given an audio segment s1;

[0103] S2.2. Process the audio segment s1 given in S1 into a spectrum using short-time Fourier transform;

[0104] S2.3. Input the spectrogram into the convolutional neural network VGGish, and extract audio features A using pre-trained weights from the AudioSet dataset. The function is expressed as:

[0105] A=T×d

[0106] Where T represents the time in seconds, and d represents the feature dimension;

[0107] S2.4. The audio features A obtained in S2.3 are transformed into a feature space with the same dimension as the visual feature mapping Fi through a linear layer;

[0108] S2.5. Copy the converted audio features in the feature space. h i × w i Next, and reshape it to the same size as Fi, denoted as . ;

[0109] S3. The image processed by S1 is input into a series of sliding window self-attention transformation modules for further feature extraction and processing. Then, the image size adaptation module is used to restore the size of the output feature map. Specifically, this includes the following:

[0110] S3.1. Divide the image into multiple small blocks, flatten the blocks and convert them into fixed-dimensional vectors through a linear layer, and add positional information to the feature vector of each block to ensure that the model understands the spatial relationship between different blocks.

[0111] S3.2 After processing in S3.1, the sequence containing positional encoding information is retained as a residual and input into the normalization layer. LN For each element of the input sequence, multi-head self-attention is calculated and added to the residual. The specific formula is as follows:

[0112]

[0113] in, Indicates the output sequence. x l-1 Represents the input sequence;

[0114] S3.3. Retain the output obtained in S3.2 as a residual and input it sequentially into the normalization layer. LN Multilayer perceptron network MLP Then add it to the residual, the specific formula is as follows:

[0115]

[0116] in, x l Indicates the output sequence;

[0117] S3.4. Retain the output sequence from S3.3 as a residual and input it into the normalization layer. LN For each element of the input sequence, a moving window self-attention calculation is performed, which is then added to the residual. The specific formula is as follows:

[0118]

[0119] in, The output sequence is represented;

[0120] S3.5. Using the output obtained in S3.4 as the residual and input, repeat the operation in S2.3. The specific formula is as follows:

[0121]

[0122] in, x l+1 Indicates the output sequence;

[0123] S3.6 After a series of sliding window self-attention transformation modules, the video frame feature size is restored to the input size using the Resize adaptive module, which is beneficial for the design of the hierarchical frame feature extraction structure.

[0124] S4. A hierarchical encoder is used to generate feature maps of different depths at each stage through downsampling, and the features in each encoding stage and the original image resolution features are recorded; specifically, this includes the following:

[0125] Based on the video frame feature coding network, feature downsampling is implemented through convolutional layers with a stride of 2 and a kernel of 3×3. For each deeper layer, the width and height are half of the previous layer, and the number of channels is twice that of the previous layer, forming feature maps of four different depths: 1 / 4, 1 / 8, 1 / 16, and 1 / 32. The shallow feature maps contain detailed feature information, while the deep feature maps contain high-level semantic information.

[0126] S5. Input the audio and image feature information obtained in S2 and S4 into the AFI module for feature alignment. Introduce audio feature information to guide segmentation and establish the relationship between temporal audio and video frame pixels. The video frame features after passing through the AFI module are expected to be enhanced in the regions that match the audio. Specifically, this includes the following:

[0127] Visual feature maps V i and the audio characteristics of the entire video A Input the AFI module to get the first i Phase update feature map Z i Its function is expressed as:

[0128]

[0129] in, θ , ϕ , g and μ This represents a 1×1×1 convolution operation. NRepresents a normalization factor. α i Indicates audio-visual similarity, where each visual pixel interacts with all audio elements through the AFI module; superscript T Indicates the transpose operation;

[0130] S6. Input the image feature information output in S4 into the decoder, upsample the image, and restore it to the feature map of the input size; specifically, this includes the following:

[0131] The image feature information output from S4 is upsampled through a convolutional layer with a stride of 2 and a kernel size of 3×3. For each shallower layer, its width and height are twice that of the previous layer, and its number of channels is half that of the previous layer, thus restoring feature maps at depths of 1 / 8, 1 / 4, and 1 / 2. The restored feature maps contain both detailed feature information and high-level semantic information extracted from the deeper feature maps.

[0132] S7. Use feature fusion technology to fuse the audio sampling features fused by the AFI module in S5 and the features upsampled after decoding in S6; specifically including the following:

[0133] S7.1. Using feature fusion, the deepest 1 / 16 of the encoded feature map is upsampled to restore it to 1 / 8 of the size of the previous layer.

[0134] S7.2. The feature map output in S7.1 is fused with the corresponding 1 / 8 feature map output in S5 after alignment with the audio features by the AFI module, in order to ensure the integrity and richness of information during feature transmission and improve the accuracy of segmentation and edge details.

[0135] S8. After fusion is completed, the fused features are processed using the operations in S1 and S3.

[0136] S9. Input the feature map output from S8 into the segmentation head, classify each pixel into a predetermined category, obtain the semantic segmentation and recognition result, calculate the recognition result using the fusion loss function, and improve the network parameters through backpropagation; specifically including the following:

[0137] The semantic segmentation and recognition results are fused with Soft Cross Entrophy Loss and Lovza Loss at a weight ratio of 1:1, and backpropagation is performed to iterate the network parameters, thereby improving the accuracy of video frame segmentation and classification.

[0138] The loss function employs a 1:1 weighted fusion of Soft Cross Entropy Loss and Lovasz Loss. This fusion improves model robustness and generalization ability, reduces overfitting, provides more accurate gradient signals, and facilitates better model learning and parameter tuning. Furthermore, because Lovasz Loss is mathematically differentiable, it is easier to combine with other loss functions to further enhance model performance. Additionally, by adjusting the weights of the two loss functions during the fusion calculation, the weights of different objectives can be balanced, allowing for flexible control over the optimization level of the model across various task metrics to adapt to practical needs.

[0139] The specific formula for calculating Soft Cross Entropy Loss is as follows:

[0140]

[0141] in, L SCE This represents the value of the Soft Cross Entropy Loss function; y i The label representing the i-th category in the real labels is set as a soft label. y i ∈(0,1); P ( x i () represents the probability of the i-th class predicted by the model;

[0142] Lovasz Loss not only focuses on correctly classified samples but also considers boundary samples and misclassified samples. Therefore, it can handle problems of class imbalance and imbalance between easy and difficult samples. The specific calculation formula is as follows:

[0143]

[0144]

[0145]

[0146]

[0147] in, This represents the loss function to be optimized. Represents the actual value; c This represents the set of pixels with prediction errors. M c This represents the set of segments where the network segmentation results and labels do not match. M cThe domain is {0,1} p , p Indicates the number of pixels;

[0148] against We utilize the lovasz extension for smooth extension and implement it in multi-class segmentation. f i ( c ) indicates the first c The probability value after a softmax-like operation; using a scoring function. f i ( c To construct a pixel error m i ( c )vector;

[0149] against Using errors m ( c Vector construction to replace Δ Jc The loss function;

[0150] against To optimize the evaluation of mIoU metrics across all categories, the average of the above... loss ( f ( c )).

[0151] In summary, this invention applies a speech encoder and a visual encoder to map speech and visual features to a unified semantic space, and then uses the AFI module to perform semantic association and feature alignment between modalities. This invention proposes multimodal fusion, which combines speech recognition with video processing, to achieve multimodal information fusion, thereby improving the overall system performance and robustness. It can not only provide rich contextual information, but also improve the system's accuracy and real-time processing capabilities, providing better solutions for various application scenarios.

[0152] Example 2:

[0153] Based on Example 1, but with some differences, please refer to Figure 4 This invention utilizes audio and image information to achieve sound source localization and detailed segmentation. The specific segmentation results are as follows: Figure 4 As shown. Figure 4 The first row contains the audio information for each frame of the image, and the second row contains the corresponding image information. The AFI module performs semantic association and feature alignment between modalities, realizing multimodal fusion and ultimately achieving accurate target segmentation.

[0154] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for audiovisual target recognition based on an improved Swing Transformer segmentation network, characterized in that, Includes the following steps: S1. Input the video frame feature information into the image size adaptive module to extract image feature information; S2. Input the audio information into the convolution module to extract audio feature information; S3. Input the image processed by S1 into a series of sliding window self-attention transformation modules to further extract and process features, and then use the image size adaptive module to restore the size of the output feature map. S4. Use a hierarchical encoder to generate feature maps of different depths through downsampling at each stage, and record the features in each encoding stage as well as the original image resolution features. S5. Input the audio feature information and image feature information obtained in S2 and S4 into the AFI module for feature alignment. Introduce audio feature information to guide segmentation in order to establish the relationship between temporal audio and video frame pixels. The video frame features after passing through the AFI module are expected to be enhanced in the region that matches the audio. S6. Input the image feature information output in S4 into the decoder, upsample the image, and restore it to the feature map of the input size. S7. Use feature fusion technology to fuse the audio sampling features fused by the AFI module in S5 and the features upsampled after decoding in S6; S8. After fusion is completed, the fused features are processed using the operations in S1 and S3. S9. Input the feature map output from S8 into the segmentation head, classify each pixel into a predetermined category, obtain the semantic segmentation and recognition result, calculate the recognition result using the fusion loss function, and improve the network parameters through backpropagation.

2. The audiovisual target recognition method based on the improved Swing Transformer segmentation network according to claim 1, characterized in that, S1 specifically includes the following: S1.1 Input the video frames into the convolution module to initially extract image features; S1.2 Input the image features into the image size adaptive module to make them conform to the fixed size requirements of the sliding window self-attention transformation module; S1.

3. Feature extraction and downsampling are performed using the sliding window self-attention transformation module.

3. The audiovisual target recognition method based on the improved Swing Transformer segmentation network according to claim 1, characterized in that, S2 specifically includes the following: S2.1 Extract audio independently using an audio encoder, given an audio segment s1; S2.

2. Process the audio segment s1 given in S1 into a spectrum using short-time Fourier transform; S2.

3. Input the spectrogram into the convolutional neural network VGGish, and extract audio features A using pre-trained weights from the AudioSet dataset. The function is expressed as: A=T×d Where T represents the time in seconds, and d represents the feature dimension; S2.

4. The audio features A obtained in S2.3 are transformed into a feature space with the same dimension as the visual feature mapping Fi through a linear layer; S2.

5. Copy the converted audio features in the feature space. h i × w i Next, and reshape it to the same size as Fi, denoted as . ; h i × w i This indicates the size of the image embedding feature map.

4. The audiovisual target recognition method based on the improved Swing Transformer segmentation network according to claim 1, characterized in that, S5 specifically includes the following: Visual feature maps V i and the audio characteristics of the entire video A Input the AFI module to get the first i Phase update feature map Z i Its function is expressed as: in, θ , ϕ , g and μ This represents a 1×1×1 convolution operation. N Represents a normalization factor. α i Indicates audio-visual similarity, where each visual pixel interacts with all audio elements through the AFI module; superscript T This indicates the transpose operation.

5. The audiovisual target recognition method based on the improved Swing Transformer segmentation network according to claim 1, characterized in that, S6 specifically includes the following: The image feature information output from S4 is upsampled through a convolutional layer with a stride of 2 and a kernel size of 3×3. Each shallower layer has twice the width and height of the previous layer and half the number of channels, thus restoring feature maps at depths of 1 / 8, 1 / 4, and 1 / 2. The feature map that restores the resolution contains both detailed feature information and high-level semantic information extracted from the deep feature map.

6. The audiovisual target recognition method based on the improved Swing Transformer segmentation network according to claim 1, characterized in that, S7 specifically includes the following: S7.

1. Using feature fusion, the deepest 1 / 16 of the encoded feature map is upsampled to restore it to 1 / 8 of the size of the previous layer. S7.

2. The feature map output in S7.1 is fused with the corresponding 1 / 8 feature map output in S5 after alignment with audio features by the AFI module, in order to ensure the integrity and richness of information during feature transmission and improve the accuracy of segmentation and edge details.

Images