An unsupervised audiovisual segmentation method, system, and electronic device

Abstract

This invention discloses an unsupervised audiovisual segmentation method, system, and electronic device. The method includes: acquiring an audiovisual dataset and extracting audio and visual features; mapping the features of the audio and visual modalities to a common feature space, calculating the feature association between the two modalities through cross-attention, and determining the coarse-grained localization of the sound source target in the video frame; mining the neighborhood semantic associations of the visual features, constructing a semantic similarity matrix, and performing fine-grained correction on the sound source target segmentation to obtain a first sound source target segmentation map; generating a second sound source target segmentation map through an encoding / decoding network based on audiovisual feature interaction learning; aligning the first and second sound source target segmentation maps to obtain the optimized final audiovisual segmentation result. The embodiments of this invention enable more accurate sound source target boundary information and more stable and accurate generation of sound source target segmentation maps, and can be widely applied in the field of audio and video processing technology.

Description

Technical Field

[0001] This invention relates to the field of audio and video processing technology, and in particular to an unsupervised audio-visual segmentation method, system, and electronic device. Background Technology

[0002] Audiovisual segmentation is an advanced cross-modal audio-visual analysis technique designed to extract sound-producing targets from videos and generate refined segmentation maps of these targets. Audiovisual segmentation uses audio as a guiding signal to determine which object to segment and obtain its complete pixel-level mask map. Compared to sound source localization, this task not only locates the sound-producing object based on the sound source but also accurately segments the entire object emitting sound within the video frame. This technology has wide applications in speech recognition, audio separation, and audio editing. It also holds promising prospects in fields such as smart homes, autonomous driving, medical image processing, and robot navigation.

[0003] Hearing and vision are two of the most important sensors for humans to perceive the world. In daily life, sound signals and visual signals are often complementary. While audiovisual segmentation technology has recently made some progress by leveraging the close relationship between sight and sound, several challenges remain. One major challenge is the high precision required for sound source localization, demanding the learning of accurate pixel-level sound source maps, rather than approximate image patch-level localization maps. Existing sound source localization methods typically learn audiovisual correspondences through contrastive learning, then directly calculate audiovisual attention to estimate the sound source region. These methods often output coarse and unreliable sound source localization information, lacking detailed descriptions such as object shape.

[0004] Another challenge is fine-grained sound source localization in a completely unsupervised paradigm. Supervised methods typically require building pixel-level labeled datasets and generating pixel-level results using semantic segmentation models to accurately identify sound source regions. However, supervised methods have a large need for annotation, but the annotation process is cumbersome and difficult to implement in real-world environments. Summary of the Invention

[0005] The main objective of this invention is to propose an unsupervised audiovisual segmentation method, system, and electronic device that can make the sound source target boundary information more accurate, and make the generation of sound source target segmentation maps more stable and accurate.

[0006] To achieve the above objectives, one aspect of this invention proposes an unsupervised audiovisual segmentation method, comprising the following steps:

[0007] Obtain the audiovisual dataset and extract audio and visual features;

[0008] The features of the audio and visual modalities are mapped to a common feature space, and then the feature association between the two modalities is calculated through cross-attention to determine the coarse-grained localization of the sound source target in the video frame.

[0009] By mining the neighborhood semantic associations of visual features, constructing a semantic similarity matrix, and performing fine-grained correction on the sound source target segmentation, the first sound source target segmentation map is obtained.

[0010] Based on interactive learning of audio-visual features, a second sound source target segmentation map is generated through an encoding and decoding network;

[0011] By training the model through self-supervised learning, the first sound source target segmentation map and the second sound source target segmentation map are aligned to obtain the optimized final audiovisual segmentation result.

[0012] In some embodiments, the step of feature mapping the features of the audio and visual modalities to a common feature space, and then calculating the feature association between the two modalities through cross-attention to determine the coarse-grained localization of the sound source target in the video frame includes the following steps:

[0013] Feature mapping is performed using a multilayer perceptron to obtain visual and audio feature representations after feature mapping;

[0014] By calculating the cross-attention between audio and visual information, a coarse-grained sound source target segmentation map is obtained;

[0015] Normalize and binarize the attention matrix to obtain the sound source target mask matrix;

[0016] Based on the preliminary attention calculation results, the visual features are divided into a feature set that semantically matches the audio features and a feature set that does not match. The audio-visual matching features are paired to form a positive sample set, and the non-matching features are combined to form a negative sample set. The positive and negative sample feature sets are learned through a contrastive learning method to complete the audio-visual cross-modal semantic alignment.

[0017] When combining audio-visual matching features as positive sample pairs, a random mask enhancement operation is added to the contrastive learning model. During the contrastive learning training process, visual features at some positions in the positive sample combination are randomly discarded to determine the coarse-grained localization of the sound source target in the video frame.

[0018] In some embodiments, the formula for calculating cross-attention is:

[0019]

[0020]

[0021] Where Q represents the query vector used for attention matrix calculation; ω cThis represents the weight matrix that transforms audio features into a query vector. ω represents the audio feature representation after feature mapping; K represents the key vector used for attention matrix calculation; k V represents the weight matrix that transforms audio features into key vectors; V represents the value vector used for attention matrix calculation; ω v This represents the weight matrix that converts visual features into value vectors. Represents the visual feature representation after feature mapping; A cs Represents the attention matrix; d k Represents the adjustment parameter;

[0022] The expression for the loss function in contrastive learning is:

[0023]

[0024] Among them, L cs τ represents the loss function for contrastive learning; τ is the temperature parameter; Θ() represents the random masking enhancement processing of visual features; k represents the neighborhood feature of the k-th non-sound source target in the neighborhood set; j represents the neighborhood feature of the j-th sound source target in the neighborhood set.

[0025] In some embodiments, the process of mining the neighborhood semantic associations of visual features, constructing a semantic similarity matrix, and performing fine-grained correction on the sound source target segmentation to obtain a first sound source target segmentation map includes the following steps:

[0026] A semantic similarity matrix is ​​constructed by calculating the feature similarity between neighboring pixels. Specifically, a set of neighborhoods with a distance r from the coordinate center is set as the base point, and the semantic similarity values ​​between neighborhood features i and j in the neighborhood set are calculated to obtain the semantic similarity matrix.

[0027] The semantic similarity matrix is ​​used to predict the semantic association between adjacent coordinate features in a video frame.

[0028] Construct a self-supervised learning model, using the affinity matrix M A The feature value is set as the target segmentation map M of the sound source. c Set as the target matrix for feature mapping;

[0029] Coarse-grained sound source target segmentation map M c Quantization is performed to generate a semantic association label matrix; specifically, a pixel-wise adaptive mean shift algorithm is used to segment the sound source target image M. c Pixel-by-pixel feature adjustment yields a more refined sound source target segmentation map M. cfNext, two thresholds α1 and α2 are set to quantize the sound source target segmentation map M. cf The sound source target segmentation map M cf It is divided into a reliable sound source region, a silent region, and an uncertain region;

[0030] Generate pseudo-semantic association label Y based on the region division. cA Specifically, based on the region division matrix R... cf If the coordinates of neighborhood features i and j (x i y i ), (x j y j If they are located in the same semantic region, including the sound source region and the silent region, set their associated label to the positive label Y. cA (i, j) = y + If neighboring features i and j are located in different semantic regions of the sound source region and the silent region, respectively, set their associated labels to negative labels Y. cA (i, j) = y - If either of the neighborhood features i or j comes from the ignored region, their associated labels will be set to ignored;

[0031] The semantic similarity matrix and the sound source target segmentation map M are combined using a self-supervised learning method. c Association, based on semantic association of the tag matrix Y cA The objective function value is obtained by minimizing the loss function L. A The trained model generates an affinity matrix M that accurately represents semantic relationships. A ;

[0032] The learned affinity matrix M is obtained using a random walk algorithm. A The semantic transition matrix T is transformed into a semantic transition matrix, and the coarse-grained sound source target segmentation map M is corrected based on the semantic transition matrix T. c The first sound source target segmentation map is obtained.

[0033] In some embodiments, the formula for calculating the semantic similarity matrix is:

[0034] M A (i, j) = exp(-||F aff (I(x i y i ))-F aff (I(x j y j ))|| L1 )

[0035] Among them, M A (i, j) represents the semantic similarity matrix of neighborhood features i and j; I(x iy i ) represents (x) in the image i y i I(x) location area visual information; j y j ) represents (x) in the image i y i ) Location-specific visual information; F aff () represents the computation function that uses multi-layer convolutional networks and multi-layer perceptrons to aggregate and map features; L1 represents the L1 norm calculation;

[0036] The formula for calculating the semantic transition matrix T is as follows:

[0037]

[0038] Where β represents the hyperparameter; D represents the parameter used to adjust M. A The diagonal matrix after normalization; D ii This represents the value in the i-th row and i-th column of the diagonal matrix D;

[0039] The expression for the fine-grained correction process of the coarse-grained audiovisual segmentation results is as follows:

[0040]

[0041] Where vec(·) denotes the vectorization of the matrix, M CR It is a refined and corrected sound source target segmentation map.

[0042] In some embodiments, the step of generating a second sound source target segmentation map through an encoding / decoding network based on audio-visual feature interaction learning includes the following steps:

[0043] A visual encoder is constructed using a ResNet50 network. Multi-scale visual features are obtained from different coding layers of ResNet50, and the Atrous spatial pyramid pooling method is used to generate feature sets.

[0044] The final encoding layer of the ResNet50 network fuses the output features with the coarse-grained sound source target segmentation map to obtain fused features;

[0045] The fused features and multi-scale feature sets are input into the pyramid decoder. After encoding and decoding training, the decoder finally generates the second sound source target segmentation map.

[0046] The second sound source target segmentation map and the first sound source target segmentation map are configured as mutual supervision information, and the difference between the two is calculated using the L2 norm;

[0047] The calculation formula for the fusion feature is: F av =Fv +F v ⊙norm(M c ),

[0048] Among them, F av Represents fusion characteristics; F v Represents visual feature representation; ⊙ represents the Hadamard product; norm() refers to the minimum-maximum normalization calculation; M c A coarse-grained sound source target segmentation map.

[0049] In some embodiments, the step of aligning the first sound source target segmentation map and the second sound source target segmentation map through self-supervised learning training to obtain the optimized final audiovisual segmentation result includes the following steps:

[0050] We construct a loss function for feature contrast learning in audio-visual matching, a mapping loss between semantic affinity matrix and sound source-target segmentation map, and a self-supervised loss for sound source-target segmentation map. The weighted combination of the three loss functions is used as the final loss function.

[0051] Based on the final loss function, the model is trained through self-supervised learning, and the first sound source target segmentation map and the second sound source target segmentation map are aligned to obtain the optimized final audiovisual segmentation result.

[0052] The final loss function L is expressed as: L = L cs +λ1L S +λ2L A ,

[0053] Among them, L cs The loss for feature contrastive learning representing audio-visual matching; L s The self-supervised loss represents the source-target segmentation map; L A λ1 and λ2 represent the mapping loss between the semantic affinity matrix and the source-target segmentation map; λ1 and λ2 represent weighting factors.

[0054] Another aspect of this invention provides an unsupervised audiovisual segmentation system, comprising:

[0055] The first module is used to acquire audiovisual datasets and extract audio and visual features;

[0056] The second module is used to map the features of the audio and visual modalities to a common feature space, and then calculate the feature association between the two modalities through cross-attention to determine the coarse-grained localization of the sound source target in the video frame.

[0057] The third module is used to mine the neighborhood semantic associations of visual features, construct a semantic similarity matrix, perform fine-grained correction on the sound source target segmentation, and obtain the first sound source target segmentation map.

[0058] The fourth module is used for interactive learning based on audio-visual features to generate a second sound source target segmentation map through an encoding and decoding network;

[0059] The fifth module is used to train a model through self-supervised learning, align the first sound source target segmentation map and the second sound source target segmentation map, and obtain the optimized final audiovisual segmentation result.

[0060] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0061] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0062] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0063] The embodiments of this invention include at least the following beneficial effects: This invention provides an unsupervised audiovisual segmentation method, system, and electronic device. This scheme acquires an audiovisual dataset and extracts audio and visual features; maps the features of the audio and visual modalities to a common feature space, then calculates the feature association between the two modalities through cross-attention to determine the coarse-grained localization of the sound source target in the video frame; mines the neighborhood semantic associations of the visual features, constructs a semantic similarity matrix, and performs fine-grained correction on the sound source target segmentation to obtain a first sound source target segmentation map; based on audiovisual feature interaction learning, a second sound source target segmentation map is generated through an encoding / decoding network; a model is trained through self-supervised learning, and the first and second sound source target segmentation maps are aligned to obtain the optimized final audiovisual segmentation result. The embodiments of this invention enable more accurate sound source target boundary information and make the generation of sound source target segmentation maps more stable and accurate. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0065] Figure 2 This is a flowchart of the overall steps provided in the embodiments of the present invention;

[0066] Figure 3 This is a flowchart illustrating the specific implementation steps provided in the embodiments of the present invention;

[0067] Figure 4 This is a diagram illustrating the implementation steps of the cross-attention mechanism provided in an embodiment of the present invention;

[0068] Figure 5 This is a schematic diagram of the fine-grained sound source target segmentation algorithm based on semantic similarity provided in this embodiment of the invention;

[0069] Figure 6 This is a flowchart of generating a sound source target segmentation map through an audio-visual interactive encoding and decoding model provided in an embodiment of the present invention;

[0070] Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0072] It is understood that the terms “first,” “second,” etc., used in this invention may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to determination” as used herein may be interpreted as “when…” or “when…” or “in response to determination.”

[0073] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0074] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0075] The unsupervised audiovisual segmentation method, system, and electronic device provided in this invention relates to the field of audio and video processing technology. The unsupervised audiovisual segmentation method provided in this invention can be applied to terminals, servers, or software running on either a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster consisting of multiple physical servers, or a distributed system. It can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the unsupervised audiovisual segmentation method, but is not limited to the above forms.

[0076] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0077] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0078] Server 101 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0079] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0080] Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc. It can also be a vehicle-mounted terminal of the various device types described above, but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0081] Exemplary based on Figure 1 The implementation environment shown in this embodiment of the invention provides an unsupervised audiovisual segmentation method. The following description uses the application of this unsupervised audiovisual segmentation method in server 101 as an example. It can be understood that this method can also be applied to terminal 102.

[0082] Reference Figure 2 , Figure 2 The flowchart illustrates an unsupervised audiovisual segmentation method applied to a server, provided in an embodiment of the present invention. The execution subject of this method can be any of the aforementioned computer devices (including a server or a terminal). See also... Figure 2 The method may include the following steps:

[0083] Obtain the audiovisual dataset and extract audio and visual features;

[0084] The features of the audio and visual modalities are mapped to a common feature space, and then the feature association between the two modalities is calculated through cross-attention to determine the coarse-grained localization of the sound source target in the video frame.

[0085] By mining the neighborhood semantic associations of visual features, constructing a semantic similarity matrix, and performing fine-grained correction on the sound source target segmentation, the first sound source target segmentation map is obtained.

[0086] Based on interactive learning of audio-visual features, a second sound source target segmentation map is generated through an encoding and decoding network;

[0087] By training the model through self-supervised learning, the first sound source target segmentation map and the second sound source target segmentation map are aligned to obtain the optimized final audiovisual segmentation result.

[0088] In some embodiments, the step of feature mapping the features of the audio and visual modalities to a common feature space, and then calculating the feature association between the two modalities through cross-attention to determine the coarse-grained localization of the sound source target in the video frame includes the following steps:

[0089] Feature mapping is performed using a multilayer perceptron to obtain visual and audio feature representations after feature mapping;

[0090] By calculating the cross-attention between audio and visual information, a coarse-grained sound source target segmentation map is obtained;

[0091] Normalize and binarize the attention matrix to obtain the sound source target mask matrix;

[0092] Based on the preliminary attention calculation results, the visual features are divided into a feature set that semantically matches the audio features and a feature set that does not match. The audio-visual matching features are paired to form a positive sample set, and the non-matching features are combined to form a negative sample set. The positive and negative sample feature sets are learned through a contrastive learning method to complete the audio-visual cross-modal semantic alignment.

[0093] When combining audio-visual matching features as positive sample pairs, a random mask enhancement operation is added to the contrastive learning model. During the contrastive learning training process, visual features at some positions in the positive sample combination are randomly discarded to determine the coarse-grained localization of the sound source target in the video frame.

[0094] In some embodiments, the formula for calculating cross-attention is:

[0095]

[0096]

[0097] Where Q represents the query vector used for attention matrix calculation; ω c This represents the weight matrix that transforms audio features into a query vector. ω represents the audio feature representation after feature mapping; K represents the key vector used for attention matrix calculation; k V represents the weight matrix that transforms audio features into key vectors; V represents the value vector used for attention matrix calculation; ω v This represents the weight matrix that converts visual features into value vectors. Represents the visual feature representation after feature mapping; Acs Represents the attention matrix; d k Represents the adjustment parameter;

[0098] The expression for the loss function in contrastive learning is:

[0099]

[0100] Among them, L cs τ represents the loss function for contrastive learning; τ is the temperature parameter; Θ() represents the random masking enhancement processing of visual features; k represents the neighborhood feature of the k-th non-sound source target in the neighborhood set; j represents the neighborhood feature of the j-th sound source target in the neighborhood set.

[0101] In some embodiments, the process of mining the neighborhood semantic associations of visual features, constructing a semantic similarity matrix, and performing fine-grained correction on the sound source target segmentation to obtain a first sound source target segmentation map includes the following steps:

[0102] A semantic similarity matrix is ​​constructed by calculating the feature similarity between neighboring pixels. Specifically, a set of neighborhoods with a distance r from the coordinate center is set as the base point, and the semantic similarity values ​​between neighborhood features i and j in the neighborhood set are calculated to obtain the semantic similarity matrix.

[0103] The semantic similarity matrix is ​​used to predict the semantic association between adjacent coordinate features in a video frame.

[0104] Construct a self-supervised learning model, using the affinity matrix M A The feature value is set as the target segmentation map M of the sound source. c Set as the target matrix for feature mapping;

[0105] Coarse-grained sound source target segmentation map M c Quantization is performed to generate a semantic association label matrix; specifically, a pixel-wise adaptive mean shift algorithm is used to segment the sound source target image M. c Pixel-by-pixel feature adjustment yields a more refined sound source target segmentation map M. cf Next, two thresholds α1 and α2 are set to quantize the sound source target segmentation map M. cf The sound source target segmentation map M cf It is divided into a reliable sound source region, a silent region, and an uncertain region;

[0106] Generate pseudo-semantic association label Y based on the region division. cA Specifically, based on the region division matrix R... cf If the coordinates of neighborhood features i and j (x i y i ), (x j yj If they are located in the same semantic region, including the sound source region and the silent region, set their associated label to the positive label Y. cA (i, j) = y + If neighboring features i and j are located in different semantic regions of the sound source region and the silent region, respectively, set their associated labels to negative labels Y. cA (i, j) = y - If either of the neighborhood features i or j comes from the ignored region, their associated labels will be set to ignored;

[0107] The semantic similarity matrix and the sound source target segmentation map M are combined using a self-supervised learning method. c Association, based on semantic association of the tag matrix Y cA The objective function value is obtained by minimizing the loss function L. A The trained model generates an affinity matrix M that accurately represents semantic relationships. A ;

[0108] The learned affinity matrix M is obtained using a random walk algorithm. A The semantic transition matrix T is transformed into a semantic transition matrix, and the coarse-grained sound source target segmentation map M is corrected based on the semantic transition matrix T. c The first sound source target segmentation map is obtained.

[0109] In some embodiments, the formula for calculating the semantic similarity matrix is:

[0110] M A (i, j) = exp(-||F aff (I(x i y i ))-F aff (I(x j y j ))|| L1 )

[0111] Among them, M A (i, j) represents the semantic similarity matrix of neighborhood features i and j; I(x i y i ) represents (x) in the image i y i I(x) location area visual information; j y j ) represents (x) in the image i y i ) Location-specific visual information; F aff () represents the computation function that uses multi-layer convolutional networks and multi-layer perceptrons to aggregate and map features; L1 represents the L1 norm calculation;

[0112] The formula for calculating the semantic transition matrix T is as follows:

[0113]

[0114] Where β represents the hyperparameter; D represents the parameter used to adjust M. A The diagonal matrix after normalization; D ii This represents the value in the i-th row and i-th column of the diagonal matrix D;

[0115] The expression for the fine-grained correction process of the coarse-grained audiovisual segmentation results is as follows:

[0116]

[0117] Where vec(·) denotes the vectorization of the matrix, M CR It is a refined and corrected sound source target segmentation map.

[0118] In some embodiments, the step of generating a second sound source target segmentation map through an encoding / decoding network based on audio-visual feature interaction learning includes the following steps:

[0119] A visual encoder is constructed using a ResNet50 network. Multi-scale visual features are obtained from different coding layers of ResNet50, and the Atrous spatial pyramid pooling method is used to generate feature sets.

[0120] The final encoding layer of the ResNet50 network fuses the output features with the coarse-grained sound source target segmentation map to obtain fused features;

[0121] The fused features and multi-scale feature sets are input into the pyramid decoder. After encoding and decoding training, the decoder finally generates the second sound source target segmentation map.

[0122] The second sound source target segmentation map and the first sound source target segmentation map are configured as mutual supervision information, and the difference between the two is calculated using the L2 norm;

[0123] The calculation formula for the fusion feature is: F av =F v +F v ⊙norm(M c ),

[0124] Among them, F av Represents fusion characteristics; F v Represents visual feature representation; ⊙ represents the Hadamard product; norm() refers to the minimum-maximum normalization calculation; M c A coarse-grained sound source target segmentation map.

[0125] In some embodiments, the step of aligning the first sound source target segmentation map and the second sound source target segmentation map through self-supervised learning training to obtain the optimized final audiovisual segmentation result includes the following steps:

[0126] We construct a loss function for feature contrast learning in audio-visual matching, a mapping loss between semantic affinity matrix and sound source-target segmentation map, and a self-supervised loss for sound source-target segmentation map. The weighted combination of the three loss functions is used as the final loss function.

[0127] Based on the final loss function, the model is trained through self-supervised learning, and the first sound source target segmentation map and the second sound source target segmentation map are aligned to obtain the optimized final audiovisual segmentation result.

[0128] The final loss function L is expressed as: L = L cs +λ i L S +λ2L A ,

[0129] Among them, L cs The loss for feature contrastive learning representing audio-visual matching; L s The self-supervised loss represents the source-target segmentation map; L A λ1 and λ2 represent the mapping loss between the semantic affinity matrix and the source-target segmentation map; λ1 and λ2 represent weighting factors.

[0130] The implementation steps of this invention will be described in detail below using specific application scenarios as examples:

[0131] This invention proposes an unsupervised audiovisual segmentation method to achieve pixel-level segmentation and localization of unsupervised sound source targets. To achieve pixel-level audiovisual segmentation, this invention designs a novel multi-instance contrastive learning method based on mask enhancement to align paired audiovisual features. This method effectively alleviates overfitting and suboptimal solutions, thus achieving better audiovisual alignment. Next, an audiovisual attention mechanism is used to obtain coarse-grained sound source target segmentation results at the image patch level. To further obtain pixel-level sound source segmentation results, this invention introduces an unsupervised sound source segmentation refinement module. This module proposes visual semantic association learning to explore the relationship between pixels with adjacent coordinates and predict the semantic association between adjacent pixels in the image. The coarse-grained sound source segmentation map is corrected based on the association matrix to generate a fine-grained sound source segmentation map. This operation optimizes the sound source target segmentation results, making the sound source target boundary information more accurate. In unsupervised audiovisual segmentation, due to the lack of true segmentation values ​​for the sound source targets, the accuracy of the sound source segmentation map obtained in the previous step cannot be evaluated. Therefore, this invention proposes an audiovisual fusion encoding / decoding algorithm to generate pixel-level segmentation maps of sound source targets. This algorithm consists of an encoder-decoder combination that generates a segmentation map of the sound source target under the guidance of the audio signal. This result is used to evaluate the segmentation result of the previous contrastive learning step. This approach can construct a self-supervised learning system, improve the training effect of contrastive learning segmentation, make it achieve better convergence, and also make the generation of sound source target segmentation maps more stable and accurate.

[0132] The following is in conjunction with the appendix Figure 3 The steps of the unsupervised audiovisual segmentation method according to embodiments of the present invention are described below:

[0133] S101: Audio and video data preprocessing.

[0134] Given an audiovisual dataset, with known video sequences and audio information D = {(v i a i The data, i = 1, ..., N, are processed by a pre-trained audio encoder VGGish and a visual encoder ResNet50 to perform preliminary feature encoding on the audio and video data, respectively, and to extract audio features F. a ={f a {i}, i = 1, ..., N, and visual feature F v ={f v (i), i = 1, ..., N}.

[0135] S102: Coarse-grained audiovisual segmentation based on cross-attention mechanism.

[0136] Coarse-grained audiovisual segmentation is implemented through a cross-attention mechanism, the specific algorithm principle of which is as follows: Figure 4As shown, this module consists of two parts: feature mapping and cross-attention calculation. The mapping layer maps the features of the two modalities to a common feature space, which facilitates the calculation of the feature association between the two modalities through cross-attention, thereby determining the visual region of the sound source target.

[0137] Here, a multilayer perceptron (MLP) is used for feature mapping, and the feature mapping process is represented by the function g(·). The visual features after feature mapping are represented as follows: The audio features after feature mapping are represented as follows: Audio-visual modal feature sequences are represented as follows and

[0138] This embodiment obtains a coarse-grained sound source-target segmentation map through audiovisual cross-attention calculation. The cross-attention calculation is shown in Equation 1. Sound source-target mask matrix M c It can be obtained from the attention matrix A cs Obtained through normalization and binarization operations.

[0139]

[0140] To compute an accurate sound source-target segmentation map using a cross-attention mechanism, semantic alignment of visual and audio features is necessary. This invention designs a multi-instance contrastive learning method to align audiovisual semantic features. Specifically, based on the initial attention calculation results, visual features are divided into a feature set that semantically matches the audio features and a feature set that does not. Audio-visual matching features are paired to form a positive sample set, while non-matching features are combined to form a negative sample set. Contrastive learning is used to learn from both the positive and negative sample feature sets, promoting cross-modal semantic alignment between audio and visual features.

[0141] To improve the generalization of audio feature matching with the visual location of the sound source target and to more accurately locate all sound source target locations using audio when combining audio-visual matching features as positive sample pairs, this invention further adds a random masking enhancement operation to the contrastive learning model. Specifically, during contrastive learning training, visual features at some locations in the positive sample combinations are randomly discarded to ensure that potential sound source target locations have more optimization opportunities. The random masking enhancement processing of visual features is expressed as follows: The improved loss function setting for contrastive learning is shown in Equation 2.

[0142]

[0143] Where τ is a temperature parameter, and the similarity between audio features and visual features is calculated by cosine distance.

[0144] S103: Fine-grained audiovisual segmentation based on semantic similarity.

[0145] To further obtain pixel-level sound source segmentation results, this invention designs a fine-grained sound source target segmentation module based on semantic similarity. This module uses visual semantic association learning to explore the semantic relationship between pixels with adjacent coordinates, corrects the coarse-grained sound source target segmentation map based on the semantic association matrix of adjacent pixels in the video frame, and generates fine-grained sound source target segmentation results.

[0146] Specifically, a semantic similarity matrix is ​​constructed to predict the semantic correlation between adjacent coordinate features in a video frame. This semantic similarity matrix is ​​used to compare the sound source target segmentation map M. c Alignment is achieved by correcting the coarse sound source map through self-supervised learning.

[0147] Figure 5 The diagram shows the principle of a fine-grained sound source target segmentation algorithm based on semantic similarity. This embodiment sets a semantic similarity matrix M. A The matrix value is obtained by calculating the feature similarity between neighboring pixels. Specifically, the visual features F extracted by ResNet50 are... v The input is a multilayer perceptron (MLP) to perform feature aggregation calculations. The calculated features are then used to mine the similarity relationships between neighboring pixels.

[0148] Specifically, taking a certain coordinate center as the base point, a set of neighborhoods with a distance r from the base point in the coordinate space is set. The semantic similarity value between neighborhood features i and j in this neighborhood set is calculated, denoted as M. A (i, j), i.e., the semantic similarity matrix. Assume the coordinates of neighborhood features i and j are determined by (x, j). i y i ), (x j y j The formula for calculating the semantic similarity matrix is ​​shown below.

[0149] M A (i, j) = exp(-||F aff (I(x i y i ))-F aff (I(x j y j ))|| L1 (3)

[0150] To more accurately represent the semantic similarity matrix M A Sound source target segmentation map M c In this embodiment, a self-supervised learning model is constructed, where the affinity matrix M... A The feature value is set as the source target segmentation map M. cThe target matrix is ​​set as the feature mapping matrix. To quantize the target value, this embodiment further refines the coarse-grained sound source target segmentation map M. c Perform quantization to generate a semantic association label matrix Y. cA The specific operation involves first employing a pixel-wise adaptive mean shift (PAMR) algorithm to segment the sound source target image M. c Pixel-by-pixel feature adjustment is performed to obtain a more refined matrix M. cf Next, two thresholds α1 and α2 (0 < α1 < α2 < 1) are set to quantize the sound source target segmentation map M. cf The region is divided into a reliable sound source region, a silent region, and an uncertain region. The division rules are defined in Formula 4.

[0151]

[0152] In this context, 1 and 0 represent the sound source region and the silent region, respectively, while Null represents the ignored region.

[0153] Based on the region division, this embodiment generates pseudo-semantic association label Y. cA The specific operation involves using the region partitioning matrix R... cf If the coordinates of neighborhood features i and j (x i y i ), (x j y j If they are located in the same semantic region, including the sound source region and the silent region, set their associated label to the positive label Y. cA (i, j) = y + Otherwise, if neighborhood features i and j are located in different semantic regions of the sound source region and the silent region, respectively, their associated labels are set to negative labels Y. cA (i, j) = y - If either neighborhood feature i or j comes from the ignored region, their associated labels will be set to ignored.

[0154] This invention utilizes a self-supervised learning method to transform the semantic similarity matrix M... A Sound source target segmentation map M e Association, based on semantic association of the tag matrix Y cA The objective function value is obtained by minimizing the loss function L. A The trained model generates an affinity matrix M that accurately represents semantic relationships. A Loss function L A As defined in Formula 5.

[0155] L A =-y + ·log(MA )-y - ·log(1-M A (5)

[0156] Semantic affinity matrix M A This is used to refine the sound source target segmentation map of the coarse-grained audiovisual segmentation result obtained in step S102.

[0157] The refinement of the sound source-target segmentation map is achieved through random walks. For the learned semantic affinity matrix M... A Here, a random walk algorithm is used to transform it into a semantic transition matrix T, which is used to correct the coarse-grained audiovisual segmentation result M. c The semantic transition matrix T is calculated as shown in Formula 6. Here, β is a hyperparameter used to ignore M. A The smaller correlation values ​​in the middle, and D is a diagonal matrix used to correlate M. A Normalization is performed.

[0158] T=D -1 (M A ) β D ii =∑ j (M A (i, j)) β (6)

[0159] Coarse-grained audiovisual segmentation result M C The fine-tuning process is shown in Formula 7.

[0160]

[0161] Where vec(·) denotes the vectorization of the matrix, M CR This is the refined source-target segmentation map. This refinement operation activates highly relevant semantic regions and suppresses erroneously activated regions, allowing the source-target segmentation map to better align with semantic boundaries.

[0162] S104: Audio-visual interactive encoding and decoding generates a sound source target segmentation map.

[0163] Step S103 involves refining the coarse-grained sound source target segmentation map by analyzing neighborhood semantic similarity to obtain a fine-grained sound source target segmentation map. This invention is implemented unsupervised, and the evaluation of the fine-grained sound source target segmentation map lacks real-world comparisons. Therefore, as... Figure 6 As shown, this embodiment further designs an audio-visual interactive encoding and decoding model to generate a sound source target segmentation map. This segmentation result and the fine-grained sound source target segmentation map obtained in step S103 serve as mutual supervision information, and the model is trained in a self-supervised learning manner.

[0164] P1: Audio-Visual Interaction Encoder

[0165] like Figure 6 As shown, the visual encoder uses a ResNet50 network to obtain multi-scale visual features at different encoding layers of ResNet50. The Atrous Spatial Pyramid Pooling (ASPP) method is used to generate feature sets {F1, F2, F3, F4} for subsequent feature processing.

[0166] On the other hand, the final encoding layer output feature F of the ResNet50 network v The coarse-grained sound source target segmentation matrix M obtained in step S102 c Feature fusion is performed to obtain the fused feature F. av .

[0167] F av =F v +F v ⊙norm(M c (8)

[0168] Here, norm refers to the minimum-maximum normalization calculation, and ⊙ represents the Hadamard product.

[0169] P2: Decode and generate a sound source target segmentation map.

[0170] In this embodiment, audio-visual fusion features and multi-scale feature sets are input into a Feature Pyramid Network (FPN). After encoding and decoding training, the decoder finally generates a sound source target segmentation map M. CG .

[0171] The sound source target segmentation map M generated by audio-visual interactive decoding CG The fine-grained sound source-target segmentation map obtained in step S103 serves as mutual supervision information, and the difference between the two is calculated using the L2 norm. The loss function is defined as shown in Equation 9.

[0172] L S =||M CG -M CR || L2 (9)

[0173] S105: Loss settings for training the model.

[0174] The method proposed in this invention includes three loss terms: the loss L for feature contrast learning in audio-visual matching. cs The mapping loss L between the semantic affinity matrix and the sound source-target segmentation mapA And the self-supervised loss L of the sound source target segmentation map S The weighted combination of the three loss functions forms the final loss function in this embodiment, as shown in Formula 10.

[0175] L = L cs +λ1L S +λ2L A (10)

[0176] λ1 and λ2 serve as weighting factors to balance the contributions of different loss functions.

[0177] The unsupervised audiovisual segmentation algorithm model proposed in this invention, after sufficient model training, is used to segment complete sound source target regions from video frames.

[0178] To better illustrate the technical effects of this invention, specific sample examples were collected to experimentally verify it. Extensive experiments were conducted on the AVSBench-S4 and VGGSS datasets, and the experimental results demonstrate the effectiveness of our proposed method. This embodiment selects three methods for comparative experiments to compare the audiovisual segmentation performance. These three methods are: FNAC (false negative aware contrastive learning), EZ-VSL (Easy Visual Sound Localization), and LVS (Localizing Visual Sounds). Four evaluation metrics—Mean Intersection over Union (mloU), F-score, Consensus Intersection over Union (cloU), and Area UnderCurye (AUC)—were used to evaluate the algorithm's segmentation performance. Higher values ​​for these four metrics indicate better segmentation performance.

[0179] Table 1 is a comparison table of the unsupervised audiovisual source target segmentation results of the present invention and the comparison method on the audiovisual dataset AVSBench-S4.

[0180] Table 1

[0181] Methods mloU (%) F-score FNAC 32.46 0.434 EZ-VSL 34.01 0.562 LVS 37.94 0.510 Method of the present invention 47.19 0.617

[0182] Table 2 is a comparison table of unsupervised audiovisual source target segmentation of the present invention and the comparison method on the VGGSS audio-visual dataset.

[0183] Table 2

[0184] Methods cIoU (%) AUC(%) LVS 34.40 38.20 EZ-VSL 34.38 37.70 FNAC 39.50 39.66 Method of the present invention 44.47 45.11

[0185] As shown in Tables 1 and 2, compared with these algorithms, the unsupervised audiovisual segmentation algorithm of this invention achieves a 9.25% improvement in mIoU on the AVSBench-S4 dataset and a 2.62% improvement in cIoU on the VGGSS benchmark. The experimental results fully demonstrate the effectiveness of the method of this invention.

[0186] Another aspect of this invention provides an unsupervised audiovisual segmentation system, comprising:

[0187] The first module is used to acquire audiovisual datasets and extract audio and visual features;

[0188] The second module is used to map the features of the audio and visual modalities to a common feature space, and then calculate the feature association between the two modalities through cross-attention to determine the coarse-grained localization of the sound source target in the video frame.

[0189] The third module is used to mine the neighborhood semantic associations of visual features, construct a semantic similarity matrix, perform fine-grained correction on the sound source target segmentation, and obtain the first sound source target segmentation map.

[0190] The fourth module is used for interactive learning based on audio-visual features to generate a second sound source target segmentation map through an encoding and decoding network;

[0191] The fifth module is used to train a model through self-supervised learning, align the first sound source target segmentation map and the second sound source target segmentation map, and obtain the optimized final audiovisual segmentation result.

[0192] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0193] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned unsupervised audiovisual segmentation method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0194] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0195] Please see Figure 7 , Figure 7 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0196] The processor 701 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0197] The memory 702 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 702 and is called and executed by the processor 701 to execute the unsupervised audiovisual segmentation method of the embodiments of this invention.

[0198] The input / output interface 703 is used to implement information input and output;

[0199] The communication interface 704 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0200] Bus 705 transmits information between various components of the device (e.g., processor 701, memory 702, input / output interface 703, and communication interface 704);

[0201] The processor 701, memory 702, input / output interface 703, and communication interface 704 are connected to each other within the device via bus 705.

[0202] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned unsupervised audiovisual segmentation method.

[0203] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0204] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0205] It should be noted that in various specific embodiments of the present invention, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of the present invention require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data for the normal operation of the embodiments of the present invention acquired.

[0206] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0207] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0208] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0209] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0210] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0211] It should be understood that in this invention, "at least one (item)" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0212] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0213] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0214] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0215] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0216] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. An unsupervised audiovisual segmentation method, characterized in that, Includes the following steps: Obtain the audiovisual dataset and extract audio and visual features; The features of the audio and visual modalities are mapped to a common feature space, and then the feature association between the two modalities is calculated through cross-attention to determine the coarse-grained localization of the sound source target in the video frame. By mining the neighborhood semantic associations of visual features, constructing a semantic similarity matrix, and performing fine-grained correction on the sound source target segmentation, the first sound source target segmentation map is obtained. Based on interactive learning of audio-visual features, a second sound source target segmentation map is generated through an encoding and decoding network; By training the model through self-supervised learning, the first sound source target segmentation map and the second sound source target segmentation map are aligned to obtain the optimized final audiovisual segmentation result.

2. The unsupervised audiovisual segmentation method according to claim 1, characterized in that, The process of mapping features from both audio and visual modalities to a common feature space, and then calculating the feature association between the two modalities through cross-attention to determine the coarse-grained localization of the sound source target in the video frame includes the following steps: Feature mapping is performed using a multilayer perceptron to obtain visual and audio feature representations after feature mapping; By calculating the cross-attention between audio and visual information, a coarse-grained sound source target segmentation map is obtained; Normalize and binarize the attention matrix to obtain the sound source target mask matrix; Based on the preliminary attention calculation results, the visual features are divided into a feature set that semantically matches the audio features and a feature set that does not match. The audio-visual matching features are paired to form a positive sample set, and the non-matching features are combined to form a negative sample set. The positive and negative sample feature sets are learned through a contrastive learning method to complete the audio-visual cross-modal semantic alignment. When combining audio-visual matching features as positive sample pairs, a random mask enhancement operation is added to the contrastive learning model. During the contrastive learning training process, visual features at some positions in the positive sample combination are randomly discarded to determine the coarse-grained localization of the sound source target in the video frame.

3. The unsupervised audiovisual segmentation method according to claim 2, characterized in that, The formula for calculating cross-attention is: Where Q represents the query vector used for attention matrix calculation; ω c This represents the weight matrix that converts audio features into query vectors; K represents the audio feature representation after feature mapping; K represents the key vector used for attention matrix calculation; ω k V represents the weight matrix that converts audio features into key vectors; V represents the value vector used for attention matrix calculation; ω v This represents the weight matrix that converts visual features into value vectors; Represents the visual feature representation after feature mapping; A cs Represents the attention matrix; d k Represents the adjustment parameter; The expression for the loss function in contrastive learning is: Among them, L cs τ represents the loss function for contrastive learning; τ is the temperature parameter; Θ() represents the random masking enhancement processing of visual features; k represents the neighborhood feature of the k-th non-sound source target in the neighborhood set; j represents the neighborhood feature of the j-th sound source target in the neighborhood set.

4. The unsupervised audiovisual segmentation method according to claim 1, characterized in that, The process of mining the neighborhood semantic associations of visual features, constructing a semantic similarity matrix, and performing fine-grained correction on the sound source target segmentation to obtain the first sound source target segmentation map includes the following steps: A semantic similarity matrix is ​​constructed by calculating the feature similarity between neighboring pixels. Specifically, a set of neighborhoods with a distance r from the coordinate center is set as the base point, and the semantic similarity values ​​between neighborhood features i and j in the neighborhood set are calculated to obtain the semantic similarity matrix. The semantic similarity matrix is ​​used to predict the semantic association between adjacent coordinate features in video frames. Construct a self-supervised learning model, using the affinity matrix M A The feature value is set as the target segmentation map M of the sound source. c Set as the target matrix for feature mapping; Coarse-grained sound source target segmentation map M c Quantization is performed to generate a semantic association label matrix; specifically, a pixel-wise adaptive mean shift algorithm is used to segment the sound source target image M. c Pixel-by-pixel feature adjustment is performed to obtain a more refined sound source target segmentation map M. cf Next, two thresholds α1 and α2 are set to quantize the sound source target segmentation map M. cf The sound source target segmentation map M cf It is divided into a reliable sound source region, a silent region, and an uncertain region; Generate pseudo-semantic association label Y based on the region division. cA Specifically, based on the region division matrix R... cf If the coordinates of neighborhood features i and j (x i ,y i ),(x j ,y j If they are located in the same semantic region, including the sound source region and the silent region, set their associated label to the positive label Y. cA (i, j) = y + If neighboring features i and j are located in different semantic regions of the sound source region and the silent region, respectively, set their associated labels to negative labels Y. cA (i, j) = y - If either of the neighborhood features i or j comes from the ignored region, their associated labels will be set to ignored; The semantic similarity matrix and the sound source target segmentation map M are combined using a self-supervised learning method. c Association, based on semantic association of the tag matrix Y cA The objective function value is obtained by minimizing the loss function L. A The trained model generates an affinity matrix M that accurately represents semantic relationships. A ; The learned affinity matrix M is obtained using a random walk algorithm. A The semantic transition matrix T is transformed into a semantic transition matrix, and the coarse-grained sound source target segmentation map M is corrected based on the semantic transition matrix T. c The first sound source target segmentation map is obtained.

5. The unsupervised audiovisual segmentation method according to claim 4, characterized in that, The formula for calculating the semantic similarity matrix is: M A (i,j)=exp(-||F aff (I(x i ,y i ))-F aff (I(x j ,y j ))|| L1 ) Among them, M A (i, j) represents the semantic similarity matrix of neighborhood features i and j; I(x i ,y i ) represents (x) in the image i ,y i I(x) location area visual information; j ,y j ) represents (x) in the image j ,y j ) Location-specific visual information; F aff () represents the computation function that uses multi-layer convolutional networks and multi-layer perceptrons to aggregate and map features; L1 represents the L1 norm calculation; The formula for calculating the semantic transition matrix T is as follows: Where β represents the hyperparameter; D represents the parameter used to adjust M. A The diagonal matrix after normalization; D ii This represents the value in the i-th row and i-th column of the diagonal matrix D; The expression for the fine-grained correction process of the coarse-grained audiovisual segmentation results is as follows: Where vec(·) denotes the vectorization of the matrix, M CR It is a refined and corrected sound source target segmentation map.

6. The unsupervised audiovisual segmentation method according to claim 1, characterized in that, The step of generating a second sound source target segmentation map through an encoding / decoding network based on interactive learning of audio-visual features includes the following steps: A visual encoder is constructed using a ResNet50 network. Multi-scale visual features are obtained from different coding layers of ResNet50, and the Atrous spatial pyramid pooling method is used to generate feature sets. The final encoding layer of the ResNet50 network fuses the output features with the coarse-grained sound source target segmentation map to obtain fused features; The fused features and multi-scale feature sets are input into the pyramid decoder. After encoding and decoding training, the decoder finally generates the second sound source target segmentation map. The second sound source target segmentation map and the first sound source target segmentation map are configured as mutual supervision information, and the difference between the two is calculated using the L2 norm; The calculation formula for the fusion feature is: F av =F v +F v ⊙norm(M c ), Among them, F av Represents fusion characteristics; F v Represents visual feature representation; ⊙ represents the Hadamard product; norm() refers to the minimum-maximum normalization calculation; M c A coarse-grained sound source target segmentation map.

7. The unsupervised audiovisual segmentation method according to claim 1, characterized in that, The step of training a model through self-supervised learning, aligning the first sound source target segmentation map and the second sound source target segmentation map to obtain the optimized final audiovisual segmentation result includes the following steps: We construct a loss function for feature contrast learning in audio-visual matching, a mapping loss between semantic affinity matrix and sound source-target segmentation map, and a self-supervised loss for sound source-target segmentation map. The weighted combination of the three loss functions is used as the final loss function. Based on the final loss function, the model is trained through self-supervised learning, and the first sound source target segmentation map and the second sound source target segmentation map are aligned to obtain the optimized final audiovisual segmentation result. The final loss function L is expressed as: L = L cs +λ1L s +λ2L A , Among them, L cs The loss for feature contrastive learning representing audio-visual matching; L s The self-supervised loss represents the source-target segmentation map; L a λ1 and λ2 represent the mapping loss between the semantic affinity matrix and the source-target segmentation map; λ1 and λ2 represent weighting factors.

8. An unsupervised audiovisual segmentation system, characterized in that, include: The first module is used to acquire audiovisual datasets and extract audio and visual features; The second module is used to map the features of the audio and visual modalities to a common feature space, and then calculate the feature association between the two modalities through cross-attention to determine the coarse-grained localization of the sound source target in the video frame. The third module is used to mine the neighborhood semantic associations of visual features, construct a semantic similarity matrix, perform fine-grained correction on the sound source target segmentation, and obtain the first sound source target segmentation map. The fourth module is used for interactive learning based on audio-visual features to generate a second sound source target segmentation map through an encoding and decoding network; The fifth module is used to train a model through self-supervised learning, align the first sound source target segmentation map and the second sound source target segmentation map, and obtain the optimized final audiovisual segmentation result.

9. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 7.

Images