Unsupervised audio-visual segmentation method and system and electronic equipment

Abstract

The invention discloses an unsupervised audio-visual segmentation method and system and electronic equipment, and the method comprises the steps: obtaining an audio-visual data set, and carrying out the extraction to obtain audio features and visual features; performing feature mapping on the features of the audio mode and the visual mode to a public feature space, calculating feature association of the two modes through cross attention, and determining coarse-grained positioning of a video frame on a sound source target; mining neighborhood semantic association of the visual features, constructing a semantic similar matrix, and performing fine-grained correction on sound source target segmentation to obtain a first sound source target segmentation map; generating a second sound source target segmentation image through the coding and decoding network according to audio and visual feature interactive learning; and aligning the first sound source target segmentation image and the second sound source target segmentation image to obtain an optimized final audio-visual segmentation result. According to the embodiment of the invention, the boundary information of the sound source target can be more accurate, the sound source target segmentation map can be generated more stably and accurately, and the method can be widely applied to the technical field of audio and video processing.

Description

Technical Field

[0001] The present invention relates to the technical field of audio and video processing, and in particular to an unsupervised audio-visual segmentation method, system and electronic equipment. Background Art

[0002] Audiovisual segmentation is an advanced technology for audio-visual cross-modal analysis, which aims to extract the sound target from the video and generate a refined segmentation map of the sound target object. Audiovisual segmentation uses audio as a guiding signal to determine which object to segment and obtain its complete pixel-level mask map. Compared with the sound source localization task, this task not only locates the sound object based on the sound source, but also accurately segments the entire picture of the object that is making the sound in the video frame. This technology is widely used in speech recognition, audio separation, audio editing and other fields. In the fields of smart home, autonomous driving, medical image processing, robot navigation, etc., audiovisual segmentation technology also has broad application prospects.

[0003] Hearing and vision are the two most important sensors for humans to perceive the world. In life, sound signals and visual signals are often complementary. Taking advantage of the close relationship between audio and vision, audio-visual segmentation technology has made some progress recently, but there are still some challenges that need to be addressed. One of the main challenges is the high requirements for sound source localization results, which requires learning accurate pixel-level sound source maps rather than approximate image block-level localization maps. Existing sound source localization methods usually learn audio-visual correspondences through contrastive learning, and then directly calculate audio-visual attention to estimate the sound source area. These methods often output rough and unreliable sound source localization information, lacking detailed information descriptions such as object shape.

[0004] Another challenge is to perform fine-grained sound source localization in a completely unsupervised paradigm. In order to accurately identify the sound source area, supervised methods usually need to build a pixel-level annotation dataset and use a semantic segmentation model to generate pixel-level results. However, supervised methods require a large amount of annotation, but the annotation process is cumbersome and difficult to implement in practical environments. Summary of the invention

[0005] The main purpose of the embodiments of the present invention is to provide an unsupervised audio-visual segmentation method, system and electronic device, which can make the sound source target boundary information more accurate and make the generation of the sound source target segmentation map more stable and accurate.

[0006] To achieve the above objective, an embodiment of the present invention provides an unsupervised audio-visual segmentation method, comprising the following steps:

[0007] Obtain an audio-visual dataset and extract audio and visual features;

[0008] The features of the audio and visual modalities are mapped to a common feature space, and then the feature association of the two modalities is calculated through cross-attention to determine the coarse-grained positioning of the sound source target in the video frame;

[0009] Mining the neighborhood semantic association of visual features, constructing a semantic similarity matrix, and performing fine-grained correction on the sound source target segmentation to obtain the first sound source target segmentation map;

[0010] Based on interactive learning of audio and visual features, a second sound source target segmentation map is generated through a codec network;

[0011] The first sound source target segmentation map and the second sound source target segmentation map are aligned through a self-supervised learning training model to obtain an optimized final audio-visual segmentation result.

[0012] In some embodiments, the feature mapping of the features of the audio and visual modalities to a common feature space, and then calculating the feature association of the two modalities by cross attention to determine the coarse-grained positioning of the video frame to the sound source target includes the following steps:

[0013] A multi-layer perceptron is used to implement feature mapping to obtain visual feature representation and audio feature representation after feature mapping;

[0014] Through audio-visual cross-attention calculation, a coarse-grained sound source target segmentation map is obtained;

[0015] Normalize and binarize the attention matrix to obtain the sound source target mask matrix;

[0016] According to the preliminary attention calculation results, the visual features are divided into a feature set that matches the audio features semantically and a feature set that does not match them. The audio-visual matching features are paired to form a positive sample set, and the mismatched features are combined into a negative sample set. The positive and negative sample feature sets are learned through a comparative learning method to complete the audio-visual cross-modal semantic alignment.

[0017] When the features of the combined audio-visual matching are used as positive sample pairs, a random mask enhancement operation is added to the contrastive learning model. During the contrastive learning training process, the visual features of some positions in the positive sample combination are randomly discarded to determine the coarse-grained positioning of the sound source target in the video frame.

[0018] In some embodiments, the formula for cross-attention calculation is:

[0019]

[0020]

[0021] Where Q represents the query vector (Query) used to calculate the attention matrix; ω cRepresents the weight matrix that converts audio features into query vectors (Query); Represents the audio feature representation after feature mapping; K represents the key vector (Key) used for attention matrix calculation; ω k represents the weight matrix that converts audio features into key vectors (Key); V represents the value vector (Value) used for attention matrix calculation; ω v Represents the weight matrix that converts visual features into value vectors (Value); Represents the visual feature representation after feature mapping; A cs represents the attention matrix; d k represents the adjustment parameter;

[0022] The expression of the loss function of contrastive learning is:

[0023]

[0024] Among them, L cs represents the loss function of contrastive learning; τ is the temperature parameter; Θ() represents the random mask enhancement processing of visual features; k represents the kth non-sound source target neighborhood feature in the neighborhood set; j represents the jth sound source target neighborhood feature in the neighborhood set.

[0025] In some embodiments, mining the neighborhood semantic association of visual features, constructing a semantic similarity matrix, and performing fine-grained correction on the sound source target segmentation to obtain a first sound source target segmentation map includes the following steps:

[0026] A semantic similarity matrix is ​​constructed by calculating the feature similarity between neighborhood pixels; specifically, taking a certain coordinate center as a base point, setting a set of neighborhoods with a distance r from the base point in the coordinate space, calculating the semantic similarity value between neighborhood features i and j in the neighborhood set, and obtaining a semantic similarity matrix;

[0027] Predicting the semantic relevance between adjacent coordinate features in the video frame according to the semantic similarity matrix;

[0028] Construct a self-supervised learning model and transform the affinity matrix M A Set as the eigenvalue, and the sound source target segmentation map M c Set as the target matrix of feature mapping;

[0029] For the coarse-grained sound source target segmentation map M c Quantization operation is performed to generate a semantic association label matrix; specifically, the pixel-by-pixel adaptive mean shift algorithm is used to segment the sound source target map M c Perform pixel-by-pixel feature adjustment to obtain a more refined sound source target segmentation map M cf; Then set two thresholds α1 and α2 to quantify the sound source target segmentation map M cf , segment the sound source target map M cf Divide into reliable sound source area, silent area and uncertain area;

[0030] Generate pseudo semantic association labels Y based on the divided regions cA ; Specifically: according to the regional division matrix R cf , if the coordinate positions (x i ,y i ), (x j ,y j ) are located in the same semantic area, including the sound source area and the silent area, and their associated labels are set to the positive label Y cA (i, j) = y + ; If the neighborhood features i and j are located in different semantic areas of the sound source area and the silent area, their associated labels are set to negative labels Y cA (i, j) = y - , if any of the neighborhood features i and j are from the ignored region, their associated labels will be set to ignored;

[0031] The semantic similarity matrix and the sound source target segmentation map M are transformed into c Association, with semantic association label matrix Y cA is the objective function value, by minimizing the loss function L A , the training model generates an affinity matrix M that accurately represents the semantic relationship A ;

[0032] The learned affinity matrix M is transformed into A Converted into a semantic transition matrix T, and the coarse-grained sound source target segmentation map M is corrected according to the semantic transition matrix T c , and obtain the first sound source target segmentation map.

[0033] In some embodiments, the calculation formula of the semantic similarity matrix is:

[0034] M A (i, j) = exp(-||F aff (I(x i ,y i ))-F aff (I(x j ,y j ))|| L1 )

[0035] Among them, M A (i, j) represents the semantic similarity matrix of neighborhood features i and j; I(x i,y i ) indicates that (x i ,y i ) regional visual information of the position; I(x j ,y j ) indicates that (x i ,y i ) Regional visual information of the location; F aff () represents the calculation function of using multi-layer convolutional network and multi-layer perceptron to aggregate and map features; L1 represents 1-norm calculation;

[0036] The calculation formula of the semantic transition matrix T is:

[0037]

[0038] Among them, β represents the hyperparameter; D represents the A The diagonal matrix for normalization; D ii Represents the value of the i-th row and i-th column on the diagonal of the diagonal matrix D;

[0039] The expression of the refinement correction process of the coarse-grained audio-visual segmentation result is:

[0040]

[0041] Among them, vec(·) represents the vectorization of the matrix, M CR It is the sound source target segmentation map after fine-tuning and correction.

[0042] In some embodiments, the interactive learning based on audio-visual features and generating a second sound source target segmentation map through a codec network comprises the following steps:

[0043] The ResNet50 network is used to build a visual encoder, multi-scale visual features are obtained at different encoding layers of ResNet50, and the Atrous spatial pyramid pooling method is used to generate a feature set;

[0044] The final encoding layer of the ResNet50 network fuses the output features with the coarse-grained sound source target segmentation map to obtain the fused features;

[0045] The fusion feature and the multi-scale feature set are input into a pyramid decoder, and after encoding and decoding training, the decoder finally generates a second sound source target segmentation map;

[0046] The second sound source target segmentation map and the first sound source target segmentation map are configured to be supervisory information for each other, and the difference between the two is calculated using the L2 norm;

[0047] The calculation formula of the fusion feature is: av =Fv +F v ⊙norm(M c ),

[0048] Among them, F av represents fusion feature; F v represents visual feature representation; ⊙ represents Hadamard product; norm() refers to minimum-maximum normalization calculation; M c Represents the coarse-grained sound source target segmentation map.

[0049] In some embodiments, the step of aligning the first sound source target segmentation map and the second sound source target segmentation map through the self-supervised learning training model to obtain an optimized final audio-visual segmentation result includes the following steps:

[0050] Construct the loss of feature contrast learning for audio-visual matching, the mapping loss of semantic affinity matrix and sound source target segmentation map, and the self-supervised loss of sound source target segmentation map, and use the weighted combination of the three loss functions as the final loss function;

[0051] According to the final loss function, the first sound source target segmentation map and the second sound source target segmentation map are aligned with each other through a self-supervised learning training model to obtain an optimized final audio-visual segmentation result;

[0052] Among them, the expression of the final loss function L is: L = L cs +λ1L S +λ2L A ,

[0053] Among them, L cs represents the loss of feature contrast learning for audio-visual matching; L s represents the self-supervised loss of the sound source target segmentation map; L A represents the mapping loss between the semantic affinity matrix and the sound source target segmentation map; λ1 and λ2 represent weighting factors.

[0054] Another aspect of the present invention provides an unsupervised audio-visual segmentation system, comprising:

[0055] The first module is used to obtain audio-visual data sets and extract audio features and visual features;

[0056] The second module is used to map the features of the audio and visual modalities into a common feature space, and then calculate the feature association of the two modalities through cross-attention to determine the coarse-grained positioning of the sound source target in the video frame;

[0057] The third module is used to mine the neighborhood semantic association of visual features, construct a semantic similarity matrix, perform fine-grained correction on the sound source target segmentation, and obtain the first sound source target segmentation map;

[0058] The fourth module is used to generate a second sound source target segmentation map through a codec network according to interactive learning of audio and visual features;

[0059] The fifth module is used to align the first sound source target segmentation map and the second sound source target segmentation map through a self-supervised learning training model to obtain an optimized final audio-visual segmentation result.

[0060] To achieve the above objective, another aspect of an embodiment of the present invention provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0061] To achieve the above objective, another aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method is implemented.

[0062] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0063] The embodiments of the present invention include at least the following beneficial effects: the present invention provides an unsupervised audio-visual segmentation method, system and electronic device, which extracts audio features and visual features by acquiring an audio-visual data set; features of the audio and visual modalities are mapped to a common feature space, and then the feature association of the two modalities is calculated through cross-attention to determine the coarse-grained positioning of the video frame to the sound source target; neighborhood semantic associations of visual features are mined, a semantic similarity matrix is ​​constructed, and fine-grained corrections are made to the sound source target segmentation to obtain a first sound source target segmentation map; a second sound source target segmentation map is generated through a codec network according to interactive learning of audio-visual features; the first sound source target segmentation map and the second sound source target segmentation map are aligned through a self-supervised learning training model to obtain an optimized final audio-visual segmentation result. The embodiments of the present invention can make the boundary information of the sound source target more accurate, and make the generation of the sound source target segmentation map more stable and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0065] Figure 2 is a flow chart of overall steps provided by an embodiment of the present invention;

[0066] Figure 3 It is a flowchart of specific implementation steps provided by an embodiment of the present invention;

[0067] Figure 4 is a diagram of implementation steps of the cross-attention mechanism provided by an embodiment of the present invention;

[0068] Figure 5 It is a principle diagram of a fine-grained sound source target segmentation algorithm based on semantic similarity provided by an embodiment of the present invention;

[0069] Figure 6 It is a flow chart of generating a sound source target segmentation map through an audio-visual interactive coding and decoding model provided by an embodiment of the present invention;

[0070] Figure 7 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0072] It is understood that the terms "first", "second", etc. used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0073] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0074] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0075] The unsupervised audio-visual segmentation method, system and electronic device provided by the embodiment of the present invention relate to the field of audio and video processing technology. The unsupervised audio-visual segmentation method provided by the embodiment of the present invention can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements an unsupervised audio-visual segmentation method, etc., but is not limited to the above forms.

[0076] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0077] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0078] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0079] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0080] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 102 may also be a vehicle-mounted terminal of various device types described above, but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the present invention.

[0081] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides an unsupervised audio-visual segmentation method. The following is an example of the unsupervised audio-visual segmentation method being applied in the server 101. It can be understood that the method can also be applied in the terminal 102.

[0082] Reference Figure 2 , Figure 2 The flowchart of the unsupervised audio-visual segmentation method applied to the server provided by the embodiment of the present invention, the execution subject of the method can be any of the aforementioned computer devices (including servers or terminals). Figure 2 , the method may include the following steps:

[0083] Obtain an audio-visual dataset and extract audio and visual features;

[0084] The features of the audio and visual modalities are mapped to a common feature space, and then the feature association of the two modalities is calculated through cross-attention to determine the coarse-grained positioning of the sound source target in the video frame;

[0085] Mining the neighborhood semantic association of visual features, constructing a semantic similarity matrix, and performing fine-grained correction on the sound source target segmentation to obtain the first sound source target segmentation map;

[0086] Based on interactive learning of audio and visual features, a second sound source target segmentation map is generated through a codec network;

[0087] The first sound source target segmentation map and the second sound source target segmentation map are aligned through a self-supervised learning training model to obtain an optimized final audio-visual segmentation result.

[0088] In some embodiments, the feature mapping of the features of the audio and visual modalities to a common feature space, and then calculating the feature association of the two modalities by cross attention to determine the coarse-grained positioning of the video frame to the sound source target includes the following steps:

[0089] A multi-layer perceptron is used to implement feature mapping to obtain visual feature representation and audio feature representation after feature mapping;

[0090] Through audio-visual cross-attention calculation, a coarse-grained sound source target segmentation map is obtained;

[0091] Normalize and binarize the attention matrix to obtain the sound source target mask matrix;

[0092] According to the preliminary attention calculation results, the visual features are divided into a feature set that matches the audio features semantically and a feature set that does not match them. The audio-visual matching features are paired to form a positive sample set, and the mismatched features are combined into a negative sample set. The positive and negative sample feature sets are learned through a comparative learning method to complete the audio-visual cross-modal semantic alignment.

[0093] When the features of the combined audio-visual matching are used as positive sample pairs, a random mask enhancement operation is added to the contrastive learning model. During the contrastive learning training process, the visual features of some positions in the positive sample combination are randomly discarded to determine the coarse-grained positioning of the sound source target in the video frame.

[0094] In some embodiments, the formula for cross-attention calculation is:

[0095]

[0096]

[0097] Where Q represents the query vector (Query) used to calculate the attention matrix; ω c Represents the weight matrix that converts audio features into query vectors (Query); Represents the audio feature representation after feature mapping; K represents the key vector (Key) used for attention matrix calculation; ω k represents the weight matrix that converts audio features into key vectors (Key); V represents the value vector (Value) used for attention matrix calculation; ω v Represents the weight matrix that converts visual features into value vectors (Value); Represents the visual feature representation after feature mapping; Acs represents the attention matrix; d k represents the adjustment parameter;

[0098] The expression of the loss function of contrastive learning is:

[0099]

[0100] Among them, L cs represents the loss function of contrastive learning; τ is the temperature parameter; Θ() represents the random mask enhancement processing of visual features; k represents the kth non-sound source target neighborhood feature in the neighborhood set; j represents the jth sound source target neighborhood feature in the neighborhood set.

[0101] In some embodiments, mining the neighborhood semantic association of visual features, constructing a semantic similarity matrix, and performing fine-grained correction on the sound source target segmentation to obtain a first sound source target segmentation map includes the following steps:

[0102] A semantic similarity matrix is ​​constructed by calculating the feature similarity between neighborhood pixels; specifically, taking a certain coordinate center as a base point, setting a set of neighborhoods with a distance r from the base point in the coordinate space, calculating the semantic similarity value between neighborhood features i and j in the neighborhood set, and obtaining a semantic similarity matrix;

[0103] Predicting the semantic relevance between adjacent coordinate features in the video frame according to the semantic similarity matrix;

[0104] Construct a self-supervised learning model and transform the affinity matrix M A Set as the eigenvalue, and the sound source target segmentation map M c Set as the target matrix of feature mapping;

[0105] For the coarse-grained sound source target segmentation map M c Quantization operation is performed to generate a semantic association label matrix; specifically, the pixel-by-pixel adaptive mean shift algorithm is used to segment the sound source target map M c Perform pixel-by-pixel feature adjustment to obtain a more refined sound source target segmentation map M cf ; Then set two thresholds α1 and α2 to quantify the sound source target segmentation map M cf , segment the sound source target map M cf Divide into reliable sound source area, silent area and uncertain area;

[0106] Generate pseudo semantic association labels Y based on the divided regions cA ; Specifically: according to the regional division matrix R cf , if the coordinate positions (x i ,y i ), (x j ,yj ) are located in the same semantic area, including the sound source area and the silent area, and their associated labels are set to the positive label Y cA (i, j) = y + ; If the neighborhood features i and j are located in different semantic areas of the sound source area and the silent area, their associated labels are set to negative labels Y cA (i, j) = y - , if any of the neighborhood features i and j are from the ignored region, their associated labels will be set to ignored;

[0107] The semantic similarity matrix and the sound source target segmentation map M are transformed into c Association, with semantic association label matrix Y cA is the objective function value, by minimizing the loss function L A , the training model generates an affinity matrix M that accurately represents the semantic relationship A ;

[0108] The learned affinity matrix M is transformed into A Converted into a semantic transition matrix T, and the coarse-grained sound source target segmentation map M is corrected according to the semantic transition matrix T c , and obtain the first sound source target segmentation map.

[0109] In some embodiments, the calculation formula of the semantic similarity matrix is:

[0110] M A (i, j) = exp(-||F aff (I(x i ,y i ))-F aff (I(x j ,y j ))|| L1 )

[0111] Among them, M A (i, j) represents the semantic similarity matrix of neighborhood features i and j; I(x i ,y i ) indicates that (x i ,y i ) regional visual information of the position; I(x j ,y j ) indicates that (x i ,y i ) Regional visual information of the location; F aff () represents the calculation function of using multi-layer convolutional network and multi-layer perceptron to aggregate and map features; L1 represents 1-norm calculation;

[0112] The calculation formula of the semantic transition matrix T is:

[0113]

[0114] Among them, β represents the hyperparameter; D represents the A The diagonal matrix for normalization; D ii Represents the value of the i-th row and i-th column on the diagonal of the diagonal matrix D;

[0115] The expression of the refinement correction process of the coarse-grained audio-visual segmentation result is:

[0116]

[0117] Among them, vec(·) represents the vectorization of the matrix, M CR It is the sound source target segmentation map after fine-tuning and correction.

[0118] In some embodiments, the interactive learning based on audio-visual features and generating a second sound source target segmentation map through a codec network comprises the following steps:

[0119] The ResNet50 network is used to build a visual encoder, multi-scale visual features are obtained at different encoding layers of ResNet50, and the Atrous spatial pyramid pooling method is used to generate a feature set;

[0120] The final encoding layer of the ResNet50 network fuses the output features with the coarse-grained sound source target segmentation map to obtain the fused features;

[0121] The fusion feature and the multi-scale feature set are input into a pyramid decoder, and after encoding and decoding training, the decoder finally generates a second sound source target segmentation map;

[0122] The second sound source target segmentation map and the first sound source target segmentation map are configured to be supervisory information for each other, and the difference between the two is calculated using the L2 norm;

[0123] The calculation formula of the fusion feature is: av =F v +F v ⊙norm(M c ),

[0124] Among them, F av represents fusion feature; F v represents visual feature representation; ⊙ represents Hadamard product; norm() refers to minimum-maximum normalization calculation; M c Represents the coarse-grained sound source target segmentation map.

[0125] In some embodiments, the step of aligning the first sound source target segmentation map and the second sound source target segmentation map through the self-supervised learning training model to obtain an optimized final audio-visual segmentation result includes the following steps:

[0126] Construct the loss of feature contrast learning for audio-visual matching, the mapping loss of semantic affinity matrix and sound source target segmentation map, and the self-supervised loss of sound source target segmentation map, and use the weighted combination of the three loss functions as the final loss function;

[0127] According to the final loss function, the first sound source target segmentation map and the second sound source target segmentation map are aligned with each other through a self-supervised learning training model to obtain an optimized final audio-visual segmentation result;

[0128] Among them, the expression of the final loss function L is: L = L cs +λ i L S +λ2L A ,

[0129] Among them, L cs represents the loss of feature contrast learning for audio-visual matching; L s represents the self-supervised loss of the sound source target segmentation map; L A represents the mapping loss between the semantic affinity matrix and the sound source target segmentation map; λ1 and λ2 represent weighting factors.

[0130] The following takes a specific application scenario as an example to describe the implementation steps of the present invention in detail:

[0131] The present invention proposes an unsupervised audio-visual segmentation method to implement pixel-level segmentation and positioning of unsupervised sound source targets. In order to achieve pixel-level audio-visual segmentation, the present invention designs a novel multi-instance contrast learning method based on mask enhancement to align paired audio-visual features. The method effectively alleviates the problems of overfitting and suboptimal solutions, thereby better achieving audio-visual alignment. Next, the coarse-grained sound source target segmentation result at the image block level is obtained using the audio-visual attention mechanism. In order to further obtain the pixel-level sound source segmentation result, the present invention introduces an unsupervised sound source segmentation refinement module, which proposes visual semantic association learning to explore the relationship between pixels of adjacent coordinates and predict the semantic association of adjacent pixels in the image. The coarse-grained sound source segmentation map is corrected according to the association matrix to generate a fine-grained sound source segmentation map. The sound source target segmentation result is optimized through this operation, so that the boundary information of the sound source target is more accurate. In unsupervised audio-visual segmentation, due to the lack of the true segmentation value of the sound source target, the accuracy of the sound source segmentation map obtained in the previous step cannot be evaluated. Therefore, the present invention proposes an audio-visual fusion encoding and decoding algorithm to generate a pixel-level segmentation map of the sound source target. The algorithm is composed of an encoder-decoder combination, which generates a segmentation map of the sound source target under the guidance of the audio signal. The result is used to evaluate the contrastive learning segmentation result of the previous step. This measure can build a self-supervised learning system, improve the training effect of contrastive learning segmentation, make it more convergent, and make the generation of sound source target segmentation map more stable and accurate.

[0132] The following is combined with Figure 3 , the steps of the unsupervised audio-visual segmentation method of the embodiment of the present invention are described:

[0133] S101: audio and video data preprocessing.

[0134] Given an audio-visual dataset, with known video sequences and audio information D = {(v i , a i ), i = 1, ..., N}, the two respectively perform preliminary feature encoding on the audio and video data through the pre-trained audio encoder VGGish and visual encoder ResNet50, and extract the audio feature F a ={f a (i), i=1,…,N} and visual features F v ={f v (i), i=1,…,N}.

[0135] S102: Coarse-grained audio-visual segmentation based on cross-attention mechanism.

[0136] The implementation of coarse-grained audio-visual segmentation is achieved through the cross-attention mechanism. The specific algorithm principle is as follows Figure 4As shown in Figure 2, this module consists of two parts: feature mapping and cross-attention calculation. The mapping layer maps the features of the two modalities to a common feature space, making it easier to calculate the feature association of the two modalities through cross-attention and determine the visual area of ​​the sound source target.

[0137] Here, a multi-layer perceptron (MLP) is used to implement feature mapping, and the feature mapping process is represented by the function g(·). The visual features after feature mapping are expressed as The audio features after feature mapping are expressed as The audio-visual modal feature sequence is expressed as and

[0138] Through audio-visual cross attention calculation, this embodiment obtains a coarse-grained sound source target segmentation map. The cross attention calculation is shown in Formula 1. The sound source target mask matrix M c The attention matrix A cs Obtained after normalization and binarization operations.

[0139]

[0140] In order to calculate an accurate sound source target segmentation map through the cross-attention mechanism, it is necessary to semantically align visual features and audio features. The present invention designs a multi-instance contrastive learning method to align audio-visual semantic features. Specifically, according to the preliminary attention calculation results, the visual features are divided into a feature set that matches the audio features semantically and a feature set that does not match. The audio-visual matching features are paired to form a positive sample set, and the unmatched features are combined into a negative sample set. Contrastive learning is used to learn the positive and negative sample feature sets to promote cross-modal semantic alignment of audio and vision.

[0141] When combining audio-visual matching features as positive sample pairs, in order to improve the generalization of matching audio features with the visual positions of sound source targets and more accurately locate all sound source target positions by audio, the present invention further adds a random mask enhancement operation in the contrastive learning model, that is, randomly discarding the visual features of some positions in the positive sample combination during the contrastive learning training process to ensure that the potential sound source target positions have more optimization opportunities. The random mask enhancement processing of visual features is expressed as The loss function setting of the improved contrastive learning is shown in Formula 2.

[0142]

[0143] Among them, τ is the temperature parameter, and the similarity between audio features and visual features is calculated by cosine distance.

[0144] S103: Fine-grained audio-visual segmentation based on semantic similarity.

[0145] In order to further obtain pixel-level sound source segmentation results, the present invention designs a fine-grained sound source target segmentation module based on semantic similarity. This module uses visual semantic association learning to explore the semantic relationship between pixels of adjacent coordinates, and corrects the coarse-grained sound source target segmentation map according to the semantic association matrix of adjacent pixels in the video frame to generate a fine-grained sound source target segmentation result.

[0146] Specifically, a semantic similarity matrix is ​​constructed to predict the semantic relevance between adjacent coordinate features in the video frame. The semantic similarity matrix is ​​used to segment the sound source target map M c Aligning,corrected coarse sound source maps via self-supervised learning.

[0147] Figure 5 The figure shows the principle diagram of the fine-grained sound source target segmentation algorithm based on semantic similarity. In this embodiment, a semantic similarity matrix M is set A , the matrix value is obtained by calculating the feature similarity between neighboring pixels. The specific implementation method is to extract the visual features F v The multi-layer perceptron (MLP) is input to perform feature aggregation calculation, and the calculated features are used to mine the similarity relationship between neighboring pixels.

[0148] Specifically, a set of neighborhoods with a distance r from a certain coordinate center in the coordinate space is set. The semantic similarity between neighborhood features i and j in the neighborhood set is calculated, denoted as M A (i, j), that is, the semantic similarity matrix. Assume that the coordinate positions of neighborhood features i and j are represented by (x i ,y i ), (x j ,y j ) indicates that the calculation formula of the semantic similarity matrix is ​​as follows.

[0149] M A (i, j) = exp(-||F aff (I(x i ,y i ))-F aff (I(x j ,y j ))|| L1 ) (3)

[0150] In order to more accurately transform the semantic similarity matrix M A and sound source target segmentation map M c This embodiment constructs a self-supervised learning model, in which the affinity matrix M A Set as the eigenvalue, and the sound source target segmentation map M cSet as the target matrix of the feature map. To quantify the target value, this embodiment further performs a coarse-grained sound source target segmentation map M c Perform quantization operations to generate the semantic association label matrix Y cA The specific operation is to first use the pixel-wise adaptive mean shift algorithm (PAMR) to segment the sound source target map M c Perform pixel-by-pixel feature adjustment to obtain a more refined matrix M cf Then, two thresholds α1 and α2 are set (0<α1<α2<1) to quantify the sound source target segmentation map M cf , it is divided into reliable sound source area, silent area and uncertain area. The division rule is defined as formula 4.

[0151]

[0152] Among them, 1 and 0 represent the sound source area and the silent area respectively, and Null represents the ignored area.

[0153] According to the divided areas, this embodiment generates a pseudo semantic association label Y cA The specific operation is to divide the matrix R according to the region cf , if the coordinate positions (x i ,y i ), (x j ,y j ) are located in the same semantic area, including the sound source area and the silent area, and their associated labels are set to the positive label Y cA (i, j) = y + Otherwise, the neighborhood features i and j are located in different semantic areas of the sound source area and the silent area, respectively, and their associated labels are set to negative labels Y cA (i, j) = y - If any of the neighborhood features i and j are from the ignored region, their associated labels will be set to ignored.

[0154] The present invention uses a self-supervised learning method to transform the semantic similarity matrix M A and sound source target segmentation map M e Association, with semantic association label matrix Y cA is the objective function value, by minimizing the loss function L A , the training model generates an affinity matrix M that accurately represents the semantic relationship A The loss function L A As defined in Equation 5.

[0155] L A =-y + ·log(MA )-y - ·log(1-M A ) (5)

[0156] Semantic affinity matrix M A It is used to refine and correct the sound source target segmentation map of the coarse-grained audio-visual segmentation result obtained in step S102.

[0157] The operation of refining the sound source target segmentation map is achieved through random walk. A Here, the random walk algorithm is used to convert it into a semantic transition matrix T, which is used to correct the coarse-grained audio-visual segmentation result M c The calculation method of the semantic transition matrix T is shown in Formula 6. Where β is a hyperparameter used to ignore M A The smaller value of the correlation value in M ​​is used, and D is a diagonal matrix for A Perform normalization.

[0158] T=D -1 (M A ) β , D ii =∑ j (M A (i, j)) β (6)

[0159] Coarse-grained audio-visual segmentation result M C The refinement correction process of is shown in Formula 7.

[0160]

[0161] Among them, vec(·) represents the vectorization of the matrix, M CR It is the sound source target segmentation map after fine correction. This correction operation can activate semantic areas with high correlation and suppress the wrongly activated areas, so that the sound source target segmentation map is better aligned with the semantic boundary.

[0162] S104: generate a sound source target segmentation map by audio-visual interactive coding and decoding.

[0163] S103 implementation step corrects the coarse-grained sound source target segmentation map by analyzing the neighborhood semantic similarity to obtain a fine-grained sound source target segmentation map. The present invention is implemented in an unsupervised manner, and the result evaluation of the fine-grained sound source target segmentation map lacks a real value for comparison. Therefore, Figure 6 As shown, this embodiment further designs an audio-visual interactive coding model to generate a sound source target segmentation map. The segmentation result and the fine-grained sound source target segmentation map obtained in step S103 serve as supervisory information for model training in a self-supervised learning manner.

[0164] P1: Audio-Visual Interaction Encoder

[0165] like Figure 6 As shown in the figure, the visual encoder adopts the ResNet50 network, obtains multi-scale visual features in different encoding layers of ResNet50, and uses the Atrous Spatial Pyramid Pooling (ASPP) method to generate a feature set {F1, F2, F3, F4} for subsequent feature processing.

[0166] On the other hand, the final encoding layer of the ResNet50 network outputs feature F v and the coarse-grained sound source target segmentation matrix M obtained in step S102 c Perform feature fusion to obtain fusion feature F av .

[0167] F av =F v +F v ⊙norm(M c ) (8)

[0168] Here, norm refers to the minimum-maximum normalization calculation, and ⊙ represents the Hadamard product.

[0169] P2: Decode and generate the sound source target segmentation map.

[0170] In this embodiment, the audio-visual fusion features and the multi-scale feature set are input into the pyramid decoder (Feature Pyramid Networks, FPN). After encoding and decoding training, the decoder finally generates the sound source target segmentation map M CG .

[0171] The sound source target segmentation map M generated by audio-visual interactive decoding CG The fine-grained sound source target segmentation map obtained in step S103 is used as supervisory information for each other, and the difference between the two is calculated using the L2 norm. The loss function is defined as shown in Formula 9.

[0172] L S =||M CG -M CR || L2 (9)

[0173] S105: Loss setting for training model.

[0174] The method proposed in the present invention includes three loss terms: the loss L of feature contrast learning for audio-visual matching cs , the mapping loss L between the semantic affinity matrix and the sound source target segmentation mapA , and the sound source target segmentation map self-supervision loss L S The weighted combination of the three loss functions is the final loss function of this embodiment, as shown in Formula 10.

[0175] L=L cs +λ1L S +λ2L A (10)

[0176] Among them, λ1 and λ2 are used as weighting factors to balance the contributions of different loss functions.

[0177] The unsupervised audio-visual segmentation algorithm model proposed in the present invention is used to segment a complete sound source target area from a video frame after sufficient model training.

[0178] In order to better illustrate the technical effect of the present invention, specific sample examples are collected to experimentally verify the present invention. The present invention has carried out a large number of experiments on two data sets, AVSBench-S4 and VGGSS, and the experimental results prove the effectiveness of our proposed method. This embodiment selects three methods for comparative experiments to compare the audio-visual segmentation effects. The three comparative methods are: FNAC (false negative aware contrastive learning), EZ-VSL (Easy Visual Sound Localization), and LVS (Localizing Visual Sounds). Four evaluation indicators, Mean Intersection over Union (mloU), F-score, Consensus Intersection over Union (cloU) and Area Under Curye (AUC), are used to evaluate the algorithm segmentation effect. The higher the values ​​of the four indicators, the better the algorithm segmentation effect.

[0179] Table 1 is a comparison table of the unsupervised audio-visual sound source target segmentation results of the present invention and the comparative method on the audio-visual dataset AVSBench-S4.

[0180] Table 1

[0181] Methods mloU(%) F-score FNAC 32.46 0.434 EZ-VSL 34.01 0.562 LVS 37.94 0.510 Method of the present invention 47.19 0.617

[0182] Table 2 is a comparison table of unsupervised audio-visual sound source target segmentation of the present invention and the comparative method on the audio-video dataset VGGSS.

[0183] Table 2

[0184] Methods cIoU(%) AUC (%) LVS 34.40 38.20 EZ-VSL 34.38 37.70 FNAC 39.50 39.66 Method of the present invention 44.47 45.11

[0185] As shown in Table 1 and Table 2, compared with these algorithms, the final segmentation result of the unsupervised audio-visual segmentation algorithm of the present invention exceeds the existing most advanced unsupervised method by 9.25% in the mIoU index on the AVSBench-S4 dataset, and increases the cIoU index on the VGGSS benchmark by 2.62%. The experimental results fully demonstrate the effectiveness of the method of the present invention.

[0186] Another aspect of the present invention provides an unsupervised audio-visual segmentation system, comprising:

[0187] The first module is used to obtain audio-visual data sets and extract audio features and visual features;

[0188] The second module is used to map the features of the audio and visual modalities into a common feature space, and then calculate the feature association of the two modalities through cross-attention to determine the coarse-grained positioning of the sound source target in the video frame;

[0189] The third module is used to mine the neighborhood semantic association of visual features, construct a semantic similarity matrix, perform fine-grained correction on the sound source target segmentation, and obtain the first sound source target segmentation map;

[0190] The fourth module is used to generate a second sound source target segmentation map through a codec network according to interactive learning of audio and visual features;

[0191] The fifth module is used to align the first sound source target segmentation map and the second sound source target segmentation map through a self-supervised learning training model to obtain an optimized final audio-visual segmentation result.

[0192] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0193] The embodiment of the present invention further provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned unsupervised audio-visual segmentation method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0194] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0195] See also Figure 7 , Figure 7 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0196] The processor 701 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0197] The memory 702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 702, and the processor 701 calls and executes the unsupervised audio-visual segmentation method of the embodiment of the present invention;

[0198] Input / output interface 703, used to implement information input and output;

[0199] Communication interface 704, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0200] A bus 705 that transmits information between the various components of the device (e.g., the processor 701, the memory 702, the input / output interface 703, and the communication interface 704);

[0201] The processor 701 , the memory 702 , the input / output interface 703 and the communication interface 704 are connected to each other in communication within the device via a bus 705 .

[0202] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned unsupervised audio-visual segmentation method is implemented.

[0203] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0204] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0205] It should be noted that in various specific embodiments of the present invention, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present invention needs to obtain the user's sensitive personal information, it will obtain the user's separate permission or consent through a pop-up window or jump to a confirmation page, and after clearly obtaining the user's separate permission or consent, it will obtain the necessary user-related data for the normal operation of the embodiment of the present invention.

[0206] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0207] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0208] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0209] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0210] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0211] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0212] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0213] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0214] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0215] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0216] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. An unsupervised audio-visual segmentation method, characterized in that The following steps are involved: Obtain an audio-visual dataset and extract audio and visual features; The features of the audio and visual modalities are mapped to a common feature space, and then the feature association of the two modalities is calculated through cross-attention to determine the coarse-grained positioning of the sound source target in the video frame; Mining the neighborhood semantic association of visual features, constructing a semantic similarity matrix, and performing fine-grained correction on the sound source target segmentation to obtain the first sound source target segmentation map; Based on interactive learning of audio and visual features, a second sound source target segmentation map is generated through a codec network; The first sound source target segmentation map and the second sound source target segmentation map are aligned through a self-supervised learning training model to obtain an optimized final audio-visual segmentation result.

2. The unsupervised audio-visual segmentation method according to claim 1, characterized in that: The feature mapping of the audio and visual features to a common feature space, and then calculating the feature association of the two modalities through cross attention to determine the coarse-grained positioning of the sound source target by the video frame, includes the following steps: A multi-layer perceptron is used to implement feature mapping to obtain visual feature representation and audio feature representation after feature mapping; Through audio-visual cross-attention calculation, a coarse-grained sound source target segmentation map is obtained; Normalize and binarize the attention matrix to obtain the sound source target mask matrix; According to the preliminary attention calculation results, the visual features are divided into a feature set that matches the audio features semantically and a feature set that does not match them. The audio-visual matching features are paired to form a positive sample set, and the mismatched features are combined into a negative sample set. The positive and negative sample feature sets are learned through a comparative learning method to complete the audio-visual cross-modal semantic alignment. When the features of the combined audio-visual matching are used as positive sample pairs, a random mask enhancement operation is added to the contrastive learning model. During the contrastive learning training process, the visual features of some positions in the positive sample combination are randomly discarded to determine the coarse-grained positioning of the sound source target in the video frame.

3. The unsupervised audio-visual segmentation method according to claim 2, characterized in that: The formula for calculating cross attention is: Where Q represents the query vector used for attention matrix calculation; ω c Represents the weight matrix that transforms audio features into query vectors; represents the audio feature representation after feature mapping; K represents the key vector used for attention matrix calculation; ω k represents the weight matrix that converts audio features into key vectors; V represents the value vector used for attention matrix calculation; ω v Represents the weight matrix that converts visual features into value vectors; Represents the visual feature representation after feature mapping; A cs represents the attention matrix; d k represents the adjustment parameter; The loss function of contrastive learning is expressed as: Among them, L cs represents the loss function of contrastive learning; τ is the temperature parameter; Θ() represents the random mask enhancement processing of visual features; k represents the kth non-sound source target neighborhood feature in the neighborhood set; j represents the jth sound source target neighborhood feature in the neighborhood set.

4. The unsupervised audio-visual segmentation method according to claim 1, characterized in that: The method of mining the neighborhood semantic association of visual features, constructing a semantic similarity matrix, and performing fine-grained correction on the sound source target segmentation to obtain a first sound source target segmentation map includes the following steps: A semantic similarity matrix is ​​constructed by calculating the feature similarity between neighborhood pixels; specifically, taking a certain coordinate center as a base point, setting a set of neighborhoods with a distance r from the base point in the coordinate space, calculating the semantic similarity value between neighborhood features i and j in the neighborhood set, and obtaining a semantic similarity matrix; Predicting the semantic relevance between adjacent coordinate features in the video frame according to the semantic similarity matrix; Construct a self-supervised learning model and transform the affinity matrix M A Set as the eigenvalue, and the sound source target segmentation map M c Set as the target matrix of feature mapping; For the coarse-grained sound source target segmentation map M c Quantization operation is performed to generate a semantic association label matrix; specifically, the pixel-by-pixel adaptive mean shift algorithm is used to segment the sound source target map M c Perform pixel-by-pixel feature adjustment to obtain a more refined sound source target segmentation map M cf ; Then set two thresholds α1 and α2 to quantify the sound source target segmentation map M cf , segment the sound source target map M cf Divide into reliable sound source area, silent area and uncertain area; Generate pseudo semantic association labels Y based on the divided regions cA ; Specifically: according to the regional division matrix R cf , if the coordinate positions (x i ,y i ),(x j ,y j ) are located in the same semantic area, including the sound source area and the silent area, and their associated labels are set to the positive label Y cA (i, j) = y + ; If the neighborhood features i and j are located in different semantic areas of the sound source area and the silent area, their associated labels are set to negative labels Y cA (i, j) = y - , if any of the neighborhood features i and j are from the ignored region, their associated labels will be set to ignored; The semantic similarity matrix and the sound source target segmentation map M are transformed into c Association, with semantic association label matrix Y cA is the objective function value, by minimizing the loss function L A , the training model generates an affinity matrix M that accurately represents the semantic relationship A ; The learned affinity matrix M is transformed into A Converted into a semantic transition matrix T, and the coarse-grained sound source target segmentation map M is corrected according to the semantic transition matrix T c , and obtain the first sound source target segmentation map.

5. The unsupervised audio-visual segmentation method according to claim 4, characterized in that: The calculation formula of the semantic similarity matrix is: M A (i,j)=exp(-||F aff (I(x i ,y i ))-F aff (I(x j ,y j ))|| L1 ) Among them, M A (i, j) represents the semantic similarity matrix of neighborhood features i and j; I(x i ,y i ) indicates that (x i ,y i ) regional visual information of the position; I(x j ,y j ) indicates that (x j ,y j ) Regional visual information of the location; F aff () represents the calculation function of using multi-layer convolutional network and multi-layer perceptron to aggregate and map features; L1 represents 1-norm calculation; The calculation formula of the semantic transition matrix T is: Among them, β represents the hyperparameter; D represents the A The diagonal matrix for normalization; D ii Represents the value of the i-th row and i-th column on the diagonal of the diagonal matrix D; The expression of the refinement correction process of the coarse-grained audio-visual segmentation result is: Among them, vec(·) represents the vectorization of the matrix, M CR It is the sound source target segmentation map after fine-tuning and correction.

6. The unsupervised audio-visual segmentation method according to claim 1, characterized in that: The method of generating a second sound source target segmentation map through a codec network according to interactive learning of audio and video features includes the following steps: The ResNet50 network is used to build a visual encoder, multi-scale visual features are obtained at different encoding layers of ResNet50, and the Atrous spatial pyramid pooling method is used to generate a feature set; The final encoding layer of the ResNet50 network fuses the output features with the coarse-grained sound source target segmentation map to obtain the fused features; The fusion feature and the multi-scale feature set are input into a pyramid decoder, and after encoding and decoding training, the decoder finally generates a second sound source target segmentation map; The second sound source target segmentation map and the first sound source target segmentation map are configured to be supervisory information for each other, and the difference between the two is calculated using the L2 norm; The calculation formula of the fusion feature is: av =F v +F v ⊙norm(M c ), Among them, F av represents fusion feature; F v represents visual feature representation; ⊙ represents Hadamard product; norm() refers to minimum-maximum normalization calculation; M c Represents the coarse-grained sound source target segmentation map.

7. The unsupervised audio-visual segmentation method according to claim 1, characterized in that: The method of aligning the first sound source target segmentation map and the second sound source target segmentation map through the self-supervised learning training model to obtain an optimized final audio-visual segmentation result includes the following steps: Construct the loss of feature contrast learning for audio-visual matching, the mapping loss of semantic affinity matrix and sound source target segmentation map, and the self-supervised loss of sound source target segmentation map, and use the weighted combination of the three loss functions as the final loss function; According to the final loss function, the first sound source target segmentation map and the second sound source target segmentation map are aligned with each other through a self-supervised learning training model to obtain an optimized final audio-visual segmentation result; Among them, the expression of the final loss function L is: L = L cs +λ1L s +λ2L A , Among them, L cs represents the loss of feature contrast learning for audio-visual matching; L s represents the self-supervised loss of the sound source target segmentation map; L a represents the mapping loss between the semantic affinity matrix and the sound source target segmentation map; λ1 and λ2 represent weighting factors.

8. An unsupervised audio-visual segmentation system, characterized in that include: The first module is used to obtain audio-visual data sets and extract audio features and visual features; The second module is used to map the features of the audio and visual modalities into a common feature space, and then calculate the feature association of the two modalities through cross-attention to determine the coarse-grained positioning of the sound source target in the video frame; The third module is used to mine the neighborhood semantic association of visual features, construct a semantic similarity matrix, perform fine-grained correction on the sound source target segmentation, and obtain the first sound source target segmentation map; The fourth module is used to generate a second sound source target segmentation map through a codec network according to interactive learning of audio and visual features; The fifth module is used to align the first sound source target segmentation map and the second sound source target segmentation map through a self-supervised learning training model to obtain an optimized final audio-visual segmentation result.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 7.

Images