Audio-video association target analysis method and system fusing multi-modal scene understanding

Abstract

The application discloses an audio-video associated target analysis method and system fusing multi-modal scene understanding, a terminal and a medium, and relates to the technical field of audio-video target analysis. The method comprises the following steps: extracting a spatial feature map from a video frame image; converting an audio signal into a global audio embedding vector; performing cosine similarity calculation on the spatial feature map and the audio embedding vector at each spatial position to generate an audio thermal response map; generating a foreground mask and a background mask from the audio thermal response map; performing weighted summation on the spatial feature map by using the foreground mask and the background mask respectively, and then performing global average pooling to obtain a foreground visual representation vector and a background visual representation vector of a response region; converting the audio thermal response map into a structured sound-emitting target output, and outputting a response map and a spatial position coordinate of a target occurrence entity in the video frame image. The method can provide visual semantic information through a multi-modal large model, can effectively understand the semantic state of whether to emit sound, and can improve the accuracy of sound-emitting target recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio-video target analysis, and particularly relates to an audio-video associated target analysis method, system, terminal and medium fusing multi-modal scene understanding. BACKGROUND

[0002] With the development of multi-modal understanding technology, audio-video linkage analysis shows great potential in tasks such as monitoring perception, voice-driven interaction and video retrieval. Traditional audio-video sound source positioning methods mostly rely on visual and audio feature similarity calculation, and generate a spatial response map through audio guidance to estimate which areas in the image may correspond to the audio signal. Such methods often use attention mechanisms or explicit similarity calculation for cross-modal association, but have key bottlenecks such as weak semantic level understanding ability, poor target differentiation ability, and difficulty in modeling semantic ambiguity and occlusion in complex scenes.

[0003] Especially when there are multiple appearance similar targets (such as multiple musical instruments and multiple speakers) or silent pseudo targets (such as objects that are not making sounds but are conspicuous), traditional methods cannot determine which object is actually making a sound, and lack a clear semantic recognition mechanism for the sound-making behavior itself. In addition, current methods mostly use hard contrast loss or region cross constraint, which often fails under conditions of semantic uncertainty or multi-source reverberation. Existing methods lack precise modeling of the semantic state of the sound-making target, making it difficult to achieve truly explainable and robust cross-modal positioning. SUMMARY

[0004] The purpose of the present application is to provide an audio-video associated target analysis method, system, terminal and medium fusing multi-modal scene understanding, which solves the problem of inaccurate sound-making target recognition in multi-target scenes in the prior art.

[0005] The present application is implemented by the following technical solutions:

[0006] In a first aspect, the present application provides an audio-video associated target analysis method fusing multi-modal scene understanding, comprising:

[0007] obtaining a video frame image containing multiple people and multiple objects and a synchronous audio signal collected therefrom;

[0008] extracting a spatial feature map from the video frame image using a visual encoder, the spatial feature map retaining visual semantic information of each spatial position, the visual semantic information being provided by a multi-modal large model, and inputting the visual semantic information into a text encoder to obtain a foreground semantic vector set and a background semantic vector set;

[0009] converting the audio signal into an audio spectrum map, and inputting the audio spectrum map into an audio encoder to output a global audio embedding vector;

[0010] The spatial feature map and the audio embedding vector are subjected to cosine similarity calculation at each spatial position to generate an audio heat response map;

[0011] The foreground mask and the background mask are generated after the audio heat response map is normalized, and the foreground visual representation vector and the background visual representation vector of the response region are obtained by weighting and summing the spatial feature map using the foreground mask and the background mask respectively, and then performing global average pooling.

[0012] The audio heat response map is converted into a structured sound target output, and the response map and the spatial position coordinates of the target generating entity in the video frame image are output.

[0013] Further, the specific method of converting the audio heat response map into a structured sound target output comprises:

[0014] The existing object region proposal in the video frame image is used to perform regional weighted evaluation with the corresponding heat response map;

[0015] The heat center, response mean or maximum value is extracted as a regional score index, and the regional score is calculated according to the regional score index;

[0016] The candidate target region with the highest regional score is selected as the sound target.

[0017] Further, the method further comprises: when the multi-modal large model generates semantic supervision information, the multi-modal large model provides a structured semantic vector for constructing a discriminative target function for the sound target, specifically comprising:

[0018] The language prompt word, the video frame image and the corresponding audio category label are input into the multi-modal large model to generate foreground sound semantics and background silence semantics of the sound target, and the foreground sound semantics and the background silence semantics are input into a text encoder to output a foreground semantic vector set and a background semantic vector set as a supervision signal of a training loss.

[0019] Further, the method further comprises: when the multi-modal large model generates semantic supervision information, a soft contrast loss function is used to compare and learn the foreground semantic vector set and the background semantic vector set output by the text encoder with the foreground visual representation vector and the background visual representation vector, and the positive sample similarity term of the soft contrast loss function is:

[0020] ;

[0021] Wherein, the foreground visual representation vector is the foreground semantic vector of the i th sample is

[0022] ​Semantically weighted negative sample item is:

[0023] ;

[0024] wherein, is a background visual representation vector, is used to measure the similarity between the foreground semantic vector of the i-th sample and the foreground semantic vector of the j-th sample ;

[0025] The calculation formula of is:

[0026] ;

[0027] wherein, denotes the cosine similarity between the foreground semantic vector of the i-th sample and the foreground semantic vector of the j-th sample ;

[0028] The foreground loss The calculation formula of is:

[0029] ;

[0030] The background loss The calculation formula of is:

[0031] ;

[0032] wherein, is a background semantic vector;

[0033] The semantic-guided alignment loss The calculation formula of is:

[0034] ;

[0035] wherein, B denotes the number of samples in each training batch, and is used to normalize and average the foreground loss and the background loss .

[0036] Further, when generating semantic supervision information by using a multi-modal large model, the foreground semantic vector of the k-th sample is used to generate a corresponding audio heat response map , the audio heat response map is normalized to obtain a normalized response map , the foreground semantic vector of the j-th sample is used to generate a corresponding audio heat response map , and the audio heat response map The normalized response map is obtained through normalization processing , and is optimized through an adversarial loss of the Wasserstein distance and the region discriminator, wherein the Wasserstein decoupling loss is calculated according to the following formula:

[0037] ;

[0038] wherein, is an approximate Wasserstein distance calculated based on a Sinkhorn algorithm; K is the number of sound-uttering semantic vectors contained in the current video frame image;

[0039] The adversarial loss of the region discriminator is calculated according to the following formula:

[0040] ;

[0041] wherein, is a region discriminator, used to determine whether the input normalized response map corresponds to a real sound-uttering target region; is a region discriminator, used to determine whether the input normalized response map corresponds to a real sound-uttering target region;

[0042] The Wasserstein decoupling loss and the adversarial loss of the region discriminator are combined to obtain a spatial response decoupling loss , and the calculation formula is as follows:

[0043] ;

[0044] wherein, is the weight of the Wasserstein decoupling loss, is the weight of the adversarial loss of the region discriminator;

[0045] The final training loss is obtained according to the spatial response decoupling loss and the semantic-guided alignment loss, and the calculation formula is as follows:

[0046] ;

[0047] wherein, is the weight of the semantic-guided alignment loss, is the weight of the spatial response decoupling loss.

[0048] In a second aspect, another embodiment of the present application provides an audio-video associated target analysis system for fusing multi-modal scene understanding, which is used to implement the method of the first aspect. The system comprises a signal acquisition module, an image processing module, an audio processing module, a response map generation module, a visual representation vector generation module, and an output module.

[0049] The signal acquisition module is configured to acquire video frame images containing multiple people and multiple objects and audio signals synchronously collected therefrom.

[0050] The image processing module is configured to extract a spatial feature map from the video frame images using a visual encoder, wherein the spatial feature map retains visual semantic information of each spatial position, the visual semantic information is provided by a multi-modal large model, and the visual semantic information is input into a text encoder to obtain a foreground semantic vector set and a background semantic vector set.

[0051] The audio processing module is configured to convert the audio signals into an audio spectrum map, and input the audio spectrum map into an audio encoder to output a global audio embedding vector.

[0052] The response map generation module is configured to perform cosine similarity calculation on the spatial feature map and the audio embedding vector at each spatial position to generate an audio heat response map.

[0053] The visual representation vector generation module is configured to generate a foreground mask and a background mask from the audio heat response map after normalization, and then perform weighted summation on the spatial feature map using the foreground mask and the background mask respectively, and then perform global average pooling to obtain a foreground visual representation vector and a background visual representation vector of a response region.

[0054] The output module is configured to convert the audio heat response map into a structured sound-emitting target output, and output a response map and a spatial position coordinate of a target occurrence entity in the video frame image.

[0055] Further, the output module comprises a conversion unit configured to perform regional weighted evaluation on the existing object region proposal in the video frame image and the corresponding heat response map, extract a heat center, a response average or a maximum value as a regional score index, calculate a regional score according to the regional score index, and select a candidate target region with the highest regional score as a sound-emitting target.

[0056] Further, the system further comprises a model training module configured to generate a structured semantic vector using a multi-modal large model when the multi-modal large model generates semantic supervision information, and use the structured semantic vector to construct a discriminative target function for the sound-emitting target, specifically comprising:

[0057] The language prompt word, the video frame image and the corresponding audio category label are input into a multi-modal large model to generate foreground sound semantics and background silence semantics of the sound target, and the foreground sound semantics and the background silence semantics are input into a text encoder to output a foreground semantic vector set and a background semantic vector set as a supervision signal of a training loss.

[0058] In a third aspect, another embodiment of the present application provides a computer terminal, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to the first aspect when executing the program.

[0059] In a fourth aspect, another embodiment of the present application provides a computer readable medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the method according to the first aspect.

[0060] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0061] The method, system, terminal and medium provided by the embodiment of the present application can effectively understand the semantic state of whether to make a sound through the visual semantic information provided by the multi-modal large model, and improve the accuracy of sound target recognition. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be considered as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor. In the drawings:

[0063] Figure 1 A flowchart of a method for analyzing an audio-video associated target by fusing multi-modal scene understanding is provided for the first embodiment of the present application;

[0064] Figure 2 A general flowchart of a method for analyzing an audio-video associated target by fusing multi-modal scene understanding is provided for the first embodiment of the present application;

[0065] Figure 3 A structural block diagram of a system for analyzing an audio-video associated target by fusing multi-modal scene understanding is provided for another embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the objects, technical solutions and advantages of the present application clearer, further detailed description of the present application will be given below in combination with embodiments and drawings, the illustrative embodiments of the present application and the description thereof are only used to explain the present application, and do not limit the present application.

[0067] Embodiment 1

[0068] As shown in the first embodiment of the present application, a method for fusing multi-modal scene understanding to analyze audio-video associated targets is provided, which comprises the following steps: Figure 1

[0069] Obtaining video frame images containing multiple persons and multiple objects and audio signals synchronously collected therefrom;

[0070] Extracting spatial feature maps from the video frame images by using a visual encoder, wherein the spatial feature maps reserve visual semantic information of each spatial position, the visual semantic information is provided by a multi-modal large model, and the visual semantic information is input into a text encoder to obtain a foreground semantic vector set and a background semantic vector set;

[0071] Converting the audio signals into audio spectrum maps, and inputting the audio spectrum maps into an audio encoder to output global audio embedding vectors;

[0072] Calculating cosine similarity of the spatial feature maps and the audio embedding vectors at each spatial position to generate an audio heat response map;

[0073] Generating foreground masks and background masks from the audio heat response map after normalization, and then performing weighted summation on the spatial feature maps by using the foreground masks and the background masks respectively, and then performing global average pooling to obtain foreground visual representation vectors and background visual representation vectors of response regions;

[0074] Converting the audio heat response map into a structured sound-emitting target output, and outputting a response map and spatial position coordinates of a target occurrence entity in the video frame images.

[0075] Specifically, the obtained data is a video segment containing multiple persons and multiple objects and audio signals synchronously collected therefrom, the image frames are in RGB format, and the audio is single-channel waveform data with a sampling rate of 16 kHz. The input image is , wherein B represents the number of images processed at the same time, H is the vertical pixel number of the image, W is the horizontal pixel number of the image, and 3 is the color channel number of the image. The spatial feature map is extracted by a deep convolutional network , and C is the color channel number of the image. The feature map reserves visual semantic information of each spatial position and establishes a spatial response basis for subsequent matching with the audio.

[0076] ​The audio segment collected synchronously with the video frame image is processed, converted into a log spectrum graph, and then input into an audio encoder (such as AudioCLIP) to output a global audio embedding vector . The audio embedding vector represents the semantic information of the current audio content and serves as the modal guidance basis for calculating the visual response map.

[0077] As shown in Figure 2 , a semantic supervision process is introduced in the training stage, which is implemented based on a multi-modal large model (MLLM). Specifically, the language prompt, the video frame, and the corresponding audio category label are constructed as an input prompt and input into a multi-modal large model (such as InternVL or Qwen-VL) to generate two types of natural language descriptions: foreground sound semantic (such as the person is playing a violin) and background silent semantic (such as there is an unplayed violin hanging in the background). Then the above semantic input text encoder TE is used to generate a set of foreground semantic vectors and a set of background semantic vectors, which are used as supervision signals for downstream training loss. The foreground semantic vector set and the background semantic vector set provided by the MLLM are only used for inference in the training stage, and do not need to be called in the inference stage, ensuring deployment efficiency. The multi-modal large model is used to generate structured semantic vectors, which are used as supervision signals in the training stage to guide the model (audio-video correlation target analysis system integrating multi-modal scene understanding) to distinguish sound and non-sound objects. This strategy enables the model to have semantic-level discrimination ability when facing multiple similar targets, fundamentally solving the semantic bottleneck problem of distinguishing sound and non-sound, and breaking through the limitation of traditional visual and audio alignment methods that can only perceive synchronization signals and cannot understand semantic states. This method follows the training and inference decoupling principle, introduces semantic supervision in the training stage, and completely removes the language model in the inference stage, only retaining the image and audio dual-modal backbone network. The overall architecture is lightweight and easy to deploy efficiently on low-power devices. This invention has good scalability and universality, and can be widely used in intelligent monitoring, voice-driven interaction, media analysis, virtual anchors, and other multi-modal fusion tasks, and has significant industrial landing value and technical leading potential.

[0078] The audio embedding vector is combined with the spatial feature map to perform cosine similarity calculation at each spatial position to generate an audio heat response map :

[0079] ;

[0080] wherein, represents the visual feature vector extracted by the image at the spatial position (i, j), represents the global audio semantic embedding vector extracted by the audio encoder, represents the vector inner product operation, L2 norm of vector, the formula is used to calculate the cosine similarity of visual features and audio semantics at each spatial position, so as to construct the sound source response heat map.

[0081] The audio heat response map is normalized by Sigmoid to generate a foreground mask and a background mask . After weighted sum of spatial feature maps with the mask, the foreground visual representation vector of the response region is obtained after global average pooling and the background visual representation vector The expressions of the foreground visual representation vector and the background visual representation vector are as follows:

[0082] .

[0083] When generating semantic supervision information by using a multimodal large model, a soft contrast loss function is used to compare and learn the constraint of the foreground semantic vector set and the background semantic vector set output by the text encoder with the foreground visual representation vector and the background visual representation vector, so as to improve the accuracy and robustness of semantic alignment. The traditional triplet loss method has problems such as improper gradient punishment and unstable adversarial training when facing pseudo-negative samples with similar semantics but not target. Unlike the traditional triplet loss, the soft contrast loss assigns a lower weight to pseudo-negative samples with similar semantics but not target in the batch, avoiding gradient misdirection caused by semantic errors. The calculation formula of the positive sample similarity term of the soft contrast loss function is:

[0084] ;

[0085] wherein, is the foreground visual representation vector, is the foreground semantic vector of the i-th sample;

[0086] The calculation formula of the semantic weighted negative sample term (Soft-Negative) is:

[0087] ;

[0088] wherein, is the background visual representation vector, is used to measure the similarity between the foreground semantic vector of the i-th sample and the foreground semantic vector of the j-th sample;

[0089] The calculation formula of is:

[0090] ;

[0091] wherein, represents the foreground semantic vector of the i-th sample and the foreground semantic vector of the j-th sample The coefficient is used to reduce the gradient influence of non-target samples with similar semantics, thereby improving the robustness of the contrast loss;

[0092] foreground loss The calculation formula is:

[0093] ;

[0094] background loss The calculation formula is:

[0095] ;

[0096] wherein, is the background semantic vector;

[0097] semantic-guided alignment loss The calculation formula is:

[0098] ;

[0099] wherein, B represents the number of samples in each training batch (i.e. Batch Size), used to normalize and average the foreground loss and the background loss .

[0100] To enhance the stability and discrimination ability in the training process, a soft contrast loss function based on semantic similarity weighting is adopted to dynamically adjust the punishment strength of pseudo-negative samples. This method can effectively alleviate the misidentification interference between semantics close to the target, improve the alignment accuracy between the sound-emitting area and the semantic description, and is especially suitable for robust training in weakly labeled or fuzzy label scenarios.

[0101] Due to the serious overlap of audio corresponding heat maps in multi-target sound-emitting scenarios, a spatial response decoupling mechanism is introduced to further improve the response independence between multiple sound-emitting targets. The foreground semantic vector of the k-th sample generates a corresponding audio heat response map , and the audio heat response map is normalized to obtain a normalized response map . The foreground semantic vector of the j-th sample generates a corresponding audio heat response map , and the audio heat response map is normalized to obtain a normalized response map ., the Wasserstein decoupling loss is calculated as follows:

[0102] ;

[0103] wherein, is an approximate Wasserstein distance calculated based on Sinkhorn algorithm; is used to measure the difference between two normalized response maps and in the distribution of the image space, the Sinkhorn distance takes the probability distribution as input, and calculates the minimum transmission cost between the two by the optimal transmission solution, which can effectively describe the overlap and difference between the sound emitting regions. K is the number of sound emitting semantic vectors contained in the current video frame image;

[0104] the adversarial loss of the region discriminator is calculated as follows:

[0105] ;

[0106] wherein, is the region discriminator, which is used to determine whether the input normalized response map corresponds to the real sound emitting target region; is the region discriminator, which is used to determine whether the input normalized response map corresponds to the real sound emitting target region; the region discriminator outputs a probability value between 0 and 1, which is used to determine whether the input response map corresponds to the real sound emitting target region. The region discriminator includes at least one convolutional feature extraction unit and one fully connected classification unit, the convolutional feature extraction unit is used to encode the spatial features of the normalized response map, and the fully connected classification unit is used to map the encoded features to a single probability output, so as to distinguish the real and non-real sound emitting regions, thereby constructing the adversarial loss of the region discriminator.

[0107] The Wasserstein decoupling loss and the adversarial loss of the region discriminator are combined to obtain the spatial response decoupling loss , which is calculated as follows:

[0108] ;

[0109] wherein, is the weight of the Wasserstein decoupling loss, is the weight of the adversarial loss of the region discriminator.

[0110] The final training loss is obtained according to the spatial response decoupling loss and the semantic guidance alignment loss , and the calculation formula is:

[0111] ;

[0112] wherein, is the weight of the semantic guidance alignment loss, is the weight of the spatial response decoupling loss.

[0113] The spatial response decoupling mechanism comprehensively uses the Wasserstein distance based on the probability distribution and the regional level adversarial discriminator, effectively improves the boundary independence and separation clarity of the response regions among multiple targets, and significantly enhances the structural analysis ability of the audio-video associated target analysis method of the multi-modal scene understanding fusion in a complex sound source environment.

[0114] In the inference stage, the audio heat response graph calculated according to the video frame image and the audio represents the correlation between each position in the video frame image and the audio content. In order to further convert the audio heat response graph into a structured sound- emitting target output, the system introduces a target recognition process, which fuses the audio heat response graph with the potential target region in the video frame image and completes the fine identification of the sound-emitting region. The specific method includes: using the existing object region proposal (such as the bounding box, the candidate target region, the segmentation mask) in the video frame image and the corresponding heat response graph to perform regional weighted evaluation, extracting the heat center, the response mean or maximum value as the region score index, and calculating the region score according to the region score index, and selecting the candidate target region with the highest region score as the sound-emitting target. The response graph and the spatial position coordinates of the output target sound-emitting entity in the video frame image can be output, and the mask or the bounding box can be optionally output for video labeling, clipping or content analysis. At the same time, this process can further filter the non-semantic related regions in combination with the visual semantic consistency of the candidate target, to ensure that the output positioning result covers the real sound-emitting object. This recognition process is completely completed in the forward inference process, and the calculation efficiency is high, and the output result can be used in various application scenarios such as downstream behavior recognition, voice interaction and video semantic clipping.

[0115] The audio-video associated target analysis method of the multi-modal scene understanding fusion provided by the embodiment of the application adopts decoupling in the training stage and the inference stage. In the training stage, the semantic distinguishability and target focusing ability are improved by relying on the MLLM output semantic supervision signal; and in the inference stage, the system only retains the video frame image and the audio dual-modal encoding path, and no longer relies on the language model, so that the overall structure is lightweight, the calculation overhead is low, and good edge deployment adaptability and engineering implementation feasibility are achieved. Compared with most current systems that need complex semantic annotation or online operation of semantic modules, the application significantly reduces the runtime resource demand and system complexity.

[0116] The method for analyzing audio-video associated targets by fusing multi-modal scene understanding provided by the embodiment of the application can effectively understand the semantic state of whether to make a sound by providing visual semantic information through a multi-modal large model, and improve the accuracy of sound target recognition. The method has good universality and expandability, is not dependent on specific scenes, specific object categories or specific data structures, can adapt to various audio-video contents, including instrument playing, human speaking, animal calling, mechanical device operation, etc., and is applicable to various multi-modal fusion application scenarios such as monitoring video analysis, voice interaction system, audio-video synchronous processing, content recommendation system, etc., and has high industry application promotion value.

[0117] As shown in Figure 3 The system for analyzing audio-video associated targets by fusing multi-modal scene understanding provided by another embodiment of the application is used for implementing the method described in the first embodiment, and the system comprises a signal acquisition module, an image processing module, an audio processing module, a response map generation module, a visual representation vector generation module and an output module.

[0118] The signal acquisition module is used for acquiring video frame images containing multiple persons and multiple objects and audio signals synchronously acquired therefrom.

[0119] The image processing module is used for extracting a spatial feature map from the video frame images by using a visual encoder, the spatial feature map retains visual semantic information of each spatial position, the visual semantic information is provided by a multi-modal large model, the visual semantic information is input into a text encoder, and a foreground semantic vector set and a background semantic vector set are obtained; and the audio processing module is used for converting the audio signals into an audio spectrum map, inputting the audio spectrum map into an audio encoder, and outputting a global audio embedding vector.

[0120] The response map generation module is used for performing cosine similarity calculation on the spatial feature map and the audio embedding vector at each spatial position, and generating an audio heat response map.

[0121] The visual representation vector generation module is used for generating a foreground mask and a background mask from the audio heat response map after normalization, performing weighted summation on the spatial feature map by using the foreground mask and the background mask respectively, and then performing global average pooling to obtain a foreground visual representation vector and a background visual representation vector of a response region.

[0122] The output module is used for converting the audio heat response map into a structured sound target output, and outputting a response map and spatial position coordinates of a target occurrence entity in the video frame image.

[0123] The system output module comprises a conversion unit, which uses the existing object region proposal in the image and the corresponding heat response map to perform regional weighted evaluation, extracts the heat center, response mean or maximum value as a region score index, and calculates a region score according to the region score index, and selects a candidate target region with the highest region score as the sound- emitting target.

[0124] The system further comprises a model training module, which uses a multi-modal large model to generate semantic supervision information, adopts a structured semantic vector provided by the multi-modal large model to construct a discriminative target function for the sound-emitting target, and specifically comprises the following steps:

[0125] The language prompt word, the video frame image and the corresponding audio category label are input into the multi-modal large model to generate foreground sound-emitting semantics and background silence semantics of the sound-emitting target, and the foreground sound-emitting semantics and the background silence semantics are input into a text encoder to output a foreground semantic vector set and a background semantic vector set as a supervision signal of a training loss.

[0126] Another embodiment of the present application provides a computer terminal, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the method as described in the first embodiment of the present application when executing the program.

[0127] Another embodiment of the present application provides a computer readable medium, which stores a computer program, and the computer program is executable by a processor to implement the method as described in the first embodiment of the present application.

[0128] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can adopt a computer program product implemented on one or more computer usable storage media containing computer usable program code (including but not limited to disk storage, CD-ROM, optical storage, etc.).

[0129] The present application is described with reference to the flowcharts and / or block diagrams according to the methods, devices (systems) and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions described in the flowcharts and / or block diagrams. Figure 1 one flow or multiple flows and / or blocks Figure 1means for performing the function specified by the block or blocks.

[0130] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 one or more flows and / or blocks Figure 1 means for performing the function specified by the block or blocks.

[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 one or more flows and / or blocks Figure 1 means for performing the function specified by the block or blocks.

[0132] The above description is only specific implementation of the present application, and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An audio-video associated target analysis method fusing multi-modal scene understanding, characterized in that, The method comprises the following steps: acquiring video frame images containing multiple people and multiple objects and audio signals synchronously collected therewith; extracting spatial feature maps from the video frame images using a visual encoder, the spatial feature maps retaining visual semantic information of each spatial position, the visual semantic information being provided by a multimodal large model, and inputting the visual semantic information into a text encoder to obtain a foreground semantic vector set and a background semantic vector set; converting the audio signals into audio spectrum maps and inputting the audio spectrum maps into an audio encoder to output global audio embedding vectors; performing cosine similarity calculation on the spatial feature maps and the audio embedding vectors at each spatial position to generate an audio heat response map; generating foreground masks and background masks from the audio heat response map after normalization, and performing weighted summation on the spatial feature maps using the foreground masks and the background masks respectively, and then performing global average pooling to obtain foreground visual representation vectors and background visual representation vectors of response regions; converting the audio heat response map into a structured sound-making target output to output a response map and spatial position coordinates of a target occurrence entity in the video frame images; the specific method of converting the audio heat response map into the structured sound-making target output comprises: performing regional weighted evaluation on existing object region proposals in the video frame images and corresponding heat response maps; extracting heat center, response mean or maximum value as a regional score index, and calculating a regional score according to the regional score index; selecting a candidate target region with the highest regional score as a sound-making target; the method further comprises: when generating semantic supervision information using the multimodal large model, using a structured semantic vector provided by the multimodal large model to construct a discriminative target function for the sound-making target, specifically comprising: inputting language prompt words, video frame images and corresponding audio class labels into the multimodal large model to generate foreground sound-making semantics and background silence semantics of the sound-making target, and inputting the foreground sound-making semantics and the background silence semantics into a text encoder to output a foreground semantic vector set and a background semantic vector set as a supervision signal of a training loss; When the multimodal large model generates semantic supervision information, a soft contrast loss function is used to constrain contrast learning of a foreground semantic vector set and a background semantic vector set output by a text encoder and a foreground visual representation vector and a background visual representation vector, and a positive sample similarity term of the soft contrast loss function is: is: ； wherein, is a foreground visual representation vector, is a foreground semantic vector for the i-th sample; Semantically weighted negative sample items For: ； wherein, is a background visual representation vector, is used to measure the similarity between the foreground semantic vector of the i-th sample and the foreground semantic vector of the j-th sample . The calculation formula is: ； wherein, represents the foreground semantic vector of the i-th sample represents the foreground semantic vector of the j-th sample the cosine similarity between the foreground semantic vectors of the i-th and j-th samples; foreground loss The calculation formula is: ； Background loss The formula for calculating the background loss is: ； wherein, is a background semantic vector; Semantic-guided alignment loss The calculation formula is: ； where B denotes the number of samples in each training batch, and is used to normalize the foreground loss and the background loss is normalized by the average.

2. The method of claim 1, wherein, When a multimodal large model is used to generate semantic supervision information, a foreground semantic vector of a kth sample is obtained Generate a corresponding audio heat response map , the audio heat response map The normalized response map is obtained through normalization processing , the foreground semantic vector of the jth sample is obtained Generate a corresponding audio heat response map , the audio heat response map The normalized response map is obtained through normalization processing , the Wasserstein distance is optimized through the adversarial loss of the region discriminator, wherein the Wasserstein decoupling loss The calculation formula is: ； wherein, is an approximate Wasserstein distance computed based on Sinkhorn algorithm; K is the number of vocal semantic vectors contained in the current video frame image; Adversarial loss of the region discriminator The calculation formula is: ； wherein, is a region discriminator for discriminating whether the input normalized response map corresponds to a real sound- emitting target region or not. is a region discriminator for discriminating whether the input normalized response map corresponds to a real sound- emitting target region or not. Wasserstein disentanglement loss and adversarial loss of region discriminator combining to obtain spatial response disentanglement loss , and the calculation formula is: ； wherein, is a weight for the Wasserstein decoupling loss, is a weight for the adversarial loss of the regional discriminator; The final training loss is obtained according to the spatial response decoupling loss and the semantic guidance alignment loss The calculation formula is: ； wherein, is a weight for the semantic guidance alignment loss, is a weight for the spatial response decoupling loss.

3. An audio-video associated target analysis system fusing multi-modal scene understanding, characterized in that, the system is used to implement the method according to any one of claims 1-2, and comprises a signal acquisition module, an image processing module, an audio processing module, a response map generation module, a visual representation vector generation module and an output module; the signal acquisition module is used to acquire video frame images containing multiple people and multiple objects and audio signals synchronously collected therewith; the image processing module is used to extract spatial feature maps from the video frame images using a visual encoder, the spatial feature maps retaining visual semantic information of each spatial position, the visual semantic information being provided by a multimodal large model, and inputting the visual semantic information into a text encoder to obtain a foreground semantic vector set and a background semantic vector set; the audio processing module is used to convert the audio signals into audio spectrum maps and input the audio spectrum maps into an audio encoder to output global audio embedding vectors; the response map generation module is used to perform cosine similarity calculation on the spatial feature maps and the audio embedding vectors at each spatial position to generate an audio heat response map; The visual representation vector generation module is configured to generate foreground and background masks after normalizing the audio heat response map, and then perform weighted summation on the spatial feature map using the foreground and background masks respectively, and perform global average pooling to obtain foreground and background visual representation vectors of the response region; The output module is configured to convert the audio heat response map into a structured sound target output, and output a response map and a spatial position coordinate of the target generating entity in the video frame image; The output module includes a conversion unit configured to perform regional weighted evaluation on the existing object region proposal in the video frame image and the corresponding heat response map, extract a heat center, a response mean or maximum value as a regional score index, calculate a regional score according to the regional score index, and select a candidate target region with the highest regional score as the sound target; The model training module is configured to use a multi-modal large model to generate semantic supervision information, and use a structured semantic vector provided by the multi-modal large model to construct a discriminative target function for the sound target, and specifically includes: inputting a language prompt word, a video frame image and a corresponding audio category label into the multi-modal large model to generate foreground sound semantics and background silence semantics of the sound target, and inputting the foreground sound semantics and the background silence semantics into a text encoder to output a foreground semantic vector set and a background semantic vector set as a supervision signal of a training loss.

4. A computer terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the program to implement the method of any one of claims 1-2.

5. A computer readable medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-2.

Images