Object recognition method and device, equipment, medium and product

Abstract

The embodiment of the invention discloses an object recognition method and device, equipment, a medium and a product, and the method comprises the steps: extracting audio features and video features from a to-be-processed target video, and obtaining scene context information corresponding to the target video; performing cross-modal attention interaction processing on the audio features and the video features to obtain target audio features and target video features; according to the audio features, the video features and the scene context information, a gating vector is generated, and the gating vector is used for controlling the contribution degree of the audio features and the video features in the fusion process; and performing feature fusion on the target audio feature and the target video feature according to the gating vector to obtain a fusion feature, and performing object recognition processing according to the fusion feature. According to the technical scheme of the embodiment of the invention, the accuracy, stability and robustness of object recognition can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of object recognition, and in particular, to an object recognition method, device, equipment, medium and product. BACKGROUND

[0002] With the rapid growth of video content, people increasingly hope to automatically identify the identity of objects in the video, so as to realize intelligent retrieval and analysis of content, and therefore, object recognition technology becomes particularly important in the field of multimedia processing and intelligent interaction.

[0003] At present, the basic process of the object recognition method is as follows: first, the respective features are extracted from the audio signal and the video signal of the video, then the extracted audio features and video features are simply spliced and fused, and then the object identity is determined based on the fused features.

[0004] However, in a complex real scene, the video content is too complex, such as background music, multiple people speaking at the same time, environmental noise changes, and dynamic changes of object tone or expression (for example, the difference in performance of an actor in different roles), which will cause the audio and video features to be unstable, and the simple splicing and fusion will result in inaccurate recognition results. SUMMARY

[0005] Embodiments of the present application provide an object recognition method, object recognition device, electronic equipment, computer readable storage medium and computer program product, which can improve the accuracy, stability and robustness of object recognition.

[0006] Other characteristics and advantages of the present application will become apparent from the following detailed description, or will be learned by practice of the present application.

[0007] According to an aspect of an embodiment of the present application, an object recognition method is provided, comprising: extracting audio features and video features from a target video to be processed, and obtaining scene context information corresponding to the target video; performing cross-modal attention interaction processing on the audio features and the video features to obtain target audio features and target video features; generating a gating vector according to the audio features, the video features and the scene context information, the gating vector being used to control the contribution degree of the audio features and the video features in the fusion process; performing feature fusion on the target audio features and the target video features according to the gating vector to obtain fused features, and performing object recognition processing according to the fused features.

[0008] According to an aspect of an embodiment of the present application, an object recognition device is provided, comprising: an acquisition module configured to extract audio features and video features from a target video to be processed, and acquire scene context information corresponding to the target video; an interaction module configured to perform cross-modal attention interaction processing on the audio features and the video features to obtain target audio features and target video features; a generation module configured to generate a gating vector according to the audio features, the video features, and the scene context information, the gating vector being used to control the contribution of audio features and video features in a fusion process; and an identification module configured to perform feature fusion on the target audio features and the target video features according to the gating vector to obtain fused features, and perform object recognition processing according to the fused features.

[0009] In an embodiment of the present application, the device further comprises a calibration module configured to calibrate the audio features and the video features according to the scene context information to obtain initial audio features and initial video features; and the interaction module is further configured to perform feature interaction processing on the initial audio features and the initial video features through a cross-modal attention mechanism to obtain the target audio features and the target video features.

[0010] In an embodiment of the present application, the calibration module is further configured to generate feature regulation parameters for the audio features and the video features respectively according to the scene context information, the feature regulation parameters being used to adjust the amplitude and distribution of features; to perform transformation processing on the audio features according to the feature regulation parameters corresponding to the audio features to obtain the initial audio features; and to perform transformation processing on the video features according to the feature regulation parameters corresponding to the video features to obtain the initial video features.

[0011] In an embodiment of the present application, the feature regulation parameters comprise a gain parameter used to adjust the amplitude of features and a bias parameter used to correct the distribution of features; and the calibration module is further configured to acquire a first calibration sub-network used for audio feature calibration and a second calibration sub-network used for video feature calibration; and input the scene context information into the first calibration sub-network and the second calibration sub-network respectively, wherein the first calibration sub-network generates the gain parameter and the bias parameter corresponding to the audio features through nonlinear mapping according to context information related to audio features in the scene context information; and the second calibration sub-network generates the gain parameter and the bias parameter corresponding to the video features through nonlinear mapping according to context information related to video features in the scene context information.

[0012] In one embodiment of this application, the interaction module is further configured to perform feature processing on the initial audio features and the initial video features respectively through a multi-head self-attention mechanism to obtain intermediate audio features and intermediate video features; obtain uncertainty information corresponding to the intermediate audio features and the intermediate video features respectively, wherein the uncertainty information is used to characterize the reliability of the features in the current scene corresponding to the scene context information; calculate the cross-modal attention weight between the intermediate audio features and the intermediate video features based on the uncertainty information; and perform feature interaction processing on the intermediate audio features and the intermediate video features based on the cross-modal attention weight to obtain the target audio features and the target video features.

[0013] In one embodiment of this application, the interaction module is further configured to input the intermediate audio features and the intermediate video features into an uncertainty prediction network, respectively, to obtain the uncertainty information generated by the uncertainty prediction network based on the feature quality of the input features; or, to perform multiple forward propagations with random deactivation on the intermediate audio features and the intermediate video features, respectively, and determine the corresponding uncertainty information based on the statistical dispersion output of the multiple forward propagations.

[0014] In one embodiment of this application, the interaction module is further configured to calculate an initial attention weight value between the intermediate audio feature and the intermediate video feature based on the query vector of the intermediate audio feature and the key vector of the intermediate video feature; aggregate the uncertainty information corresponding to the intermediate video feature on a preset feature dimension to obtain an uncertainty sequence; correct the initial attention weight value based on the uncertainty sequence to obtain an intermediate attention weight value; and normalize the intermediate attention weight value to obtain the cross-modal attention weight of the intermediate audio feature on the intermediate video feature.

[0015] In one embodiment of this application, the acquisition module is further configured to perform global pooling on the audio features and the video features respectively to obtain audio pooling features corresponding to the audio features and video pooling features corresponding to the video features; and input the audio pooling features and the video pooling features into a preset context-aware network to obtain the scene context information extracted by the context-aware network from the audio pooling features and the video pooling features.

[0016] In one embodiment of this application, the generation module is further configured to calibrate the audio features and the video features according to the scene context information to obtain initial audio features and initial video features; perform global pooling on the initial audio features and the initial video features respectively to obtain corresponding target audio pooling features and target video pooling features; obtain the scene context features corresponding to the scene context information, and concatenate the target audio pooling features, target video pooling features and context features to obtain gated input features; input the gated input features to a preset dynamic modality contribution network for nonlinear mapping to obtain the gated vector.

[0017] In one embodiment of this application, the identification module is further configured to perform object identification processing based on the fusion features to obtain an initial object and the confidence level of the initial object, wherein the confidence level is used to characterize the credibility of object identification; if the confidence level of the initial object is higher than a preset confidence level, or if the initial object is identified as a registered object, then the identity embedding feature corresponding to the initial object is obtained; the fusion features are corrected based on the identity embedding feature to obtain a target fusion feature, and object identification processing is performed based on the target fusion feature.

[0018] In one embodiment of this application, the identification module is further configured to input the identity embedding features into a preset correction network to obtain affine transformation parameters for the object; and to perform affine transformation processing on the fusion features according to the affine transformation parameters for the object to obtain the target fusion features.

[0019] In one embodiment of this application, the apparatus further includes a training module for acquiring a model to be trained, as well as sample audio features and sample video features from a sample video; adding perturbations to the sample audio features and sample video features respectively to obtain sample adversarial audio features and sample adversarial video features; performing feature discarding processing on the sample audio features and sample video features respectively to obtain sample discarded audio features and sample discarded video features; training the model to be trained based on the sample adversarial audio features, the sample adversarial video features, the sample discarded audio features, and the sample discarded video features to obtain an object recognition model, which is used to perform object recognition processing to identify objects in the target video.

[0020] In one embodiment of this application, the training module is further configured to calibrate the sample audio features and sample video features according to the sample scene context information corresponding to the sample video to obtain initial sample audio features and initial sample video features; generate a corresponding discard mask according to the sample scene context information, the initial sample audio features and the initial sample video features; and perform feature discarding processing on the initial sample audio features and initial sample video features according to the discard mask to obtain discarded sample audio features and discarded sample video features.

[0021] In one embodiment of this application, the training module is further configured to input the sample scene context information, the sample initial audio features, and the sample initial video features into a preset adversarial generative network to obtain drop information describing the drop probability of feature locations; and to obtain the drop mask by sampling the drop information through a Günbel distribution.

[0022] In one embodiment of this application, the training module is further configured to obtain a first recognition result obtained by the model to be trained performing object recognition processing on the sample adversarial audio features and the sample adversarial video features, and a second recognition result obtained by the model to be trained performing object recognition processing on the sample audio features and the sample video features; generate an adversarial loss based on the differences between the first recognition result, the second recognition result and the object labels corresponding to the sample video; obtain a third recognition result obtained by the model to be trained performing object recognition on the sample discarded audio features and the sample discarded video features; generate a discard loss based on the differences between the third recognition result and the object labels corresponding to the sample video; generate a model loss based on the adversarial loss and the discard loss, and update the model parameters of the model to be trained based on the model loss to obtain the object recognition model.

[0023] In one embodiment of this application, the training module is further configured to predict the sample speech information contained in the sample audio features through a preset classification model, and generate a speech activity detection loss based on the difference between the sample speech information and the real speech information; calculate the temporal alignment error between the sample speech features and the sample video features, and generate a synchronization loss based on the temporal alignment error; calculate the similarity between the sample audio features and the sample video features to generate an audio-visual interaction loss; and generate the model loss based on the speech activity detection loss, the synchronization loss, the audio-visual interaction loss, the adversarial loss, and the dropout loss.

[0024] According to one aspect of the embodiments of this application, an electronic device is provided, including one or more processors; and a storage device for storing one or more computer programs, which, when executed by the one or more processors, cause the electronic device to implement the object recognition method as described above.

[0025] According to one aspect of the embodiments of this application, the embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of an electronic device, causes the electronic device to perform the object recognition method as described above.

[0026] According to one aspect of the embodiments of this application, this application provides a computer program product, including a computer program stored in a computer-readable storage medium, wherein a processor of an electronic device reads from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the object recognition method as described above.

[0027] In the technical solution provided by the embodiments of this application, audio features and video features are extracted from the target video to be processed, and the scene context information corresponding to the target video is obtained; cross-modal attention interaction processing is performed on the audio features and video features to obtain target audio features and target video features; based on the audio features, video features, and scene context information, a gating vector is generated, which is used to control the contribution of audio features and video features in the fusion process; feature fusion is performed on the target audio features and target video features based on the gating vector to obtain fused features, and object recognition processing is performed based on the fused features; that is, the technical solution provided by this application, by performing cross-modal attention interaction on audio features and video features, can establish a closer feature association between different modalities, so that audio and video information can be fully fused at the semantic level, avoiding This approach avoids the problem of low information utilization in traditional simple splicing methods, improves the accuracy and discriminative power of feature fusion, and further incorporates scene context information extracted from the target video. During the generation of the gating vector, the overall features of the video scene are comprehensively considered, enabling the feature fusion process to dynamically adjust the fusion strategy according to different environmental conditions, thus improving adaptability in complex scenes. Furthermore, by setting the gating vector, the contribution ratio of audio and video features in the fusion process is controlled, allowing the target audio and video features to be dynamically adjusted according to contextual information during fusion to balance the influence of each modality and avoid excessive dominance of a single modality in the recognition result. This enhances the stability and consistency of the fused features, and object recognition processing based on the fused features can improve the accuracy, stability, and robustness of object recognition.

[0028] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0030] Figure 1 This is a schematic diagram of one implementation environment involved in this application.

[0031] Figure 2 This is a flowchart illustrating an object recognition method in an exemplary embodiment of this application.

[0032] Figure 3 This is a flowchart illustrating another object recognition method as an exemplary embodiment of this application.

[0033] Figure 4 This is a flowchart illustrating another object recognition method as an exemplary embodiment of this application.

[0034] Figure 5 This is a flowchart illustrating another object recognition method as an exemplary embodiment of this application.

[0035] Figure 6 This is a flowchart illustrating another object recognition method as an exemplary embodiment of this application.

[0036] Figure 7 This is a flowchart illustrating an object recognition model training process, as shown in an exemplary embodiment of this application.

[0037] Figure 8 This is a schematic diagram illustrating an object recognition model, which is another exemplary embodiment of this application.

[0038] Figure 9 This is a structural block diagram of an object recognition device illustrated in an exemplary embodiment of this application.

[0039] Figure 10 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0040] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0041] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0042] The flowcharts shown in the accompanying diagrams are merely illustrative and do not necessarily include all content and operations, nor do they necessarily have to be executed in the described order. For example, some operations may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0043] It should also be noted that "multiple" as mentioned in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0044] The technical solutions of the embodiments of this application will be described in detail below.

[0045] Please see Figure 1 , Figure 1 This is a schematic diagram of an implementation environment related to this application. The implementation environment includes a terminal device 100 and a server 200.

[0046] Terminal device 100 is used to acquire the target video to be processed and send the target video to the server.

[0047] Server 200 is used to extract audio and video features from the target video to be processed, obtain the scene context information corresponding to the target video, perform cross-modal attention interaction processing on the audio and video features to obtain target audio features and target video features, and then generate a gating vector based on the audio features, video features and scene context information. The gating vector is used to control the contribution of audio features and video features in the fusion process. The target audio features and target video features are fused according to the gating vector to obtain fused features, and object recognition processing is performed based on the fused features to obtain the recognition result.

[0048] The server also sends the recognition results to the terminal device, which can then perform various subsequent intelligent processing operations based on the received recognition results. This can be applied to various application scenarios. For example, in intelligent retrieval and playback scenarios, the terminal device can perform object-based video content retrieval based on the recognition results, directly searching for the content spoken by a specific object in the video, or quickly locating the segment where the object appears, achieving a more efficient video browsing and playback experience. In intelligent subtitle and personalized narration scenarios, the terminal device can automatically generate or optimize subtitles based on the identified object's identity, add tags to different objects, or automatically switch subtitle and narration languages ​​based on object characteristics in a multilingual environment, thereby improving the readability of video content and the barrier-free viewing experience. In content labeling and automated management scenarios, by automatically identifying objects appearing in the video, video content can be automatically tagged, indexed, and classified, facilitating content management, archiving, and retrieval, and achieving structured management of media assets. In precise advertising and personalized recommendation scenarios, based on the identified object information in the video, the characteristics of people can be correlated with user interests, thereby achieving more precise advertising and content recommendation, and improving business conversion efficiency. In intelligent editing and content creation assistance scenarios, it can automatically extract segments of different objects in the video based on the recognition results, providing material segmentation support for post-production, realizing automatic editing, extraction of wonderful segments or thematic compilation, and significantly improving the intelligence level of video creation and editing.

[0049] Through the above application scenarios, this invention can not only achieve high-precision object recognition in the recognition stage, but also has wide application value in content management, video creation and personalized services.

[0050] It is understood that in other embodiments of this application, the terminal device or server may also implement the object recognition method independently. That is, the terminal device or server acquires the target video to be processed, extracts audio features and video features from it, acquires the scene context information corresponding to the target video, performs cross-modal attention interaction processing on the audio features and video features, generates a gating vector, and then performs feature fusion based on the gating vector to perform object recognition processing.

[0051] Among them, terminal devices can be electronic devices with display functions such as smartphones, tablets, laptops, desktop computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft. Servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. This document does not impose any restrictions on these.

[0052] Terminal devices and servers establish a communication connection in advance through a network, enabling them to communicate with each other via the network; the network can be a wired network or a wireless network, and this document does not impose any restrictions.

[0053] It should be noted that in the specific implementation of this application, if the video involves objects, when the embodiments of this application are applied to specific products or technologies, permission or consent from the objects is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0054] The following describes in detail the various implementation details of the technical solutions in the embodiments of this application.

[0055] like Figure 2 As shown, Figure 2 This is a flowchart illustrating an embodiment of an object recognition method, which can be applied to... Figure 1 In the implementation environment shown, the method can be executed by a terminal or a server, or by both a terminal device and a server. In this embodiment, the method is described as being executed by a server. The object identification method may include steps S210 to S240, which are described in detail below.

[0056] S210. Extract audio and video features from the target video to be processed, and obtain the scene context information corresponding to the target video.

[0057] In this embodiment, the target video to be processed refers to video data that has not yet undergone object recognition processing. Object recognition processing refers to the process of identifying the object (i.e., the speaker) emitting speech in the target video. Therefore, the target video includes two parts: an audio stream and a video stream. For the audio stream, preprocessing operations can be performed first, such as sampling rate unification, frame segmentation, windowing, normalization, and noise suppression, to reduce environmental interference and improve signal quality. The preprocessed audio stream is then used to extract features from the audio encoder to obtain temporal audio features that change over time. These audio features are used to characterize the acoustic properties, speech rhythm, energy distribution, and spectral morphology of the object in the target video. For the video stream, frame sampling can be performed to reduce computational complexity and retain key visual information. The sampled video frames are input to the video encoder to extract temporal video features. These video features reflect the visual representation of the object in the video, including facial shape, lip movements, lip motion trajectory, and facial dynamic information.

[0058] In order to maintain high recognition accuracy in different environments, this application embodiment also extracts information related to the overall scene state from the target video to form scene context information. This scene context information is used to reflect the audio and image quality, content complexity and environmental characteristics of the video.

[0059] Optionally, scene context information may include, but is not limited to, the following elements: Noise level: used to describe the strength of background noise in the audio signal, reflecting the purity of the recording environment; Reverberation level: used to represent the reflection and attenuation characteristics of sound in the recording space, estimated by the reverberation time or energy decay rate of the signal, used to evaluate speech intelligibility; Number of objects: refers to the number of objects that speak or appear simultaneously in the current video segment, used to help distinguish between single and multi-person dialogue scenes; Voice activity detection (VAD) confidence: used to reflect the probability or confidence value of the presence of speech in the current frame, representing the salience of speech information in the audio segment; Video clarity: used to describe the clarity of the video image and facial recognizability, which can be calculated through image sharpness or contrast indicators.

[0060] Optionally, scene context information can be obtained by aggregating statistical or encoded features of audio or video. For example, to obtain scene context information corresponding to a target video, the following steps are taken: global pooling is performed on audio features and video features respectively to obtain audio pooling features corresponding to audio features and video pooling features corresponding to video features; the audio pooling features and video pooling features are input into a preset context-aware network to obtain scene context information extracted by the context-aware network from the audio pooling features and video pooling features.

[0061] Specifically, audio features and video features are pooled separately to obtain a compressed global feature representation. The purpose of pooling is to compress information in the time dimension into a more representative global feature, so that subsequent processing can focus on global scene information rather than details at each moment. The pooling process can be global max pooling or global average pooling.

[0062] Audio pooling features and video pooling features are input into a pre-defined context-aware network. The context-aware network typically includes several fully connected layers, convolutional layers, or Transformer layers. It learns the high-level semantics of the input pooling features through non-linear mapping, and then learns to identify scene-related information such as environmental noise, image clarity, number of objects, and speech activity from global audio and video features to generate scene context information.

[0063] Through the above process, audio and video features are compressed using pooling operations. Then, the audio pooling features and video pooling features are input into the context-aware network, which can extract scene-related environmental information from these features and generate scene context information accurately and reliably.

[0064] S220. Perform cross-modal attention interaction processing on the audio and video features to obtain the target audio features and target video features.

[0065] In the embodiments of this application, cross-modal attention interaction processing refers to the process of interactively modeling the features of different modalities (audio modality and video modality) using attention mechanisms, so as to explicitly establish semantic connections and temporal correspondences between the two modalities, thereby enabling audio and video features to complement and constrain each other, resulting in more discriminative target audio features and target video features.

[0066] In this process, the feature representation of one modality is used as the query vector, and the feature representation of the other modality is used as the key vector and value vector. By calculating the relevance between the query and the key, an attention weight matrix between the two modal features is obtained. This matrix is ​​used to measure the degree of attention of the feature of one modality to each feature fragment in the other modality. The value vector is weighted and converged according to the attention weight matrix to obtain the target feature representation that is semantically aligned or complementary to the current query modality.

[0067] S230. Based on audio features, video features, and scene context information, generate a gating vector. The gating vector is used to control the contribution of audio features and video features in the fusion process.

[0068] In this embodiment, the gating vector can be generated based on the influence of audio feature quality, video feature features, and scene context information on the two modalities, thereby achieving adaptive adjustment between modalities. The gating vector can be a modal-level weight, used to directly characterize the contribution ratio of the audio modality and the video modality in the fusion process. For example, the gating vector... ,in Represents the fusion weights of audio modalities. This represents the fusion weights of the video modalities, and The gate vector can also be a weighted vector consistent with the feature dimension, enabling finer-grained dimension-wise fusion control.

[0069] S240. Based on the gating vector, the target audio features and target video features are fused to obtain fused features, and object recognition processing is performed based on the fused features.

[0070] In this embodiment, the gating vector controls the contribution of the target audio features and the target video features. When the gating vector is a modal-level weight, the target audio features and the target video features can be weighted and combined according to the gating vector to obtain the fused features. When the gating vector is a feature dimension-level weight, the fused features can be obtained by element-level weighting.

[0071] In one example, fused features are input to an identification head for object recognition processing. The identification head can be a linear / nonlinear mapping unit for classification or an embedding extraction unit for retrieval / verification. When used for object recognition (classification), the identification head outputs a probability distribution of the identities of each candidate object. When used for object verification / retrieval, the identification head outputs a fixed-length embedding vector and calculates its similarity with a pre-stored object vector to determine the object's identity.

[0072] In this embodiment, by performing cross-modal attention interaction on audio and video features, a closer feature association can be established between different modalities, enabling full semantic fusion of audio and video information. This avoids the problem of low information utilization in traditional simple splicing methods, improves the accuracy and discriminative power of feature fusion, and further introduces scene context information extracted from the target video. During the generation of the gating vector, the overall features of the video scene are comprehensively considered, allowing the feature fusion process to dynamically adjust the fusion strategy according to different environmental conditions, improving adaptability in complex scenes. Furthermore, by setting the gating vector, the contribution ratio of audio and video features in the fusion process can be controlled, allowing the target audio and video features to be dynamically adjusted according to context information during the fusion process to balance the influence of each modality feature and avoid the excessive dominance of a single modality on the recognition result. This enhances the stability and consistency of the fused features, and object recognition processing based on the fused features can improve the accuracy, stability, and robustness of object recognition.

[0073] In one embodiment of this application, another object recognition method is provided, which can be applied to... Figure 1 The implementation environment shown is as follows: Figure 3 As shown, this object recognition method is in Figure 2 Based on S210~S240 shown, the process of cross-modal attention interaction processing in S220 is extended to S310~S320; S310~S320 are described in detail below.

[0074] S310. The audio and video features are calibrated based on the scene context information to obtain the initial audio and video features.

[0075] In this embodiment, calibrating audio and video features based on scene context information refers to dynamically adjusting the intensity of different modal features using scene context information. This allows for adaptive enhancement of high-confidence features and suppression of interfered features in different environments, thereby maintaining optimal feature quality in complex scenes. For example, when strong noise or reverberation is detected in the audio signal, the parts of the audio features affected by noise or reverberation can be suppressed, thereby reducing the interference of invalid components on subsequent recognition. When the video image has high clarity and stable lighting, the expression intensity of the video features can be enhanced accordingly, placing greater emphasis on reliable visual information.

[0076] Optionally, the audio and video features are calibrated, including: generating feature adjustment parameters for the audio and video features respectively based on scene context information, wherein the feature adjustment parameters are used to adjust the amplitude and distribution of the features; transforming the audio features according to the feature adjustment parameters corresponding to the audio features to obtain initial audio features; and transforming the video features according to the feature adjustment parameters corresponding to the video features to obtain initial video features.

[0077] Among them, the feature adjustment parameters are used to adaptively adjust the amplitude and distribution characteristics of the input features. Amplitude adjustment refers to scaling the feature signal intensity to enhance information saliency or suppress noise interference. Feature distribution adjustment refers to shifting the feature center or mean to make the feature distribution more stable and in line with the requirements of standardized input. Then, according to the feature adjustment parameters corresponding to the audio features, the amplitude and distribution of the audio features are adjusted to obtain the initial audio features. Similarly, according to the feature adjustment parameters corresponding to the video features, the corresponding amplitude and distribution of the video features are adjusted to obtain the initial video features.

[0078] Optionally, the feature adjustment parameters include a gain parameter for adjusting the feature amplitude and a bias parameter for correcting the feature distribution. Generating the feature adjustment parameters includes: obtaining a first calibration sub-network for audio feature calibration and a second calibration sub-network for video feature calibration; inputting scene context information into the first and second calibration sub-networks respectively, wherein the first calibration sub-network generates the gain parameter and bias parameter corresponding to the audio feature through nonlinear mapping based on the context information related to the audio feature in the scene context information; and the second calibration sub-network generates the gain parameter and bias parameter corresponding to the video feature through nonlinear mapping based on the context information related to the video feature in the scene context information.

[0079] The first and second calibration subnetworks can be lightweight multilayer perceptrons (MLPs) or other learnable nonlinear mapping structures, with independent parameters to accommodate the statistical differences between the two modalities. Scene context information is input into the first and second calibration subnetworks respectively. The first calibration subnetwork focuses on audio-related context components (such as noise level, reverberation, VAD confidence, etc.), while the second calibration subnetwork focuses on video-related context components (such as video sharpness, lip detection stability, etc.). The scene context information is a unified vector. The calibration subnetwork can then learnable selection / projection layers from... Extract the context sub-vectors associated with each mode , .

[0080] Taking the audio path as an example (the same applies to the video path), the process of the first calibration sub-network generating gain and bias parameters includes: After normalization (such as LayerNorm), the input linear projection layer yields an intermediate representation:

[0081] Mapping intermediate representations to parameter space:

[0082] in and Align the target dimension with the audio feature dimension (e.g.) or ).

[0083] Optionally, to avoid excessive scaling / offset, the following can be applied: and Apply differentiable constraint functions:

[0084] in for Affine transformation, This is the set bias range constant.

[0085] Optionally, if it is necessary to maintain the numerical center, the gain can be further adjusted. , Control the amplitude.

[0086] At this point, the audio feature transformation process includes: performing element-wise affine transformations on the audio features using the corresponding gain and bias parameters to obtain the initial audio features.

[0087] Through the above feature modulation and transformation processing, the signal strength and distribution stability of different modalities can be dynamically balanced at the feature level, thereby improving the alignment and fusion of audio and video features in subsequent cross-modal interaction processes.

[0088] S320. The initial audio features and initial video features are processed through a cross-modal attention mechanism to obtain the target audio features and target video features.

[0089] In the embodiments of this application, taking query-key-value (QKV) as an example, the attention of audio to video and the attention of video to audio can be calculated separately. For example, when audio features are used as queries and video features are used as keys and values, the attention results of audio features in the visual modality can be obtained, forming target audio features supplemented with video information. When video features are used as queries and audio features are used as keys and values, the attention results of video features in the acoustic modality can be obtained, forming target video features enhanced with audio information.

[0090] In this application embodiment, the cross-modal attention mechanism can adopt a multi-head attention structure, that is, calculate multiple sets of attention weights in parallel in different subspaces, which can capture cross-modal relevance from multiple perspectives; or it can stack multiple layers of cross-modal attention units to achieve deeper semantic interaction.

[0091] By adaptively calibrating audio and video features using scene context information before cross-modal interaction, dynamic feature enhancement and suppression under different environmental conditions can be achieved. Subsequently, bidirectional interaction and semantic alignment between features are achieved through a cross-modal attention mechanism, thereby obtaining more stable, robust, and discriminative target audio and video features, effectively improving the accuracy and adaptability of object recognition.

[0092] Optionally, in order to avoid interference from low-confidence or unreliable modal information during cross-modal feature interaction and to improve the robustness and accuracy of the overall recognition, in this embodiment, the reliability of each modal feature is quantitatively evaluated before cross-modal feature interaction, and an uncertainty correction term is introduced in the attention weight calculation to focus more on high-confidence features and weaken the influence of noisy modalities.

[0093] In one example, such as Figure 4 As shown, step S320 includes...

[0094] S410. The initial audio features and initial video features are processed by a multi-head self-attention mechanism to obtain intermediate audio features and intermediate video features.

[0095] In this embodiment, initial audio features and initial video features are respectively input into a multi-head self-attention mechanism module to enhance the global dependency modeling capability within each modality. The multi-head self-attention module for audio features can capture the temporal dependency of speech signals by calculating the correlation between each time step, thereby obtaining a deep expression of speaking rhythm, energy changes, and prosodic information. The multi-head self-attention module for video features can extract key dynamic information related to speaking actions (such as lip movements and facial expressions) by modeling the correlation between different frames. After processing, enhanced intermediate audio features and intermediate video features are obtained, so that each modality has a more complete internal semantic expression before entering cross-modal interaction.

[0096] S420. Obtain the uncertainty information corresponding to the intermediate audio features and intermediate video features respectively. The uncertainty information is used to characterize the reliability of the features in the current scene corresponding to the scene context information.

[0097] To measure the reliability of each modal feature in the current scenario, embodiments of this application further obtain uncertainty information corresponding to intermediate audio features and intermediate video features. The uncertainty information is used to indicate whether the feature is reliable and trustworthy in the current environment corresponding to the scenario context. The higher the value of the uncertainty information, the greater the influence of noise or interference on the feature, and the lower the reliability. The lower the value, the higher the confidence level of the feature in the current scenario.

[0098] Optionally, obtaining uncertainty information includes: inputting intermediate audio features and intermediate video features into the uncertainty prediction network respectively, so as to obtain uncertainty information generated by the uncertainty prediction network based on the feature quality of the input features.

[0099] In this embodiment of the application, the uncertainty prediction network is used to estimate the confidence level and stability of the input features. The uncertainty prediction network can first perform normalization and linear mapping on the input features (i.e., intermediate audio features or intermediate video features) to unify the features to a preset uncertainty subspace dimension. Through this mapping, the distribution differences between features of different modalities or different levels can be eliminated, providing a unified feature space for subsequent uncertainty estimation.

[0100] In one example, the uncertainty prediction network can extract temporal contextual information related to feature quality through a 1–2 layer lightweight temporal encoder. This encoder aggregates dynamic change features between adjacent frames to capture perceptible information such as feature stability, abrupt changes, texture jitter, or energy fluctuations in the temporal dimension, thus providing contextual support for uncertainty assessment. The specific structure of the temporal encoder can be, but is not limited to, the following: a single-layer TransformerEncoder to capture long-range dependencies; a gated one-dimensional convolution (Gated Conv1D) to model local temporal dependencies; or a bidirectional lightweight recurrent neural network (such as a bidirectional GRU) to extract bidirectional temporal dynamic features.

[0101] After completing the temporal feature encoding, the encoding result is input into a two-layer multilayer perceptron (MLP), and the uncertainty information is output through the sigmoid activation function. The output uncertainty value can be regarded as the inverse confidence index of the feature in each dimension, with a value range of [0,1], which is used to quantify the reliability and stability of the current feature.

[0102] In one example, the training process for the uncertainty prediction network includes: constructing two slightly perturbed inputs for intermediate audio features or intermediate video features of the same sample. The perturbation methods may include, but are not limited to, mild time-frequency masking, random jitter, or additive small noise; after the two perturbed inputs are processed by the same subsequent discriminator, output representations are obtained respectively. and Next, the frame-by-frame or dimension-by-dimensional differences or variances of the two outputs are calculated, and this is used as the uncertainty of the soft target. ,For example ,in, This represents the normalization operation, and during gradient propagation... Stopping backpropagation (stop-gradient) is used to prevent target collapse.

[0103] Uncertainty in prediction results through mean squared error (MSE) loss For supervised learning, the loss function is defined as follows: By minimizing the above loss function, the uncertainty prediction network can learn the confidence fluctuation characteristics of the input features under different perturbations, thereby obtaining more accurate uncertainty estimation results.

[0104] Optionally, obtaining uncertainty information includes: performing multiple forward propagations with random deactivation on the intermediate audio features and intermediate video features respectively, and determining the corresponding uncertainty information based on the statistical dispersion of the output of the multiple forward propagations.

[0105] Specifically, for intermediate audio features, the same feature discrimination head is used for execution. The forward propagation with random inactivation (i.e., Monte Carlo Dropout inference) yields... Output results: Regarding the above The variance of each output is calculated on a frame-by-frame or dimension-by-dimensional scale to obtain the uncertainty statistics of the audio modality: .

[0106] Similarly, for intermediate video features, the same random deactivation forward propagation operation is performed to obtain the uncertainty statistics of the video modalities: .

[0107] In order to map the above variance values ​​to a unified probability space interval The uncertainty information of the audio can be obtained by using a linear calibration function. Uncertain information related to video : ,in, This represents the Sigmoid activation function. and These are adjustable calibration coefficients used to control mapping sensitivity and bias.

[0108] The advantage of determining uncertainty information through forward propagation is that it can be directly integrated into the existing model architecture without the need for additional network modules. It has good scalability and flexibility, and can reflect the stability of input features under Dropout randomness based on the variance estimation results of the model output, thus providing a reliable uncertainty measure at the feature level.

[0109] Optionally, the aforementioned uncertainty information determined by forward propagation can be combined with an uncertainty prediction network for cross-validation or as an auxiliary supervision signal to improve the stability and generalization ability of the overall uncertainty modeling.

[0110] S430. Calculate the cross-modal attention weights between intermediate audio features and intermediate video features based on uncertainty information.

[0111] When performing cross-modal attention computation, attention to low-confidence modalities can be dynamically reduced by using uncertainty information.

[0112] In one example, the initial attention weights between intermediate audio and intermediate video features are calculated, and then adjusted based on uncertainty information to obtain cross-modal attention weights. For instance, taking the cross-modal attention weights of intermediate audio features on intermediate video features as an example: based on the query vector of the intermediate audio features and the key vector of the intermediate video features, the initial attention weight values ​​between the intermediate audio features and intermediate video features are calculated; the uncertainty information corresponding to the intermediate video features is aggregated along a preset feature dimension to obtain an uncertainty sequence; the initial attention weight values ​​are corrected based on the uncertainty sequence to obtain intermediate attention weight values; and the intermediate attention weight values ​​are normalized to obtain the cross-modal attention weights between the intermediate audio features and intermediate video features.

[0113] First, linear mapping is performed on the intermediate audio features and intermediate video features to obtain the corresponding Q, K, and V. Then, the similarity between Q and K is calculated by scaling the dot product to obtain the initial attention matrix. .

[0114] First, the uncertainty information corresponding to the intermediate video features is aggregated along a preset feature dimension to obtain a frame-level uncertainty sequence. Among them, the aggregation methods include mean aggregation, weighted mean aggregation, etc., and the specific method can be adaptively selected according to the feature distribution.

[0115] To suppress the impact of high-uncertainty frames in cross-modal association, frame-level uncertainty is broadcast to the column direction of the attention energy matrix, and the initial attention matrix is ​​penalized and corrected to obtain the corrected intermediate attention weight values:

[0116] in, It is a vector of all 1s, used for column-direction broadcasting; This is the uncertainty penalty coefficient, used to control the intensity of the penalty. This can be preset using a grid search method or used as a learnable positive scalar during training to achieve adaptive adjustment. This correction can suppress the weight response of high-uncertainty frames at the attention energy level, thereby guiding more attention to modal regions with higher confidence.

[0117] The corrected intermediate attention weights are then Softmax normalized to obtain the cross-modal attention weights of intermediate audio features on intermediate video features. .

[0118] Optionally, to improve computational stability and the controllability of attention distribution, after obtaining the corrected intermediate attention weight values, further numerical stabilization processing is performed by subtracting the maximum value from each row of the intermediate attention weight values ​​to prevent softmax operation overflow. Temperature scaling can also be performed, that is, by using temperature parameters. Controlling the sharpness of attention distribution: Among them, the smaller Values ​​that result in more focused attention, while larger values... The values ​​will be distributed smoothly.

[0119] S440. Based on the cross-modal attention weights, perform feature interaction on the intermediate audio features and intermediate video features to obtain the target audio features and target video features.

[0120] In this embodiment of the application, taking the interaction between audio features and video features as an example, the target audio feature is obtained by weighting the value vector of the intermediate video feature according to the cross-modal attention weight of the intermediate audio feature to the intermediate video feature.

[0121] Through the above process, the uncertainty prediction network can output continuous uncertainty information based on the quality cues of the input features themselves, which can be used as a penalty term in cross-modal attention to participate in weight correction. This allows for prioritizing high-confidence signals and suppressing low-confidence components in complex scenarios, thereby obtaining more robust target audio / video features and providing reliable input for subsequent fusion and object recognition.

[0122] In other embodiments of this application, if the video feature dimension is unreliable, such as due to blurry images, dim lighting, or an obscured mouth, it is desirable to reduce the impact of this feature on subsequent attention fusion. Therefore, unreliable dimensions can be weighted and attenuated using uncertainty information during the Value input stage. Specifically, if the uncertainty of this dimension is high (… If the uncertainty of this dimension is close to 1, then suppress this part of the feature; if the uncertainty of this dimension is low ( If the value is close to 0, then this feature is retained, therefore it can be set. This is the confidence mask; a larger value indicates greater reliability. It is then multiplied element-wise by the feature: This indicates that unreliable feature components are scaled and weakened, while reliable feature components are left unchanged. In subsequent cross-modal attention calculations, It will replace the original Enter the formula: This makes them more inclined to extract information from areas with high confidence.

[0123] Before cross-modal attention computation, the video features (Value part) are element-wise weighted using uncertainty information, so that unreliable visual signals are automatically ignored during fusion, and only high-confidence regions are focused on, thereby improving the stability and robustness of fusion.

[0124] In one embodiment of this application, another object recognition method is provided, which can be applied to... Figure 1 The implementation environment shown is as follows: Figure 5 As shown, this object recognition method is in Figure 2 Based on S210~S240 shown, the process of generating the gated vector in S230 is extended to S510~S540, which are described in detail below.

[0125] S510. The audio and video features are calibrated based on the scene context information to obtain the initial audio and video features.

[0126] Please refer to the calibration process. Figure 3 The embodiments shown will not be described in detail here.

[0127] S520. Perform global pooling on the initial audio features and initial video features respectively to obtain the corresponding target audio pooling features and target video pooling features.

[0128] In this embodiment of the application, global pooling operations are performed on audio features and video features respectively. The pooling operation here can be global max pooling or global average pooling.

[0129] S530. Obtain the scene context features corresponding to the scene context information, and concatenate the target audio pooling features, target video pooling features and context features to obtain the gated input features.

[0130] S540. Input the gated input eigenvalue into a preset dynamic modal contribution network for nonlinear mapping to obtain the gated vector.

[0131] In this embodiment, scene context features are extracted from scene context information, and the target audio pooling features, target video pooling features, and scene context features are concatenated along the feature dimension to obtain gated input features, so as to provide a comprehensive expression of audio and video modalities and scene information.

[0132] The gated input features are fed into a predefined dynamic modality contribution network, which can be a multilayer perceptron (MLP) or other nonlinear neural network structure. Through several fully connected layers and activation functions, the network maps and captures the correlation between modalities and environmental changes. The output of the nonlinear mapping is a gated vector. ,in The dimension of the gate vector is [dimension number], where each element of the gate vector represents the contribution weight of the audio or video modality. The Sigmoid activation can be used to restrict its value to the [0,1] range, thereby representing the weight of each modality to the fusion result.

[0133] By incorporating audio, video, and scene context information through the above process, the contribution of audio and video features in the fusion can be dynamically adjusted. The gating vector generated by the dynamic modality contribution network can precisely control the weights of different modal features in the recognition process. When video or audio is disturbed, its weight can be automatically reduced, thereby enhancing robustness.

[0134] exist Figure 5 Based on the embodiment shown, the feature fusion process in step S240 includes: concatenating the initial audio features, initial video features, target audio features, and target video features to obtain initial fused features; and performing a weighted calculation on the initial fused features in the corresponding feature dimensions according to the gating vector to obtain fused features.

[0135] The process involves concatenating initial audio features, initial video features, target audio features, and target video features to form an initial fusion feature containing all the information. The initial audio and video features provide raw information that has not been processed or disturbed, containing valuable background information. This information can supplement the target audio and video features after cross-modal interaction. By concatenating the initial and target features together, the system can learn the feature changes of audio and video at different stages (such as changes before and after cross-modal interaction), avoiding over-reliance on single-modal features, enhancing the complementarity of multimodal features, and thus improving the learning ability of audio and video multimodal features.

[0136] Based on the initial fused features obtained by concatenation, a gating vector is used to perform a dimensional weighted calculation on the initial fused features. That is, the corresponding value in the gating vector is applied to each dimension of the concatenated initial fused features for weighting. For each dimension of the feature, adjustments are made dimensionally according to the corresponding gating value to obtain the fused features. Optionally, to prevent numerical instability or to prevent excessively large / small weights from having an extreme impact on the fusion result, the weighted features can be normalized to keep them within a reasonable range.

[0137] Through the above process, by concatenating the four features, the correlation between audio and video information can be better captured, enhancing the synergistic effect between the two; the dimensional weighted calculation can dynamically adjust the weight of each feature according to its reliability, making the contribution of each modality more accurate and further improving the accuracy of the fused features.

[0138] In one embodiment of this application, another object recognition method is provided, which can be applied to... Figure 1 The implementation environment shown is as follows: Figure 6 As shown, this object recognition method is in Figure 2 Based on the above, the object recognition process in S240 is extended to S610~S630, which are described in detail below.

[0139] S610. Based on the fusion features, object recognition processing is performed to obtain the initial object and the confidence level of the initial object. The confidence level is used to characterize the credibility of object recognition.

[0140] In this embodiment, preliminary object recognition processing is performed based on the fused features to obtain preliminary recognition results, namely, initial objects and their corresponding confidence scores. These confidence scores characterize the reliability of the recognition results. Specifically, the fused features can be input into a pre-trained classification model (such as a neural network or support vector machine) for object identification. The classification model outputs identity labels and corresponding confidence scores for each candidate object based on the fused features. The model calculates the classification probability distribution for each candidate identity using Softmax output probabilities, assigning a probability value to each object. This probability value represents the model's confidence in the object's identity.

[0141] S620. If the confidence level of the initial object is higher than the preset confidence level, or if the initial object is identified as a registered object, then the identity embedding feature corresponding to the initial object is obtained.

[0142] The confidence level typically ranges from [0,1]. A higher confidence level indicates greater confidence in the recognition result. If the confidence level of the initial object is greater than a preset first confidence threshold (e.g., 90%), the recognition result is considered reliable. If the confidence level is low, the object recognition result can be disregarded.

[0143] If an object has already been registered in the object system, it means that the object's identity is reliable. The determination of a registered object is based on the object's identity label. For example, if the confidence of the initial object output by the classification model is greater than the second confidence threshold (e.g., 80%), it is determined whether the identity label of the initial object matches the object label registered in the database. If they match, it means that the object has been registered.

[0144] If the confidence level of the initial object is higher than the preset first confidence threshold, or if the initial object is identified as a registered object, the identity embedding feature of the initial object is obtained. The identity embedding feature is a feature vector generated based on the object's unique speech features (such as voiceprint features), which can uniquely represent the object's individualized feature information. The identity embedding feature is generated by sampling the object's voice multiple times during the previous registration process to extract its unique speech features and by a specific voiceprint recognition model (such as ArcFace or CosFace).

[0145] S630. Correct the fusion features based on the identity embedding features to obtain the target fusion features, and perform object recognition processing based on the target fusion features.

[0146] In this embodiment of the application, the identity embedding features of the object are used to correct the fused features in order to enhance the features that are strongly correlated with the identity of the object, while suppressing noise or features that are inconsistent with the identity.

[0147] The corrected target fusion features are used for final object recognition. At this point, the features have been corrected based on object identity, so the recognition result is more accurate. The corrected target fusion features are then input into the classification model for recognition again to obtain the final target object.

[0148] In this embodiment of the application, by calculating the confidence level after preliminary identification, and correcting the fused features based on the identity embedding features when facing a registered object, the accuracy of identifying known objects can be greatly improved.

[0149] Optionally, the target fusion feature is obtained by correcting the fusion feature based on the identity embedding feature, including: inputting the identity embedding feature into a preset correction network to obtain affine transformation parameters for the object; and performing affine transformation processing on the fusion feature based on the affine transformation parameters for the object to obtain the target fusion feature.

[0150] Embed the identity features The input is fed into a correction network, which can be a lightweight multilayer perceptron (MLP) or other nonlinear neural network structure. The network maps through several fully connected layers and nonlinear activation functions to generate the original quantities of the gain and bias terms for the fused features. ;

[0151] in, and These represent the weight matrices for the gain and bias branches, respectively. It is a non-linear activation function. For feature dimensions.

[0152] Optionally, to prevent the affine parameters from causing excessive feature scale or drift, the original values ​​of gain and bias can be smoothly mapped and constrained; for example, a scaling-translation sigmoid function can be used for parameter constraints, such as: ; For example, the tanh constraint: ;in, Control the gain adjustment range, Control the bias range. This represents the truncation function.

[0153] In other embodiments of this application, a similarity constraint between the fused features and the identity embedding features may be further introduced to adaptively adjust the correction magnitude, for example, by calculating the cosine similarity between their pooled representations: Then It can be added as an additional scalar input to the generator network head, or used to scale affine transformation parameters, for example: When the identity similarity is low, the correction magnitude will be automatically weakened to prevent erroneous identity features from interfering with the fusion result.

[0154] The fused features are processed by affine transformation based on the affine transformation parameters: Corrected fusion features It better matches the embedding distribution of the target object in the feature space, thereby enhancing the ability to distinguish the object.

[0155] The above scheme generates affine transformation parameters (gain / bias) based on identity embedding through a correction network and performs element-wise affine correction on the fused features. Under the premise of ensuring numerical stability and computational efficiency, it achieves individualized enhancement and robust alignment of known objects, significantly improving the accuracy, stability and robustness of recognition.

[0156] This application provides another object recognition method, which can be applied to... Figure 1 The implementation environment shown is as follows: Figure 7As shown, this object recognition method is in Figure 2 Based on S210~S240 shown, an object recognition model training process is added, whereby the object recognition model is used to perform object recognition processing to identify objects in the target video. That is, all of the above steps S210~S240 can be executed by the object recognition model; S710~S740 are described in detail below.

[0157] S710: Obtain the model to be trained, as well as the sample audio features and sample video features from the sample video.

[0158] In this embodiment of the application, a model to be trained is first obtained. This model can perform audio and video feature extraction, cross-modal fusion, and object recognition to identify the identity of the object from multimodal input.

[0159] Optionally, a preset initialization model and a sample object dataset containing a single object are obtained. The preset initialization model is trained based on the sample object dataset to obtain an intermediate model. The intermediate model is trained based on the sample audio features and sample video features to obtain the model to be trained.

[0160] In this embodiment, phased training allows the model to have a better feature base and convergence performance before formal training. The first phase uses a sample object dataset containing single object samples to pre-train the initial model, obtaining an intermediate model with preliminary multimodal perception capabilities. The sample audio and video content in this dataset comes from the same object, enabling the model to learn the temporal synchronization relationship, feature coupling patterns, and speech-visual correlation between audio and video modalities. The second phase uses complete sample audio and video features from sample videos to further train the intermediate model, resulting in the final model to be trained. These sample videos typically contain background noise, multiple objects, and different speaking states.

[0161] Extract the corresponding sample audio features and sample video features from the sample video, which has an object identity label.

[0162] S720. Perturbations are added to the sample audio features and sample video features respectively to obtain sample adversarial audio features and sample adversarial video features.

[0163] To enhance the model's robustness to input noise, environmental changes, or malicious attacks, small-amplitude adversarial perturbations are introduced into the audio and video features during the training phase, generating corresponding sample adversarial audio and video features. The perturbation refers to applying a controlled small perturbation vector to the original feature space, enabling the model to maintain correct recognition even with slight changes in the input. For example, the sample audio and video features are calibrated based on the sample scene context information corresponding to the sample video, resulting in initial sample audio and video features. This can be achieved using either the Fast Gradient Signed Method (FGSM) or a random perturbation method, slightly shifting the initial sample audio and video features along the feature dimension. The process of obtaining the sample scene context information can be found in [link to relevant documentation]. Figure 2 For the calibration process shown in the example, please refer to [link / reference]. Figure 3 The embodiments shown will not be described in detail here.

[0164] S730. Perform feature discarding processing on the sample audio features and sample video features respectively to obtain sample discarded audio features and sample discarded video features.

[0165] To further improve the robustness of the model under conditions of missing, distorted, or occluded modal information, feature discarding processing is performed on audio and video features respectively, resulting in sample-discarded audio features and sample-discarded video features. Feature loss processing includes, but is not limited to: temporal discarding: randomly discarding audio or video frames to simulate partial frame loss; dimensional discarding: randomly masking some feature dimensions to simulate feature corruption; and region discarding: randomly masking regions in the video feature map to simulate partial image occlusion.

[0166] Optionally, feature discarding processing is performed to obtain sample discarded audio features and sample discarded video features, including: calibrating the sample audio features and sample video features according to the sample scene context information corresponding to the sample video to obtain sample initial audio features and sample initial video features; generating a corresponding discard mask according to the sample scene context information, sample initial audio features, and sample initial video features; and performing feature discarding processing on the sample initial audio features and sample initial video features according to the discard mask to obtain sample discarded audio features and sample discarded video features.

[0167] The sample scene context information is used to reflect the overall state of the environment in which the sample video is located. By using the sample scene context information, the mask generation process can simulate modal degradation modes in a targeted manner according to specific scene characteristics. For example, when the noise or reverberation level in the scene is high, the proportion of audio modal drops can be appropriately increased to simulate the situation where the speech signal is covered or distorted by noise. When the video clarity is low or the face is occluded, the probability of video modal drops can be increased to simulate scenarios such as video blurring and missing images. If there are multiple people speaking or lip movements are out of sync in the scene, some features can be randomly dropped in both modalities at the same time to approximate the real complex voice interaction environment.

[0168] The generated dropout mask is not only used for simple random deactivation features, but also, under the guidance of context awareness, can realistically simulate complex interference situations in multimodal scenarios, such as background noise occlusion, obscuring of a person's face or lips, suppression of speech signals, and blurring and loss of video frames.

[0169] Optionally, a corresponding drop mask is generated based on the sample scene context information, the initial audio features of the sample, and the initial video features of the sample, including: inputting the sample scene context information, the initial audio features of the sample, and the initial video features of the sample into a preset adversarial generative network to obtain drop information used to describe the drop probability of feature location; and sampling the drop information through a Gumbel distribution to obtain the drop mask.

[0170] In this embodiment, sample scene context information, initial audio features, and initial video features are input into the Generative Adversarial Network (GAN). To enable the network to jointly determine the dropout mode using the context and the current segment content, lightweight alignment and fusion are first performed. That is, if the time dimensions of the initial audio features and the initial video features are different, the longer sequence can be downsampled, or the shorter sequence can be interpolated to align the time steps of the two modalities. Then, the sample scene context information is input into the MLP for feature projection to obtain a conditional vector aligned with the time step. The conditional vector can be concatenated to the audio and video features of each time step through broadcasting or time step copying, so that the network input simultaneously contains multimodal semantic information and contextual information, providing conditional guidance for subsequent dropout mode generation.

[0171] In one example, the adversarial generative network includes a shared backbone network and modal branch heads. The shared backbone can be a lightweight Transformer / Conv1D / GRU with 1–2 layers, extracting temporal and local statistics related to dropability. The modal branch heads set independent output branches for audio and video modalities, and each branch outputs an element-wise binary discrimination result (i.e., whether to drop at each feature location). The logits output by the branch heads are the dropout information, which is used to describe the log odds of dropping each element.

[0172] To maintain gradient propagation over the mask, a differentiable approximation of the mask is performed using a Gumbel–Sigmoid sampling mechanism (a second-class special case of the Concrete distribution). For example, an independent random number is generated for each feature location. And calculate the corresponding Gumbel noise: The audio and video logits are respectively... and The discard mask can then be obtained through temperature-based sampling. ,in, This represents the Sigmoid function. For temperature parameters; when When the value is small (e.g., gradually degrading from 0.5 to 0.1), the sampling result approaches a 0 / 1 discrete mask; when When the mask value is larger, the distribution of mask values ​​is smoother, which can provide better gradient stability in the early stages of training.

[0173] In this embodiment of the application, the adversarial generative network first outputs the log odds of the mask, and then obtains the drop mask through Gumbel sampling. This mask is driven by the scene context information and the initial audio / video features of the current segment, and can adaptively generate the drop pattern that is most likely to interfere with recognition.

[0174] The generated drop masks are applied to the initial audio features and the initial video features respectively to achieve controlled random masking of information at the feature dimension level. For example, the drop mask of the initial audio features of the sample is multiplied element-wise with the initial audio features of the sample to obtain the sample dropped audio features; the drop mask of the initial video features of the sample is multiplied element-wise with the initial video features of the sample to obtain the sample dropped video features.

[0175] In this context, each element value in the discard mask is used to indicate the degree of preservation of the corresponding feature dimension. That is, when the mask value is 1, it means that the feature of that dimension is completely preserved; when the mask value is 0, it means that the feature of that dimension is completely masked or set to zero; when the mask value is between 0 and 1 (e.g., 0.3 or 0.7), it means that the feature of that dimension is partially suppressed or blurred, thereby achieving continuous and adjustable control over the intensity of feature information.

[0176] This element-weighted masking mechanism can simulate different degrees of modal degradation in a more granular way, such as the attenuation of audio signals due to noise interference, and the partial occlusion or blurring of facial areas in video frames. Compared with traditional random discarding or whole modal masking methods, the masking method can flexibly adjust the information retention ratio at the feature dimension level, allowing the model to be exposed to more diverse input perturbations during training, thereby learning stronger feature redundancy utilization capabilities and anti-interference robustness.

[0177] S740. Train the model to be trained based on the sample adversarial audio features, sample adversarial video features, sample discarded audio features, and sample discarded video features to obtain the object recognition model.

[0178] During the training phase, the sample adversarial audio features and sample adversarial video features are input into the model to be trained, and the sample discarded audio features and sample discarded video features are input into the model to be trained. A joint training strategy is adopted to optimize the model end-to-end, and the final object recognition model is obtained.

[0179] Optionally, during the training phase, the model receives four types of sample inputs simultaneously: original features, adversarial sample features, modal discard features, and object identity labels. The model jointly optimizes on the original samples, adversarial samples, and discarded samples. Through the supervision signals of diverse inputs, the network is guided to learn more robust and discriminative multimodal feature representations, thereby improving the final object recognition accuracy.

[0180] In this embodiment, through joint training of adversarial perturbation and feature discarding, the model can effectively resist the effects of input noise, modality loss and environmental changes, improve the overall anti-interference ability, and thus exhibit higher robustness and recognition accuracy in complex real-world scenarios.

[0181] Optionally, joint training of the model to be trained includes: obtaining a first recognition result obtained by the model to be trained performing object recognition processing on sample adversarial audio features and sample adversarial video features, and a second recognition result obtained by the model to be trained performing object recognition processing on sample audio features and sample video features; generating adversarial loss based on the differences between the first recognition result, the second recognition result and the object labels corresponding to the sample video; obtaining a third recognition result obtained by the model to be trained performing object recognition on sample discarded audio features and sample discarded video features; generating discard loss based on the differences between the third recognition result and the object labels corresponding to the sample video; generating model loss based on adversarial loss and discard loss, and updating the model parameters of the model to be trained based on model loss to obtain an object recognition model.

[0182] Among them, the sample audio features and sample video features are called clean samples, and the sample adversarial audio features and sample adversarial video features are called adversarial samples. If the clean samples are calibrated samples, then the adversarial samples are also calibrated samples to ensure the consistency of the two in the feature space. The adversarial samples and clean samples are input into the model to be trained, which will output the first recognition result and the second recognition result respectively. The recognition result is the object label predicted by the model.

[0183] To ensure the model can correctly identify objects on both clean and adversarial samples, an adversarial loss is constructed to constrain the model to maintain classification consistency when faced with perturbation inputs. Cross-entropy can be used as an example.

[0184] in, The sample object identity tags for the sample video; Represents the cross-entropy loss function; and These represent the output probability distributions of the model on adversarial examples and clean examples, respectively.

[0185] Audio and video features that are discarded samples are called discarded samples. To ensure that the model can still accurately identify objects when modal information is incomplete or some features are missing, a discard loss is constructed:

[0186] in, This is the third identification result.

[0187] In this embodiment, the model to be trained and the adversarial generative network are optimized in the form of objects. The goal of the model to be trained is to minimize the overall loss in order to improve the recognition ability on clean samples, adversarial samples and dropped samples. The goal of the adversarial generative network is to maximize the dropout loss in order to generate the most challenging dropout mask, so that the model to be trained is forced to learn a stronger robust representation during the training process.

[0188] In the embodiments of this application, the model loss is obtained by weighting the adversarial loss and the dropout loss.

[0189] Optionally, the sample speech information contained in the sample audio features is predicted by a preset classification model, and a speech activity detection loss is generated based on the difference between the sample speech information and the real speech information; the temporal alignment error between the sample speech features and the sample video features is calculated, and a synchronization loss is generated based on the temporal alignment error; the similarity between the sample audio features and the sample video features is calculated to generate an audio-visual interaction loss; and a model loss is generated based on the speech activity detection loss, synchronization loss, audio-visual interaction loss, adversarial loss, and dropout loss.

[0190] First, the sample audio features (or initial sample audio features) are input into a pre-defined classification model, which is used to predict the probability of speech presence at each time step. The probability output by the model reflects the confidence level of whether a speech signal exists in the current frame. The speech activity detection loss can be defined by calculating the binary cross-entropy frame by frame.

[0191] in: This indicates the total number of frames in the audio sequence; , where 1 indicates that there is speech in the frame, and 0 indicates silence.

[0192] To improve the temporal synchronization of audio and video features, the similarity between the two at different time offsets is calculated, including the mean cosine similarity at zero latency (i.e., aligned state): Then in the time offset window Calculate the maximum misalignment similarity: ,in, This indicates the degree to which audio and video can still be aligned even when misaligned. The larger value indicates that audio and video features are easily confused or mismatched in time.

[0193] The goal of constructing a synchronization loss is to maximize zero-latency similarity. Minimize the maximum misalignment similarity To prevent feature misalignment; for example:

[0194] in, This is a balancing coefficient used to control the weight of penalizing misalignment similarity.

[0195] To further enhance cross-modal semantic consistency and complementarity, an audio-video interaction loss is constructed. This loss aims to make the audio and video embeddings of the same object more similar in the feature space, while maintaining the separability of features of different objects. This can be achieved by maximizing the cosine similarity of positive samples (audio-video pairs in the same video). .

[0196] in, and Let and represent the sample audio features and sample audio features at the corresponding time points, respectively. This loss is achieved by minimizing . , that is, maximize This encourages models to learn consistent representations across modalities.

[0197] In this embodiment, the speech activity detection loss is used to constrain the model to accurately identify the start and end boundaries of speech, and reduce the interference of silence or background noise on subsequent cross-modal alignment; the synchronization loss enables the model to learn more accurate audio and video temporal alignment relationships, and the audio and video interaction loss encourages the model to learn consistent representations across modalities. The speech activity detection loss, synchronization loss, audio and video interaction loss, adversarial loss and dropout loss are then weighted and summed to obtain the final model loss.

[0198] By introducing speech activity detection loss, synchronization loss, and interaction loss during the training phase, and jointly optimizing them with adversarial / dropout loss, the model benefits simultaneously in four dimensions: temporal alignment, semantic alignment, anti-interference, and anti-degradation. This significantly improves the accuracy, stability, and robustness of object recognition in real-world complex video scenarios.

[0199] For ease of understanding, this application also provides an object recognition method based on an object recognition model, in which the target video to be recognized is input into the object recognition model to identify the identity of the target object in the target video; wherein, the object recognition method can be executed by a terminal device or a server.

[0200] In real-world applications, video environments are highly dynamic, with factors such as ambient noise levels, lighting conditions, and image clarity potentially changing at any time. To address this complex and ever-changing scenario, the object recognition method proposed in this application does not directly rely on the original audio and video features for recognition. Instead, through an environmental perception and adaptive calibration mechanism, the model can dynamically calibrate the input audio and video features based on environmental characteristics (such as noise intensity, multiple people speaking, and video blur) before recognition to obtain higher-quality features.

[0201] Furthermore, audio and video are not simply superimposed; sometimes audio is more reliable (e.g., the picture is blurry but the sound is clear), and sometimes video is more reliable (e.g., lip reading is clear but the sound is noisy). The object recognition model, through a dynamic fusion mechanism, can evaluate and adjust the contribution of each modality in real time, focusing on utilizing the more reliable modality information. At the same time, for the detected uncertain or low-confidence features, the model will automatically weaken their influence during the fusion process, thereby avoiding recognition bias caused by unreliable modalities and avoiding being misled by unreliable information.

[0202] For objects already registered in the system, after obtaining the initial recognition results, the model will combine the object's historical identity embedding features to further personalize the current fused features, which can enhance the model's ability to distinguish known objects and make the final recognition results more accurate and stable.

[0203] To make the object recognition model more robust and resistant to being fooled or having missing information, in addition to regular adversarial training during the training phase, the model is also made to simulate modal discarding situations, that is, to pretend that some audio and video information suddenly disappears or is damaged. Through this extreme training, the model learns to accurately identify objects even when the information is incomplete or interfered with.

[0204] In the embodiments of this application, such as Figure 8 As shown, the object recognition model includes an audio and video preprocessing module, a multimodal feature extraction module, a context-aware feature calibration (CAFC) module, a cross-modal interaction and dynamic fusion module, an object recognition module, and an object identity-driven feature rectification (SIDFR) module.

[0205] The audio and video preprocessing module is used for audio preprocessing and video preprocessing. Audio preprocessing includes framing the raw audio of the video (e.g., sampling rate 16kHz), windowing it, and calculating its logarithmic Mel spectrum. As a fundamental acoustic characteristic; for example, continuous waveforms Cut into lengths of Small segments (frame shift) ), per-frame windowing function (e.g., Hamming window), perform FFT on each frame to obtain the power spectrum. The Mel power is obtained by passing the Mel filter bank (which maps linear frequencies to the Mel axis, which is close to the human ear's perception). Then take the logarithm to obtain the acoustic features. The process involves using a deep learning-based denoising model (such as Wave-U-Net) to reduce ambient noise interference in the original audio. Video preprocessing includes face detection on video frames (e.g., using RetinaFace), cropping and aligning face regions, extracting the lip region as the key visual feature region from the aligned face, and performing normalization to obtain... .

[0206] The multimodal feature extraction module is responsible for extracting deep semantic features from the preprocessed data; this module includes an audio feature encoder, which uses a pre-trained large audio model (such as the BEATs model based on the Transformer architecture) to extract deep semantic features from the preprocessed data. Encoded as high-dimensional audio features The multimodal feature extraction module also includes a video feature encoder, which uses a pre-trained large visual model (such as the visual encoder part of CLIP) to extract features. Encoding as high-dimensional video features .

[0207] The Context-Aware Feature Calibration (CAFC) module dynamically and adaptively adjusts features based on real-time scene context information to enhance useful signals and suppress interference. This module includes a lightweight context-aware network, Context_Net, which receives raw audio and video features (or their pooled representations) as input and predicts or estimates the current scene context information. ,in, This can include: noise level, reverberation level, number of objects, confidence level of speech activity detection, video clarity, etc., for example:

[0208] This module also includes two independent calibration subnetworks. and They are As input, affine transformation parameters (gain and bias) for audio and video features are dynamically generated: ;

[0209] in, , .

[0210] Element-wise affine transformations are performed on each time step / dimension of the high-dimensional audio features and high-dimensional video features respectively: ;

[0211] in Representing element-wise multiplication, the Sigmoid activation function ensures that the gain parameter is within a reasonable range, avoiding feature oversaturation or undersaturation. In this way, CAFC can dynamically enhance or suppress parts of features that match or do not match the current scene according to the environment.

[0212] The cross-modal interaction and dynamic fusion module introduces an uncertainty perception mechanism and a dynamic contribution network when fusing audio and video information, enabling more refined and intelligent fusion. The cross-modal interaction and dynamic fusion module includes a cross-modal interaction module, which is used to perform cross-modal interaction through a modal self-attention mechanism and an uncertainty-perceived cross-modal attention mechanism.

[0213] Modal self-attention mechanism refers to the process of focusing on the initial audio features after CAFC calibration. and initial video features Multi-head self-attention mechanisms are applied separately to capture long-term dependencies and important features within each modality to obtain intermediate audio features. and intermediate video features : ;

[0214] Uncertainty-aware cross-modal attention mechanism refers to explicitly modeling and utilizing the confidence of each modality feature during modal interaction, making the model more inclined to information with high confidence and avoiding the negative impact of low-quality modalities.

[0215] Here, an uncertainty score is estimated for each feature dimension, including Monte Carlo Dropout and an uncertainty prediction network, where the uncertainty prediction network can be a lightweight network called Uncertainty_Predictor, which receives... or As input, the output is the uncertainty score for the corresponding dimension. and : ;

[0216] in, Mapped to Sigmoid (0 = high certainty, 1 = high uncertainty).

[0217] When calculating cross-modal attention weights, an uncertainty score is introduced as a penalty term. For example, calculating the target attention weights for audio to video includes: for intermediate audio features... The query vector is obtained by performing feature mapping. Features of intermediate video Feature mapping is performed to obtain the key vector. Sum value vector :

[0218] Calculate the initial attention weights:

[0219] Aggregating the video's uncertainty along the feature dimension to the time axis (e.g., taking the mean of the last dimension) yields an uncertainty sequence (i.e., a penalty term):

[0220] According to the penalty item (broadcast to) The initial attention weights are corrected to obtain the cross-modal attention weights of the intermediate audio features on the intermediate video features: ;

[0221] The target audio features are obtained by weighting the value vector using cross-modal attention weights.

[0222] Calculating the attention weights of video to audio includes: considering the intermediate video features. The query vector is obtained by performing feature mapping. For intermediate audio features Feature mapping is performed to obtain the key vector. Sum value vector :

[0223] Calculate the initial attention weights:

[0224] Aggregating audio uncertainty along the feature dimension to the time axis (e.g., taking the mean of the last dimension) yields the penalty term:

[0225] According to the penalty item (broadcast to) The initial attention weights are corrected to obtain the cross-modal attention weights of the intermediate video features on the intermediate audio features: ;

[0226] The target video features are obtained by weighting the value vector using cross-modal attention weights.

[0227] The cross-modal interaction and dynamic fusion module also includes a Dynamic Modality Contribution Network (DMCN) to replace the traditional fixed gating or splicing fusion. DMCN dynamically adjusts the contribution of each modality to the final fused features in a more refined way based on the current modal features and context.

[0228] Among them, the intermediate audio features calibrated by CAFC Intermediate video features and features after uncertainty-aware cross-modal attention interaction. , The pieces are then joined together to form initial fusion characteristics:

[0229] DMCN is a small neural network (such as a multilayer perceptron) whose inputs include Global pooling representation, DMCN uses the global pooling representation and the context information C_context (or its higher-level feature representation) output by the CAFC module as conditions to learn and generate a dynamic gating vector. For example: ;

[0230] in , They have the same dimensions.

[0231] Finally and Perform element-wise multiplication to obtain the final fused feature. For example:

[0232] In this way, Able to Each dimension is dynamically weighted to achieve fine-grained control over the contribution of each modality, thereby allowing the final fused features to better adapt to the current scenario.

[0233] The object recognition module is used to determine object identity based on fused features; among them, it can... The input is passed to a fully connected layer, and the object classification result y is output through the Softmax function: .

[0234] Or, Mapped to a fixed-dimensional object embedding space e, used for object validation or clustering: .

[0235] The Object Identity Driven Feature Correction Module (SIDFR) is used during the model inference stage to correct the current fused features using the pre-stored, high-quality identity information of objects that have been identified by the system and have high confidence, or known users that have been registered, thereby further improving the accuracy and stability of the identification.

[0236] Among them, when the object recognition module identifies a certain object The SIDFR module is activated when the recognition confidence (Softmax output probability) reaches a preset threshold (e.g., 0.9) or when the object is a known object registered in the system database.

[0237] Retrieve the object from a pre-built object identity embedding database. Corresponding pre-stored identity embedding , It can be the average embedding vector calculated from multiple clean samples.

[0238] SIDFR includes a lightweight corrector subnetwork, RectifyNet (e.g., a small MLP or a network with a gating mechanism), which receives the final fused features. and identity embedding As input; = RectifyNet .

[0239] For example, RectifyNet can learn to generate object-specific affine transformation parameters. , ]: ;

[0240] Through this conditional correction, the model can enhance features that are strongly correlated with the object's identity while suppressing noise or features that are inconsistent with the identity, thereby making the recognition result more likely to be that of the known object.

[0241] Finally based on The final object recognition result will be output. The input is fed into the object recognition module to obtain the final object recognition result.

[0242] It should be noted that, in order to improve the object recognition model's resistance to interference and attacks, this application embodiment also includes a training process for the object recognition model, which can be executed by a terminal device or a server.

[0243] The training process for the object recognition model incorporates multiple adversarial strategies, including input-level adversarial training and feature-level adversarial modality dropout (AMD). Input-level adversarial training involves adding minute, imperceptible perturbations to the training data to make the model insensitive to these perturbations. Algorithms such as FGSM (Fast Gradient Sign Method) or PGD (Projected Gradient Descent) are used to generate adversarial samples. :

[0244] During training, the model to be trained is tested on both the original sample x and the adversarial sample x. Simultaneous learning, with the loss function being:

[0245] Feature-level adversarial modality dropping refers to using an adversarial generative network to simulate the partial or complete loss or damage of modal information at the feature level, forcing the model to be trained to learn robust recognition capabilities under incomplete modal information.

[0246] First, a lightweight adversarial generative network is introduced. Its input is the CAFC-calibrated features. , and context information ,network The goal is to generate a set of binary (0 or 1) drop masks. and To make the generation process differentiable, Gumbel-Softmax is typically used.

[0247] in, M=0 indicates that the corresponding feature dimension is discarded or severely damaged, while M=1 indicates that it is retained.

[0248] Apply the generated mask and The features after discarding are obtained. and : ;

[0249] The model G to be trained (from feature extraction to recognition) and the adversarial network Adversarial training is performed, where the objective of the model G to be trained is: and Minimize the object recognition loss above: .

[0250] Generative Adversarial Networks Objective: To maximize the recognition loss of the main model G, i.e., to generate a dropout mask that makes it most difficult for G to recognize the data. .

[0251] Through this adversarial training, the model being trained is forced to learn to accurately identify objects even when modal information is maliciously damaged or partially missing.

[0252] In this embodiment of the application, in addition to the input-level adversarial loss ( ) and feature-level adversarial mode drop-off loss ( It also incorporates multiple auxiliary task losses to comprehensively improve model performance, including: object classification loss ( ), voice activity detection loss ( ), loss of lip movement synchronization ( ), audio-visual interaction loss ( The object classification loss is the cross-entropy loss between the recognition result obtained by the model under test after processing the sample audio features and sample video features, and the object label of the sample object in the sample video. For other losses, please refer to [link to relevant documentation]. Figure 7 The example shown.

[0253] Total model loss:

[0254] in It is a hyperparameter that balances various losses.

[0255] In the embodiments of this application, the object recognition model can be trained in stages: the first stage is to perform preliminary training on a clean, single-object dataset to obtain an initial model and stabilize the basic recognition ability; the second stage is to gradually introduce data containing background noise, multi-person dialogue, and different speaking states (emotions, accents) to fine-tune the initial model to obtain the model to be trained; the third stage is to perform joint training of the model to be trained on a dataset containing various complex interference and adversarial attack scenarios to obtain the object recognition model, so that the model remains robust under extreme conditions.

[0256] This application describes an apparatus embodiment that can be used to execute the object recognition method described above. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the object recognition method described above.

[0257] This application provides an object recognition device that can be deployed on a terminal device or a server, such as... Figure 9 As shown, the device includes...

[0258] The acquisition module 910 is used to extract audio features and video features from the target video to be processed, and to acquire the scene context information corresponding to the target video.

[0259] The interaction module 920 is used to perform cross-modal attention interaction processing on the audio features and the video features to obtain target audio features and target video features.

[0260] The generation module 930 is used to generate a gating vector based on the audio features, the video features and the scene context information. The gating vector is used to control the contribution of audio features and video features in the fusion process.

[0261] The recognition module 940 is used to perform feature fusion on the target audio features and the target video features according to the gating vector to obtain fused features, and to perform object recognition processing based on the fused features.

[0262] In one embodiment of this application, based on the foregoing scheme, the device further includes a calibration module, configured to calibrate the audio features and the video features according to the scene context information to obtain initial audio features and initial video features; the interaction module is further configured to perform feature interaction processing on the initial audio features and the initial video features through a cross-modal attention mechanism to obtain the target audio features and the target video features.

[0263] In one embodiment of this application, based on the foregoing scheme, the determining module is further configured to generate a chromaticity weight value based on the luminance component, wherein the chromaticity weight value is negatively correlated with the luminance component; calculate the chromaticity amplitude based on the chromaticity component, and calculate the chromaticity index of the pixel by weighting the chromaticity amplitude based on the chromaticity weight value.

[0264] In one embodiment of this application, based on the foregoing scheme, the calibration module is further configured to generate feature adjustment parameters for the audio features and the video features respectively according to the scene context information, wherein the feature adjustment parameters are used to adjust the amplitude and distribution of the features; transform the audio features according to the feature adjustment parameters corresponding to the audio features to obtain the initial audio features; and transform the video features according to the feature adjustment parameters corresponding to the video features to obtain the initial video features.

[0265] In one embodiment of this application, based on the foregoing scheme, the feature adjustment parameters include a gain parameter for adjusting the feature amplitude and a bias parameter for correcting the feature distribution; the calibration module is further configured to obtain a first calibration sub-network for audio feature calibration and a second calibration sub-network for video feature calibration; the scene context information is input to the first calibration sub-network and the second calibration sub-network respectively, wherein the first calibration sub-network generates the gain parameter and the bias parameter corresponding to the audio feature through nonlinear mapping based on the context information related to the audio feature in the scene context information; the second calibration sub-network generates the gain parameter and the bias parameter corresponding to the video feature through nonlinear mapping based on the context information related to the video feature in the scene context information.

[0266] In one embodiment of this application, based on the foregoing scheme, the interaction module is further configured to perform feature processing on the initial audio features and the initial video features respectively through a multi-head self-attention mechanism to obtain intermediate audio features and intermediate video features; obtain uncertainty information corresponding to the intermediate audio features and the intermediate video features respectively, wherein the uncertainty information is used to characterize the reliability of the features in the current scene corresponding to the scene context information; calculate the cross-modal attention weight between the intermediate audio features and the intermediate video features based on the uncertainty information; and perform feature interaction processing on the intermediate audio features and the intermediate video features based on the cross-modal attention weight to obtain the target audio features and the target video features.

[0267] In one embodiment of this application, based on the foregoing scheme, the interaction module is further configured to input the intermediate audio features and the intermediate video features into the uncertainty prediction network respectively, so as to obtain the uncertainty information generated by the uncertainty prediction network according to the feature quality of the input features; or, to perform multiple forward propagations with random deactivation on the intermediate audio features and the intermediate video features respectively, and determine the corresponding uncertainty information based on the statistical dispersion output of the multiple forward propagations.

[0268] In one embodiment of this application, based on the foregoing scheme, the interaction module is further configured to calculate an initial attention weight value between the intermediate audio feature and the intermediate video feature according to the query vector of the intermediate audio feature and the key vector of the intermediate video feature; aggregate the uncertainty information corresponding to the intermediate video feature on a preset feature dimension to obtain an uncertainty sequence; correct the initial attention weight value according to the uncertainty sequence to obtain an intermediate attention weight value; and normalize the intermediate attention weight value to obtain the cross-modal attention weight of the intermediate audio feature on the intermediate video feature. In one embodiment of this application, based on the foregoing scheme, the acquisition module is further configured to perform global pooling on the audio features and the video features respectively to obtain audio pooling features corresponding to the audio features and video pooling features corresponding to the video features; and input the audio pooling features and the video pooling features into a preset context-aware network to obtain the scene context information extracted by the context-aware network from the audio pooling features and the video pooling features.

[0269] In one embodiment of this application, based on the foregoing scheme, the generation module is further configured to calibrate the audio features and the video features according to the scene context information to obtain initial audio features and initial video features; perform global pooling on the initial audio features and the initial video features respectively to obtain corresponding target audio pooling features and target video pooling features; obtain the scene context features corresponding to the scene context information, and concatenate the target audio pooling features, target video pooling features and context features to obtain gated input features; input the gated input features to a preset dynamic modality contribution network for nonlinear mapping to obtain the gated vector.

[0270] In one embodiment of this application, based on the aforementioned scheme, the identification module is further configured to perform object identification processing based on the fusion features to obtain an initial object and the confidence level of the initial object, wherein the confidence level is used to characterize the credibility of object identification; if the confidence level of the initial object is higher than a preset confidence level, or if the initial object is identified as a registered object, then the identity embedding feature corresponding to the initial object is obtained; the fusion features are corrected based on the identity embedding features to obtain a target fusion feature, and object identification processing is performed based on the target fusion feature.

[0271] In one embodiment of this application, based on the aforementioned scheme, the identification module is further configured to input the identity embedding features into a preset correction network to obtain affine transformation parameters for the object; and to perform affine transformation processing on the fusion features according to the affine transformation parameters for the object to obtain the target fusion features.

[0272] In one embodiment of this application, based on the foregoing scheme, the apparatus further includes a training module for acquiring a model to be trained, as well as sample audio features and sample video features in a sample video; adding perturbations to the sample audio features and sample video features respectively to obtain sample adversarial audio features and sample adversarial video features; performing feature discarding processing on the sample audio features and sample video features respectively to obtain sample discarded audio features and sample discarded video features; training the model to be trained based on the sample adversarial audio features, the sample adversarial video features, the sample discarded audio features, and the sample discarded video features to obtain an object recognition model, which is used to perform object recognition processing to identify objects in the target video.

[0273] In one embodiment of this application, based on the aforementioned scheme, the training module is further configured to calibrate the sample audio features and sample video features according to the sample scene context information corresponding to the sample video to obtain initial sample audio features and initial sample video features; generate a corresponding discard mask according to the sample scene context information, the initial sample audio features, and the initial sample video features; and perform feature discarding processing on the initial sample audio features and initial sample video features according to the discard mask to obtain discarded sample audio features and discarded sample video features.

[0274] In one embodiment of this application, based on the aforementioned scheme, the training module is further configured to input the sample scene context information, the sample initial audio features, and the sample initial video features into a preset adversarial generative network to obtain drop information for describing the drop probability of feature locations; and to obtain the drop mask by sampling the drop information through a Günbel distribution.

[0275] In one embodiment of this application, based on the foregoing scheme, the training module is further configured to obtain a first recognition result obtained by the model to be trained performing object recognition processing on the sample adversarial audio features and the sample adversarial video features, and a second recognition result obtained by the model to be trained performing object recognition processing on the sample audio features and the sample video features; generate an adversarial loss based on the differences between the first recognition result, the second recognition result and the object labels corresponding to the sample video; obtain a third recognition result obtained by the model to be trained performing object recognition on the sample discarded audio features and the sample discarded video features; generate a discard loss based on the differences between the third recognition result and the object labels corresponding to the sample video; generate a model loss based on the adversarial loss and the discard loss, and update the model parameters of the model to be trained based on the model loss to obtain the object recognition model.

[0276] In one embodiment of this application, based on the foregoing scheme, the training module is further configured to predict the sample speech information contained in the sample audio features through a preset classification model, and generate a speech activity detection loss based on the difference between the sample speech information and the real speech information; calculate the temporal alignment error between the sample speech features and the sample video features, and generate a synchronization loss based on the temporal alignment error; calculate the similarity between the sample audio features and the sample video features to generate an audio-visual interaction loss; and generate the model loss based on the speech activity detection loss, the synchronization loss, the audio-visual interaction loss, the adversarial loss, and the dropout loss.

[0277] It should be noted that the apparatus provided in the above embodiments and the method provided in the above embodiments belong to the same concept, and the specific way in which each module and unit performs operations has been described in detail in the method embodiments, and will not be repeated here.

[0278] Embodiments of this application also provide an electronic device, including one or more processors and a storage device, wherein the storage device is used to store one or more computer programs, which, when executed by one or more processors, cause the electronic device to implement the object recognition method as described above.

[0279] Figure 10 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0280] It should be noted that, Figure 10 The computer system 1000 of the electronic device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application. The electronic device may be a terminal device or a server.

[0281] like Figure 10 As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on a program stored in read-only memory (ROM) 1002 or a program loaded from storage portion 1008 into random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for system operation. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0282] In some embodiments, the following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1010 as needed so that computer programs read from it can be installed into the storage section 1008 as needed.

[0283] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by processor (CPU) 1001, it performs various functions defined in the system of this application.

[0284] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0285] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and a computer program.

[0286] The units or modules described in the embodiments of this application can be implemented in software or hardware, and can also be located in a processor. The names of these units or modules do not necessarily limit the specific unit or module itself.

[0287] Another aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the object recognition method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0288] Another aspect of this application provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the object recognition method as described above in the various embodiments.

[0289] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0290] Other embodiments of this application will readily conceive of by considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0291] The above description is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

Claims

1. An object recognition method, characterized in that, include: Extract audio and video features from the target video to be processed, and obtain the scene context information corresponding to the target video; Cross-modal attention interaction processing is performed on the audio features and the video features to obtain target audio features and target video features; Based on the audio features, the video features, and the scene context information, a gating vector is generated, which is used to control the contribution of audio features and video features in the fusion process; The target audio features and the target video features are fused according to the gating vector to obtain fused features, and object recognition processing is performed based on the fused features.

2. The method according to claim 1, characterized in that, The process of performing cross-modal attention interaction processing on the audio features and the video features to obtain target audio features and target video features includes: The audio features and video features are calibrated based on the scene context information to obtain initial audio features and initial video features; The initial audio features and the initial video features are processed by a cross-modal attention mechanism to obtain the target audio features and the target video features.

3. The method according to claim 2, characterized in that, The step of calibrating the audio features and video features based on the scene context information to obtain initial audio features and initial video features includes: Based on the scene context information, feature adjustment parameters are generated for the audio features and the video features, respectively. The feature adjustment parameters are used to adjust the amplitude and distribution of the features. The audio features are transformed according to the feature modulation parameters corresponding to the audio features to obtain the initial audio features; The video features are transformed according to the feature adjustment parameters corresponding to the video features to obtain the initial video features.

4. The method according to claim 3, characterized in that, The feature modulation parameters include a gain parameter for adjusting the feature amplitude and a bias parameter for correcting the feature distribution; the step of generating feature modulation parameters for the audio features and the video features respectively based on the scene context information includes: Obtain a first calibration subnetwork for audio feature calibration and a second calibration subnetwork for video feature calibration; The scene context information is input into the first calibration subnetwork and the second calibration subnetwork respectively. The first calibration subnetwork generates the gain parameter and the bias parameter corresponding to the audio feature through nonlinear mapping based on the context information related to the audio feature in the scene context information. The second calibration sub-network generates the gain parameter and the bias parameter corresponding to the video feature through nonlinear mapping based on the context information related to the video feature in the scene context information.

5. The method according to claim 2, characterized in that, The step of performing feature interaction processing on the initial audio features and the initial video features through a cross-modal attention mechanism to obtain the target audio features and the target video features includes: The initial audio features and the initial video features are processed separately using a multi-head self-attention mechanism to obtain intermediate audio features and intermediate video features. The uncertainty information corresponding to the intermediate audio features and the intermediate video features is obtained respectively. The uncertainty information is used to characterize the reliability of the features in the current scene corresponding to the scene context information. Calculate the cross-modal attention weights between the intermediate audio features and the intermediate video features based on the uncertainty information; Based on the cross-modal attention weights, feature interaction processing is performed on the intermediate audio features and the intermediate video features to obtain the target audio features and the target video features.

6. The method according to claim 5, characterized in that, The step of obtaining the uncertainty information corresponding to the intermediate audio features and the intermediate video features includes: The intermediate audio features and the intermediate video features are respectively input into the uncertainty prediction network to obtain the uncertainty information generated by the uncertainty prediction network based on the feature quality of the input features; Alternatively, perform multiple forward propagations with random deactivation on the intermediate audio features and the intermediate video features respectively, and determine the corresponding uncertainty information based on the statistical dispersion of the output of the multiple forward propagations.

7. The method according to claim 5, characterized in that, The step of calculating the cross-modal attention weights between the intermediate audio features and the intermediate video features based on the uncertainty information includes: Based on the query vector of the intermediate audio feature and the key vector of the intermediate video feature, calculate the initial attention weight value between the intermediate audio feature and the intermediate video feature; The uncertainty information corresponding to the intermediate video features is aggregated along a preset feature dimension to obtain an uncertainty sequence; The initial attention weight value is corrected based on the uncertainty sequence to obtain the intermediate attention weight value; The intermediate attention weight values ​​are normalized to obtain the cross-modal attention weights of the intermediate audio features on the intermediate video features.

8. The method according to claim 1, characterized in that, The step of obtaining the scene context information corresponding to the target video includes: Global pooling is performed on the audio features and the video features respectively to obtain the audio pooled features corresponding to the audio features and the video pooled features corresponding to the video features; The audio pooling features and the video pooling features are input into a preset context-aware network to obtain the scene context information extracted by the context-aware network from the audio pooling features and the video pooling features.

9. The method according to claim 1, characterized in that, The step of generating a gating vector based on the audio features, the video features, and the scene context information includes: The audio features and video features are calibrated based on the scene context information to obtain initial audio features and initial video features; Global pooling is performed on the initial audio features and the initial video features respectively to obtain the corresponding target audio pooling features and target video pooling features; Obtain the scene context features corresponding to the scene context information, and concatenate the target audio pooling features, target video pooling features, and context features to obtain the gated input features; The gated input features are input into a preset dynamic modality contribution network for nonlinear mapping to obtain the gated vector.

10. The method according to claim 1, characterized in that, The object recognition process based on the fused features includes: Based on the fusion features, object recognition processing is performed to obtain an initial object and the confidence level of the initial object, wherein the confidence level is used to characterize the reliability of object recognition; If the confidence level of the initial object is higher than the preset confidence level, or if the initial object is identified as a registered object, then the identity embedding feature corresponding to the initial object is obtained; The target fusion feature is obtained by correcting the fusion feature based on the identity embedding feature, and object recognition processing is performed based on the target fusion feature.

11. The method according to claim 10, characterized in that, The step of correcting the fusion features based on the identity embedding features to obtain the target fusion features includes: The identity embedding features are input into a preset correction network to obtain affine transformation parameters for the object; The target fusion feature is obtained by performing an affine transformation on the fusion feature based on the affine transformation parameters for the object.

12. The method according to any one of claims 1 to 11, characterized in that, The method further includes: Obtain the model to be trained, as well as the audio and video features of the sample videos; Perturbations are added to the sample audio features and the sample video features respectively to obtain sample adversarial audio features and sample adversarial video features; The sample audio features and the sample video features are subjected to feature discarding processing respectively to obtain sample discarded audio features and sample discarded video features; The object recognition model is trained based on the adversarial audio features, adversarial video features, discarded audio features, and discarded video features to obtain an object recognition model. The object recognition model is used to perform object recognition processing to identify objects in the target video.

13. The method according to claim 12, characterized in that, The step of performing feature discarding processing on the sample audio features and the sample video features respectively to obtain sample discarded audio features and sample discarded video features includes: The sample audio features and sample video features are calibrated based on the sample scene context information corresponding to the sample video to obtain the initial sample audio features and initial sample video features. Based on the sample scene context information, the sample initial audio features, and the sample initial video features, a corresponding drop mask is generated; Based on the discard mask, feature discarding processing is performed on the initial audio features and initial video features of the sample to obtain discarded audio features and discarded video features of the sample.

14. The method according to claim 13, characterized in that, The step of generating a corresponding drop mask based on the sample scene context information, the sample initial audio features, and the sample initial video features includes: The sample scene context information, the sample initial audio features, and the sample initial video features are input into a preset adversarial generative network to obtain drop information that describes the drop probability of feature locations. The discard mask is obtained by sampling the discard information using a Günbel distribution.

15. The method according to claim 12, characterized in that, The step of training the model to be trained based on the sample adversarial audio features, the sample adversarial video features, the sample discard audio features, and the sample discard video features to obtain an object recognition model includes: The training model performs object recognition processing on the sample adversarial audio features and the sample adversarial video features to obtain a first recognition result, and the training model performs object recognition processing on the sample audio features and the sample video features to obtain a second recognition result. An adversarial loss is generated based on the differences between the first recognition result, the second recognition result and the object labels corresponding to the sample video; The model to be trained performs object recognition based on the discarded audio features and discarded video features of the sample to obtain a third recognition result; A dropout loss is generated based on the difference between the third recognition result and the object label corresponding to the sample video; The model loss is generated based on the adversarial loss and the dropout loss, and the model parameters of the model to be trained are updated based on the model loss to obtain the object recognition model.

16. The method according to claim 15, characterized in that, The step of generating model loss based on the adversarial loss and the dropout loss includes: The sample speech information contained in the sample audio features is predicted by a preset classification model, and a speech activity detection loss is generated based on the difference between the sample speech information and the real speech information. Calculate the temporal alignment error between the sample speech features and the sample video features, and generate a synchronization loss based on the temporal alignment error; Calculate the similarity between the sample audio features and the sample video features to generate an audio-video interaction loss; The model loss is generated based on the voice activity detection loss, the synchronization loss, the audio-visual interaction loss, the adversarial loss, and the dropout loss.

17. An object recognition device, characterized in that, include: The acquisition module is used to extract audio features and video features from the target video to be processed, and to acquire the scene context information corresponding to the target video; The interaction module is used to perform cross-modal attention interaction processing on the audio features and the video features to obtain target audio features and target video features; The generation module is used to generate a gating vector based on the audio features, the video features, and the scene context information. The gating vector is used to control the contribution of audio features and video features in the fusion process. The recognition module is used to perform feature fusion on the target audio features and the target video features according to the gating vector to obtain fused features, and to perform object recognition processing based on the fused features.

18. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more computer programs that, when executed by one or more processors, cause the electronic device to perform the method of any one of claims 1-16.

19. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the processor of the electronic device, causes the electronic device to perform the method described in any one of claims 1-16.

20. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, wherein a processor of an electronic device reads from and executes the computer program to cause the electronic device to perform the method described in any one of claims 1-16.

Images