Methods, devices, electronic equipment and storage media for detecting the authenticity of audio and video behavior

By analyzing audio and video streams using visual, acoustic, and semantic intelligent agents and combining environmental quality parameters to dynamically adjust fusion weights, this approach solves the problem of difficulty in determining the consistency between speech and behavior in traditional technologies, thereby improving the accuracy and stability of behavior authenticity detection in audio and video interaction scenarios.

CN123123645APending Publication Date: 2026-11-17PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611251799.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-18
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Traditional technologies struggle to accurately determine the authenticity of a user's identity and the reliability of their expression when there is surface consistency between voice and behavior. In particular, feature extraction is unstable in complex environments, leading to misjudgments or missed judgments.

Method used

By analyzing facial behavior, vocalization state, and speech content in audio and video streams using visual, acoustic, and semantic agents respectively, and combining environmental quality parameters, the fusion weight of each analysis result is dynamically determined to generate behavior authenticity detection results.

Benefits of technology

It improves the accuracy and stability of behavior authenticity detection in complex audio and video interaction scenarios, and enhances the ability to identify hidden abnormal behaviors such as external prompts and expressions not made by the person in question.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123645A_ABST
    Figure CN123123645A_ABST
Patent Text Reader

Abstract

This application relates to the field of artificial intelligence technology and can be applied to the fields of fintech and digital healthcare. It discloses a method, device, electronic device, and storage medium for detecting the authenticity of audio-visual behavior. The method includes: acquiring and slicing the audio-visual stream of a target object during an audio-visual interaction to obtain multiple interaction segments; parsing the interaction segments to obtain video frame sequences, audio frame sequences, speech-transcribed text, and environmental quality parameters; inputting the video frame sequences, audio frame sequences, speech-transcribed text, and visual representations to a visual agent, an acoustic agent, and a semantic agent, respectively, to generate visual representations, acoustic representations, semantic representations, and corresponding anomaly confidence scores; inputting each representation, each anomaly confidence score, and the environmental quality parameters into a meta-agent to determine the fusion weights of visual, acoustic, and semantic representations, and fusing each anomaly confidence score to obtain the behavior authenticity detection result. This method improves the accuracy and anti-interference capability of audio-visual interaction authenticity detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology and can be applied to the fields of fintech / digital healthcare, particularly to a method, device, electronic device, and storage medium for detecting the authenticity of audio and video behavior. Background Technology

[0002] Remote business processing models based on audio and video interaction are widely used in fintech and healthcare. In financial scenarios, loan interviews and remote account openings are increasingly using video to verify identity and confirm information. In healthcare, services such as remote consultations and online follow-ups also rely on real-time audio and video interaction for information exchange. In these applications, it's typically necessary to determine the authenticity and reliability of a user's identity based on their voice content and behavior in the video. However, in practice, these interactions face certain authenticity risks. For example, in financial interviews, someone else might participate in the question-and-answer session with off-screen prompts, or respond with pre-prepared voice content combined with facial expressions. In remote medical consultations, someone other than the person answering or using external assistance might complete the information expression. In such cases, while the voice information and visible behavior in the video may appear consistent, discrepancies in the actual source or controlling entity can affect the judgment of information authenticity. To address these issues, traditional technologies typically analyze the correlation between audio information and video behavior, extracting and comparing voice and image features to identify anomalies. However, in complex application environments, the above methods still have certain limitations: on the one hand, when there are factors such as changes in illumination, occlusion interference, or background noise, the stability of the extraction of relevant features is affected, thereby reducing the reliability of the judgment results; on the other hand, for cases where speech and behavior are consistent on the surface but there is an actual source of abnormality, the existing analysis methods based on direct correlation are difficult to effectively distinguish; in addition, different types of abnormal behaviors have certain similarities in feature expression, which makes the boundary blurring problem in the discrimination process, and it is easy to misjudgment or missed judgment. Summary of the Invention

[0003] The main technical problem addressed by the embodiments of this application is that traditional technologies struggle to accurately determine the authenticity of a user's identity and the reliability of their expression when voice and behavior are superficially consistent.

[0004] To address the aforementioned technical problems, the first technical solution adopted in this application is: providing a method for detecting the authenticity of audio-visual behavior, comprising: acquiring an audio-visual stream of a target object during an audio-visual interaction process, and slicing the audio-visual stream to obtain multiple interaction segments; parsing the interaction segments to obtain a video frame sequence, an audio frame sequence, a speech-transcribed text, and environmental quality parameters; inputting the video frame sequence to a visual agent, extracting facial behavior features of the target object through the visual agent, and generating a visual representation and a visual anomaly confidence level based on the facial behavior features; inputting the audio frame sequence to an acoustic agent, extracting acoustic behavior features related to the target object through the acoustic agent, and generating an acoustic representation and an acoustic anomaly confidence level based on the acoustic behavior features; inputting the speech-transcribed text... The visual representation and the aforementioned semantic agent are used to analyze the consistency between the speech content corresponding to the speech-transcribed text and the facial behavior state corresponding to the visual representation. Based on the consistency analysis results, semantic representations and semantic anomaly confidence scores are generated. The visual representation, the acoustic representation, the semantic representation, the visual anomaly confidence score, the acoustic anomaly confidence score, the semantic anomaly confidence score, and the environmental quality parameters are input into the meta-agent. The meta-agent determines the visual fusion weight, acoustic fusion weight, and semantic fusion weight. Based on the visual fusion weight, the acoustic fusion weight, and the semantic fusion weight, the visual anomaly confidence score, the acoustic anomaly confidence score, and the semantic anomaly confidence score are fused to obtain the behavior authenticity detection result of the target object in the audio-visual interaction process.

[0005] To address the aforementioned technical problems, the second technical solution adopted in this application is: providing an audio-visual behavior authenticity detection device, comprising: an interaction segment acquisition module, used to acquire the audio-visual stream of a target object during audio-visual interaction, and to slice the audio-visual stream to obtain multiple interaction segments; an interaction segment parsing module, used to perform audio-visual parsing on the interaction segments to obtain a video frame sequence, an audio frame sequence, speech-transcribed text, and environmental quality parameters; a visual feature analysis module, used to input the video frame sequence to a visual agent, extract facial behavior features of the target object through the visual agent, and generate a visual representation and a visual anomaly confidence level based on the facial behavior features; and an acoustic feature analysis module, used to input the audio frame sequence to an acoustic agent, extract acoustic behavior features related to the target object through the acoustic agent, and generate an acoustic representation and an acoustic anomaly confidence level based on the acoustic behavior features; semantic consistency. An analysis module is used to input the speech-transcribed text and the visual representation into a semantic agent. The semantic agent analyzes the consistency between the speech content corresponding to the speech-transcribed text and the facial behavior state corresponding to the visual representation, and generates a semantic representation and semantic anomaly confidence score based on the consistency analysis results. A meta-agent weighting module is used to input the visual representation, acoustic representation, semantic representation, visual anomaly confidence score, acoustic anomaly confidence score, semantic anomaly confidence score, and environmental quality parameters into the meta-agent, and determines the visual fusion weight, acoustic fusion weight, and semantic fusion weight through the meta-agent. A result fusion output module is used to fuse the visual anomaly confidence score, acoustic anomaly confidence score, and semantic anomaly confidence score based on the visual fusion weight, acoustic fusion weight, and semantic fusion weight to obtain the behavior authenticity detection result of the target object during the audio-visual interaction process.

[0006] To solve the above-mentioned technical problems, the third technical solution adopted in the embodiments of this application is: to provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the audio and video behavior authenticity detection method as described above.

[0007] To solve the above-mentioned technical problems, the fourth technical solution adopted in the embodiments of this application is: to provide a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by an electronic device, the electronic device performs the audio and video behavior authenticity detection method as described above.

[0008] Unlike related technologies, this application analyzes the facial behavior, vocalization state, and consistency between speech content and facial behavior state of the target object through visual, acoustic, and semantic agents, respectively. The fusion weights of each analysis result are dynamically determined by the meta-agent in conjunction with environmental quality parameters. In situations with poor image quality, strong background noise, or unstable network transmission, this approach reduces interference with the analysis results, increases the acceptance rate of credible analysis results, and thereby improves the accuracy and stability of behavior authenticity detection in complex audio-visual interaction scenarios. It also enhances the ability to identify hidden abnormal behaviors such as external prompts and expressions not made by the user. Attached Figure Description

[0009] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0010] Figure 1 This is a schematic diagram of the operating environment of the audio and video behavior authenticity detection method provided in the embodiments of this application.

[0011] Figure 2 This is a schematic diagram of the execution flow of the audio and video behavior authenticity detection method provided in the embodiments of this application.

[0012] Figure 3 This is a schematic diagram of the execution flow of determining weights through a meta-agent in the audio and video behavior authenticity detection method provided in this application embodiment.

[0013] Figure 4 This is a schematic diagram of the execution flow of the joint update of the meta-agent in the audio and video behavior authenticity detection method provided in the embodiments of this application.

[0014] Figure 5 This is a schematic diagram of the system structure of the audio and video behavior authenticity detection device provided in the embodiments of this application.

[0015] Figure 6 This is a schematic diagram of the hardware structure of an electronic device that performs the audio and video behavior authenticity detection method according to an embodiment of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Software tools, components, or servers not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0017] It should be noted that, unless otherwise specified, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device schematic diagram or the order in the flowchart.

[0018] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0019] To facilitate understanding of this embodiment, a detailed description of the audio / video behavior authenticity detection method disclosed in this application embodiment will be provided first. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of the operating environment of the audio and video behavior authenticity detection method provided in the embodiments of this application, such as... Figure 1 As shown, the execution subject of the audio-visual behavior authenticity detection method provided in this application embodiment is generally an electronic device with a certain computing power, such as a computer device. In some possible implementations, this audio-visual behavior authenticity detection method can be implemented by a processor calling computer-readable instructions stored in memory. Figure 1 The computer equipment mentioned can be a server. A server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. This can be understood as... Figure 1 The number of computer devices shown is merely illustrative and can be expanded in any number according to actual needs.

[0020] Please continue reading. Figure 2 , Figure 2 This is a schematic diagram of the execution flow of the audio and video behavior authenticity detection method provided in the embodiments of this application, as follows: Figure 2 As shown, it includes the following steps: S1. Obtain the audio and video streams of the target object during the audio and video interaction process, and slice the audio and video streams to obtain multiple interaction segments.

[0021] The target audience refers to users participating in audio-visual interactions and requiring verification of the authenticity of their behavior. The audio-visual stream refers to continuously collected video and audio data during the interaction. Video data records visible behaviors of the target audience, such as facial movements, changes in gaze, and posture changes. Audio data records the target audience's speech content, vocalization state, and sound changes in the interactive environment. Segmentation processing involves dividing the continuous audio-visual stream into multiple time-sequential interaction segments according to preset time windows, speech pauses, interaction rounds, or business nodes, ensuring that each segment corresponds to a relatively complete speech expression and behavioral performance. In fintech scenarios, the audio-visual interaction process can be a loan interview, remote account opening, or online identity verification process, with each interaction segment corresponding to an identity verification, information presentation, or risk inquiry. In digital healthcare scenarios, the audio-visual interaction process can be a remote consultation, online follow-up, or health information confirmation process, with each interaction segment corresponding to a symptom description, supplementary medical information, or medication feedback. Through the processing in step S1 above, continuous, long-term audio and video interaction data can be converted into multiple data segments that can be analyzed independently while maintaining temporal relationships. This facilitates the subsequent extraction of video, audio, and semantic information, and improves the stability of cross-modal feature alignment and behavior authenticity detection.

[0022] S2. Perform audio and video analysis on the interactive segments to obtain video frame sequences, audio frame sequences, speech-transcribed text, and environmental quality parameters.

[0023] Audio and video parsing refers to the separation and structuring of video data, audio data, and the interactive environment state within an interactive segment. A video frame sequence refers to multiple frames of image data extracted from the interactive segment in chronological order; an audio frame sequence refers to sound data extracted from the interactive segment in chronological order; speech-to-text (STM) refers to the text content obtained after speech recognition of the audio frame sequence; and environmental quality parameters reflect the acquisition and transmission quality of the current interactive segment, such as screen illumination, audio signal-to-noise ratio, and network packet loss rate. In fintech scenarios, video frame sequences can record the facial features of a target individual during remote account opening or loan interviews; audio frame sequences can record the target individual's speech; and speech-to-text can reflect the target individual's identity verification or information statements. In digital healthcare scenarios, video frame sequences can record the facial features of a target individual during remote consultations or online follow-ups; audio frame sequences can record the target individual's symptom descriptions or medication feedback; and speech-to-text can reflect the target individual's health information. Through the processing in step S2 above, interactive fragments can be converted into structured data that can be used for subsequent visual analysis, acoustic analysis, semantic analysis and quality assessment, thereby improving the accuracy and compatibility of multimodal data processing.

[0024] S3. Input the video frame sequence to the visual agent, extract the facial behavior features of the target object through the visual agent, and generate visual representation and visual anomaly confidence based on the facial behavior features.

[0025] The visual agent is an intelligent processing unit that analyzes visible facial behaviors in a video frame sequence. Facial behavioral features can include visual characteristics that reflect the interaction state of the target object, such as facial muscle changes, eye area changes, gaze shifts, and facial posture changes. Visual representation is the visual side-intermediate result obtained after encoding facial behavioral features, and visual anomaly confidence is the confidence result of judging whether the target object's facial behavior deviates from the actual interaction state based on the visual representation. In fintech scenarios, the visual agent can analyze the target object's facial movements and gaze changes during identity verification, information presentation, or risk inquiry, thereby identifying whether the target object exhibits obvious lip-syncing, wandering gaze, or abnormal tension. In digital healthcare scenarios, the visual agent can analyze the target object's facial state changes during symptom description, condition supplementation, or medication feedback, thereby identifying whether the facial response during speech expression is natural. Through the processing in step S3 above, the facial behavioral state of the target object can be extracted from the video dimension, providing visual basis for subsequent cross-modal consistency analysis and fusion judgment.

[0026] As an optional implementation, the execution process of step S3 may further include steps S31 to S34.

[0027] S31. Perform face region localization and 3D facial key point extraction on the video frame sequence to obtain facial key point temporal data.

[0028] Face region localization refers to identifying the facial region of a target object within each frame of a video frame sequence, and then aligning, cropping, and normalizing the facial region to reduce the impact of changes in head position, shooting distance, and image size on subsequent feature extraction. 3D facial keypoint extraction refers to extracting 3D keypoints that characterize facial structure and movement from the located face region. These 3D keypoints can cover facial areas such as eyebrows, eyes, nose, mouth, and jaw. Because video frame sequences are temporally continuous, organizing the 3D facial keypoints in consecutive frames according to time order yields temporal data of facial keypoints. This temporal data includes not only facial spatial structure information in a single frame but also keypoint displacement, angle changes, and deformation trends between consecutive frames, providing fundamental data for subsequent analysis of facial muscle coordination and gaze changes.

[0029] S32. Determine facial muscle linkage features based on temporal data of facial key points, and determine gaze wandering features based on changes in the eye area in the video frame sequence.

[0030] Among them, facial muscle coordination features refer to facial movement features characterized by the coordinated changes of multiple facial regions over time. This step determines the facial muscle coordination state of the target object during speech expression by analyzing the displacement amplitude, direction of change, speed of change, and synchronization relationship of key points in areas such as eyebrows, eyes, mouth, cheeks, and jaw in consecutive frames. Gaze-freedom features refer to gaze stability features determined based on changes in eye position, eyelid state, changes in key eye points, and gaze direction in the eye region. This step determines whether the target object's gaze remains within a reasonable interaction range and whether there are trends such as frequent gaze shifts, short-term avoidance, or abnormal lingering by tracking continuous changes in the eye region in a video frame sequence. Facial muscle coordination features and gaze-freedom features reflect the visible behavioral state of the target object from the perspectives of overall facial movement and local eye changes, respectively.

[0031] S33. Determine the facial behavioral characteristics of the target object based on facial muscle coordination characteristics and gaze wandering characteristics.

[0032] Facial behavioral features are visual behavioral features formed by comprehensively expressing facial muscle coordination features and gaze-free movement features. This step aligns the facial muscle coordination features and gaze-free movement features temporally, ensuring that the two types of features correspond to the same time range within the same interaction segment. It then performs scale normalization, noise suppression, and feature fusion processing on both types of features. The fusion process preserves facial muscle movement intensity, facial movement coordination, gaze deviation amplitude, gaze stability, and the temporal relationship between facial movements and gaze changes. The fused facial behavioral features characterize the target object's facial movement state and visual attention state in the current interaction segment, serving as an intermediate basis for the visual agent to judge the authenticity of facial behavior.

[0033] S34. Generate a visual representation based on facial behavioral features, and generate a visual anomaly confidence level based on the degree of deviation between the visual representation and the preset real interactive visual features.

[0034] Visual representation is a vectorized or structured expression of facial behavioral features encoded within the context of the interaction. It describes the visual behavior patterns of the target object in the current interaction segment. Preset realistic interaction visual features refer to pre-determined visual behavior reference features based on real interaction samples or rules, reflecting the normal range of changes in facial muscle coordination, gaze shifts, and facial response rhythm during the real interaction process. This step compares the visual representation with the preset realistic interaction visual features, calculating the degree of deviation. The degree of deviation can be determined by feature distance, temporal differences, differences in motion coordination, or differences in gaze stability. Visual anomaly confidence is a confidence result generated based on the degree of deviation, used to characterize the likelihood that the visual behavior in the current interaction segment deviates from the real interaction state.

[0035] As an example, in fintech scenarios, when a target is remotely opening an account, undergoing loan interviews, or confirming their identity online, the facial region can be located from continuous video frames. Three-dimensional facial key points in areas such as the eyebrows, eyes, mouth, and jaw can be extracted. Further analysis of the continuous changes in these key points during the presentation of identity information, confirmation of business intent, or response to risk inquiries can identify whether facial muscles move naturally, and whether there are signs of wandering gaze such as frequent gaze deviation, short-term avoidance, or abnormal lingering in the eye area. In digital healthcare scenarios, when a target is remotely consulting, undergoing online follow-ups, or confirming health information, the movement trajectory of facial key points during symptom description, supplementary information about their condition, or medication feedback can identify whether facial muscle changes match the natural expression process. Combined with changes in the eye area, this can determine the target's attentional state and facial response state when expressing health information, thus forming facial behavioral features that reflect the target's facial movements, gaze changes, and visual attention.

[0036] Through steps S31 to S34, the visible facial behaviors of the target object during audio-visual interaction can be converted into temporally correlated visual analysis results, enabling facial movements, expression changes, and gaze states to be represented under a unified visual dimension. This processing enhances the ability to identify differences between natural expression states and abnormal visual behavior states, reduces the probability of misjudgment caused by momentary expressions, shooting angles, or short-term occlusions in a single frame image, and provides a stable visual basis for subsequent semantic consistency analysis and multimodal fusion judgment.

[0037] S4. Input the audio frame sequence to the acoustic agent, extract acoustic behavior features related to the target object through the acoustic agent, and generate acoustic representation and acoustic anomaly confidence based on the acoustic behavior features.

[0038] The acoustic agent is an intelligent processing unit that analyzes the vocalization state, sound source location, and background sound in an audio frame sequence. Acoustic behavioral features can include acoustic characteristics reflecting the vocal subject and sound environment, such as main voiceprint features, sound source location features, and background speech interference features. Acoustic representation is an intermediate acoustic result obtained after encoding acoustic behavioral features. Acoustic anomaly confidence is the confidence result based on the acoustic representation to determine whether the current speech contains abnormal risks such as non-target vocalization, off-screen prompts, or background speech interference. In fintech scenarios, the acoustic agent can analyze whether the speech originates from the target subject during remote face-to-face interviews or online identity verification, and whether there are off-screen prompts, responses from others, or background guidance sounds. In digital healthcare scenarios, the acoustic agent can analyze whether the speech source is stable during remote consultations or online follow-ups, and whether there are abnormal voices indicating that the target subject is providing health information. Through the processing in step S4 above, the authenticity of the target subject's vocalization and sound environment anomalies can be identified from the audio dimension, providing acoustic evidence for subsequent multi-agent cross-validation.

[0039] As an optional implementation, the execution process of step S4 above may further include the following steps S41 to S45.

[0040] S41. Extract voiceprint features from the audio frame sequence to determine the main voiceprint features.

[0041] The audio frame sequence consists of sound data obtained from the interaction segment in chronological order. Voiceprint feature extraction involves performing spectral analysis, acoustic parameter extraction, and speaker feature encoding on the speech signals within the audio frame sequence to obtain acoustic information that characterizes the speaker's identity. The dominant voiceprint feature refers to the voiceprint representation of the dominant speaker in the current interaction segment, which can be determined based on the proportion of speech energy, speech duration, voiceprint stability, and speech continuity. This step identifies the primary source of speech from the audio data, providing a fundamental acoustic basis for subsequent judgments regarding the relevance of the speech to the target object.

[0042] S42. Perform sound source localization on the audio frame sequence and determine the sound source directional features corresponding to the main voiceprint features.

[0043] Sound source localization refers to determining the spatial orientation of the sound source relative to the acquisition device based on the arrival time difference, phase difference, energy difference, or spatial acoustic characteristics of different sound signals in the audio frame sequence. The sound source orientation features corresponding to the main voiceprint features refer to the sound direction information associated with the main sound-emitting subject, which may include the sound source azimuth angle, the trend of sound source distance change, and the stability of the sound source orientation. This step, by associating the main voiceprint features with the spatial location of the sound source, can further distinguish between normal sound from the target object, sound from off-screen personnel, and sound sources with abnormal spatial locations.

[0044] S43. Perform blind source separation on the audio frame sequence to obtain the target speech component and background speech component, and determine the background speech interference features based on the background speech component.

[0045] Blind source separation refers to separating mixed audio signals without pre-determining all sound source signals to obtain speech components corresponding to different sound sources. The target speech component refers to the speech portion that matches the main voiceprint features or the vocal state of the target object, while the background speech component refers to other speech or approximate speech signals besides the target speech component. Background speech interference features can be determined based on the energy intensity, duration, frequency of occurrence, speech intelligibility, and overlap with the target speech component, used to characterize whether background prompts, other people's voices, or other external speech interference exist in the current interactive segment.

[0046] S44. Determine acoustic behavior characteristics based on the main voiceprint features, sound source location features, and background speech interference features.

[0047] Among them, acoustic behavior features are audio-side behavioral features formed by comprehensively expressing the identity of the speaker, the spatial location of the sound, and the state of background speech interference. This step can perform time alignment on the main voiceprint features, sound source location features, and background speech interference features, so that different acoustic information corresponds to the same time range within the same interaction segment, and perform feature normalization and fusion processing. The fused acoustic behavior features can characterize the target object's vocal stability, sound source consistency, and background speech interference state in the current interaction segment, and serve as an intermediate basis for the acoustic agent to judge the authenticity of acoustic behavior.

[0048] S45. Generate acoustic representations based on acoustic behavior features, and generate acoustic anomaly confidence based on the degree of deviation between the acoustic representations and preset real interactive acoustic features.

[0049] Acoustic representation is a vectorized or structured expression of acoustic behavioral features after encoding them. It describes the state of the speaker, the spatial state of the sound source, and the state of the background sound in the current interaction segment. Preset realistic interaction acoustic features refer to acoustic reference features predetermined based on real interaction samples or rules, reflecting an acoustic state where the voiceprint is stable, the sound source location is reasonable, and background speech interference is within a normal range during real speech production. This step compares the acoustic representation with the preset realistic interaction acoustic features to determine the degree of deviation of the acoustic representation relative to the realistic interaction acoustic features. The acoustic anomaly confidence score is a confidence result generated based on the degree of deviation, used to characterize the probability that the acoustic behavior in the current interaction segment deviates from the realistic interaction state.

[0050] As an example, in fintech scenarios, when a target undergoes a loan interview, remote account opening, or online identity verification, the system can determine the stability of the primary voice source based on the main voiceprint features in the current interaction segment. It can also determine whether the voice originates from the target's location by combining the sound source's directional features. Simultaneously, it can identify off-screen prompts, whispered instructions from others, or other abnormal voice interference through background voice interference features, thus forming acoustic behavioral features that reflect the speaker, sound source location, and background sound state. In digital healthcare scenarios, when a target undergoes a remote consultation, online follow-up, or health information verification, the system can determine whether the health information is continuously expressed by the target themselves based on the main voiceprint features. It can also determine whether there is someone else speaking on their behalf, supplementary prompts from others, or environmental voice interference by combining changes in sound source location and background voice components, thus forming acoustic behavioral features that reflect the stability of the target's voice, sound source consistency, and the state of the voice environment.

[0051] Through steps S41 to S45, the audio information in the interactive segment can be converted into acoustic analysis results that reflect the speaker, sound source location, and background speech state. This allows the stability of the target object's voiceprint, the consistency of its sound location, and external speech interference to be represented within the same acoustic dimension. This processing enhances the ability to identify anomalies such as non-target object speech, off-screen prompts, responses from others, and background speech interference. It reduces the probability of misjudgment when relying solely on speech content or volume changes for judgment and provides a stable acoustic basis for subsequent semantic consistency analysis and multimodal fusion judgment.

[0052] S5. Input the speech-transcribed text and visual representation into the semantic agent. Analyze the consistency between the speech content corresponding to the speech-transcribed text and the facial behavior state corresponding to the visual representation through the semantic agent. Generate semantic representation and semantic anomaly confidence based on the consistency analysis results.

[0053] The semantic agent goes beyond simply recognizing the meaning of transcribed speech; it jointly analyzes the speech content and the corresponding facial behavioral states in relation to visual representations. Speech content reflects the target's expressive load, rhythm, and information density within the current interaction segment, while facial behavioral states reflect the intensity of facial responses, gaze changes, and cognitive load during the expression process. Semantic representation is an intermediate semantic result obtained by encoding the matching relationship between speech content and facial behavioral states. Semantic anomaly confidence is the confidence result used to determine whether there is a risk of inconsistency between speech content and facial behavioral states based on semantic representation. In fintech scenarios, the semantic agent can analyze whether the complexity of the speech content matches facial reactions, gaze changes, and cognitive load during identity verification, information presentation, or risk inquiry, thereby identifying pre-prepared content, off-site guidance, or unnatural expressions. In digital healthcare scenarios, the semantic agent can analyze whether the speech content and facial behavioral states are consistent during symptom description, medical condition supplementation, or medication feedback, thus assisting in identifying non-personal statements or abnormal auxiliary expressions. Through the processing in step S5 above, the authenticity of the relationship between speech content and facial behavior can be determined from the semantic and visual co-processing dimension, thereby improving the ability to identify situations where the surface audio and visual are consistent but the underlying behavior is abnormal.

[0054] As an optional implementation, the execution process of step S5 may further include steps S51 to S57.

[0055] S51. Segment the transcribed text to obtain multiple text segments, and determine the corresponding speech time interval for each text segment.

[0056] In this context, speech-transcribed text is the text content derived from an audio frame sequence. Sentence segmentation refers to dividing continuous text into multiple text segments with relatively complete expressive meanings based on speech pauses, punctuation boundaries, semantic completeness, or timestamp information. The speech time interval corresponding to each text segment refers to the start and end times of that text segment in the original audio frame sequence. By establishing the correspondence between text segments and speech time intervals, subsequent semantic analysis results can be aligned with visual behavioral data within the same time range.

[0057] S52. Extract facial behavior state data corresponding to each speech time interval from the visual representation.

[0058] The visual representation includes visual behavioral information such as facial movements, gaze changes, facial posture changes, and facial response states of the target object during the interactive segment. This step extracts visual behavioral data within the same time range from the visual representation based on the speech time interval corresponding to each text segment, obtaining facial behavioral state data that corresponds one-to-one with the text segment. This facial behavioral state data reflects the target object's facial response process during the expression of the corresponding text segment, ensuring consistency between speech content analysis and facial behavior analysis in the temporal dimension.

[0059] S53. Semantically encode each text segment and determine the semantic expression load value corresponding to each text segment based on the text length and corresponding speech duration of each text segment.

[0060] Semantic encoding refers to converting text fragments into semantic vectors or structured data that can represent semantic content, expressive complexity, and information density. Text length reflects the number of words, characters, or effective semantic units in the text fragment, while speech duration reflects the time consumed by the target object to complete the expression of the text fragment. This step combines the semantic encoding result, text length, and speech duration to calculate the semantic expressive load value of the text fragment in the current interaction process. The semantic expressive load value characterizes the information complexity and expressive pressure borne by the target object when performing speech expression within that time interval.

[0061] S54. Determine the facial response intensity and facial cognitive load value based on the facial behavior state data within the corresponding speech time interval.

[0062] Among them, facial response intensity refers to the strength of facial movement changes of the target object within the corresponding speech time interval, which can be determined based on the displacement of facial key points, the amplitude of expression changes, the amplitude of gaze changes, and the amplitude of facial posture changes. Facial cognitive load value refers to the degree of cognitive input inferred from facial behavioral state data, which can be determined by combining facial muscle tension, eye movement frequency, gaze stability, and facial movement coordination. This step converts visual behavioral changes within the same speech time interval into quantifiable facial response intensity and facial cognitive load value, providing basic data for subsequent judgment on whether semantic expression state matches facial behavioral state.

[0063] S55. Perform matching analysis on the semantic expression load, facial response intensity, and facial cognitive load within the same speech time interval to obtain the consistency score corresponding to each text segment.

[0064] Matching analysis refers to analyzing the correspondence between semantic expressive load values ​​and facial response intensity and facial cognitive load values ​​within the same speech time interval. Generally, text segments with high semantic expressive load usually require a corresponding degree of facial response and cognitive input from the target audience, while text segments with low semantic expressive load should also have facial responses and cognitive loads within a relatively matching range. This step determines the consistency score corresponding to the text segment based on the degree of difference, trend of change, and temporal relationship among the three types of data. The consistency score can characterize the degree of matching between the speech content and facial behavioral state of the text segment.

[0065] S56. Generate consistency analysis results based on the consistency scores corresponding to multiple text segments.

[0066] The consistency analysis result is an overall judgment obtained by comprehensively processing the consistency scores of multiple text segments. This step can weight, smooth, or statistically process multiple consistency scores according to the chronological order of the text segments, speech duration, semantic load, or importance within the interactive segments, to obtain an analysis result that reflects the overall consistency status of the current interactive segments. The consistency analysis result can reflect both abnormal matching in a single text segment and the consistency trend among multiple text segments.

[0067] S57. Generate semantic representations based on consistency analysis results, and generate semantic anomaly confidence based on the degree of deviation between semantic representations and preset real interactive semantic features.

[0068] Semantic representation, in this context, is the semantic expression formed by encoding the consistency analysis results. It represents the matching pattern between speech content and facial behavioral states. Pre-defined real-interaction semantic features are reference features predetermined based on real-interaction states, reflecting the normal correspondence between semantic load, facial response, and cognitive load during natural expression. This step compares the semantic representation with the pre-defined real-interaction semantic features to determine the degree of deviation of the semantic representation relative to the real-interaction semantic features, and generates a semantic anomaly confidence score based on the degree of deviation. The semantic anomaly confidence score represents the possibility of inconsistency between the speech content and facial behavioral states in the current interaction segment.

[0069] As an example, in fintech scenarios, when a target is undergoing loan interviews, remote account opening, or online identity verification, the transcribed speech can be divided into multiple text segments based on its content. By combining the speech time intervals corresponding to each segment, the semantic load of the target when stating identity information, explaining the purpose of funds, confirming business intentions, or responding to risk inquiries can be analyzed. Simultaneously, facial response states and facial cognitive load states are extracted from the visual representation within the corresponding time intervals to determine whether the complexity of the speech content, the rhythm of expression, and facial reactions are consistent. In digital healthcare scenarios, when a target is undergoing remote consultations, online follow-ups, or health information verification, the semantic load of text segments such as symptom descriptions, supplementary medical information, explanations of past medical history, or medication feedback within the corresponding speech time intervals can be analyzed. By combining facial response intensity and facial cognitive load values, it can be determined whether the semantic content of the target expressing health information matches their facial behavior, thus forming a semantic analysis result that reflects the consistency between speech content and facial behavior.

[0070] Through steps S51 to S57, the transcribed speech text and visual representation can be aligned in the time dimension, thus associating the expressive load corresponding to the speech content with the facial response state and facial cognitive load state within the same time interval. This process can identify the matching relationship between speech content and facial behavioral state, enhance the ability to judge abnormal situations such as pre-prepared content, external auxiliary expressions, or unnatural statements, and provide a semantic consistency basis for subsequent multimodal fusion judgment.

[0071] S6. Input the visual representation, acoustic representation, semantic representation, visual anomaly confidence, acoustic anomaly confidence, semantic anomaly confidence, and environmental quality parameters into the meta-agent, and determine the visual fusion weight, acoustic fusion weight, and semantic fusion weight through the meta-agent.

[0072] The meta-agent is an intelligent processing unit that comprehensively schedules the analysis results of visual, acoustic, and semantic agents. It receives the representations and anomaly confidence levels output by each agent and combines these with environmental quality parameters to determine the credibility of different analysis results within the current interaction segment. For example, the credibility of visual analysis results may decrease when the screen illumination is low or facial occlusion is significant; the credibility of acoustic analysis results may decrease when the audio signal-to-noise ratio is low or background noise is strong; and the credibility of semantic consistency analysis results may be affected when a high network packet loss rate leads to unstable speech and video timing. In fintech scenarios, the meta-agent can dynamically adjust the fusion weights of visual, acoustic, and semantic analysis results based on factors such as backlighting, noise, or network fluctuations in the remote interview environment, reducing the probability of misjudging normal targets due to poor environmental quality. In digital healthcare scenarios, the meta-agent can dynamically adjust the adoption level of different modal analysis results based on image clarity, sound clarity, and connection stability during remote consultations, avoiding distortion of behavioral authenticity detection results due to damage to a single modality. Through the processing in step S6 above, the influence of different modal analysis results can be dynamically determined based on the quality of the interactive environment, thereby improving the robustness of the detection results in complex interactive environments.

[0073] As an alternative implementation method, please continue reading. Figure 3 , Figure 3 This is a schematic diagram of the execution flow of determining weights through a meta-agent in the audio and video behavior authenticity detection method provided in this application embodiment, as shown below. Figure 3 As shown, the process includes steps S61 to S66.

[0074] S61. Standardize the image illuminance, audio signal-to-noise ratio and network packet loss rate in the environmental quality parameters to obtain image quality value, audio quality value and interaction stability value.

[0075] Among them, screen illuminance reflects the brightness of the video image in the current interactive segment, audio signal-to-noise ratio reflects the clarity of the target speech relative to background noise in the current interactive segment, and network packet loss rate reflects the data integrity and temporal stability during audio and video transmission. Since the dimensions, value ranges, and directions of change of these environmental quality parameters are not the same, directly inputting them into the meta-agent can easily lead to an imbalance in the weighting of different quality parameters. Therefore, this step standardizes screen illuminance, audio signal-to-noise ratio, and network packet loss rate, converting each environmental quality parameter to a unified numerical range, and enabling the converted values ​​to respectively characterize the reliability of visual analysis, the reliability of acoustic analysis, and the cross-modal temporal stability. Specifically, the screen quality value can be determined based on the deviation between the screen illuminance and a preset reasonable illuminance range, the audio quality value can be determined based on the change in audio signal-to-noise ratio relative to a preset signal-to-noise ratio range, and the interaction stability value can be determined based on the change in network packet loss rate relative to a preset transmission quality range.

[0076] S62. Concatenate the image quality value with the visual representation to obtain the visual weight input vector; concatenate the audio quality value with the acoustic representation to obtain the acoustic weight input vector; and concatenate the interaction stability value with the semantic representation to obtain the semantic weight input vector.

[0077] In this process, visual representation, acoustic representation, and semantic representation reflect the analysis results of the current interaction segment in the visual, acoustic, and semantic consistency dimensions, respectively. Meanwhile, image quality value, audio quality value, and interaction stability value reflect the credibility of the corresponding analysis results in the current environment. This step concatenates each type of representation with its corresponding quality value, enabling the meta-agent to perceive not only the abnormal characteristics of the corresponding modality itself but also the data quality of that modality in the current interaction environment when determining whether the analysis results of each agent should be adopted. For example, the visual weight input vector contains both visual representation and image quality value, thus expressing information on both "what the current visual analysis result is" and "whether the current visual analysis result is reliable." The acoustic weight input vector and semantic weight input vector also employ the same processing logic, providing a comprehensive input containing both feature state and quality state for subsequent weight mapping.

[0078] S63. The visual weight input vector, acoustic weight input vector, and semantic weight input vector are mapped by the meta-agent to obtain the visual initial adoption value, acoustic initial adoption value, and semantic initial adoption value.

[0079] The mapping process refers to the meta-agent performing nonlinear transformations, attention calculations, or policy network processing on the input vectors to output the initial adoption level of the corresponding agent's analysis results in the current interaction segment. The visual initial adoption value, acoustic initial adoption value, and semantic initial adoption value are the initial judgment results determined by the meta-agent based on each input vector. This step enables the meta-agent to learn the correspondence between visual representations and image quality values, acoustic representations and audio quality values, and semantic representations and interaction stability values, respectively, and to generate initial adoption tendencies based on the performance of each dimension in the current interaction segment. By mapping the three types of weighted input vectors separately, the direct mixing of all modal information can avoid masking certain modal features, allowing the initial contribution levels of different agents to be modeled independently.

[0080] S64. Based on the image quality value, audio quality value, and interaction stability value, perform quality correction on the corresponding initial visual adoption value, initial acoustic adoption value, and initial semantic adoption value to obtain the visual quality correction value, acoustic quality correction value, and semantic quality correction value.

[0081] Quality correction refers to further adjusting the adoption level based on the corresponding environmental quality, building upon the initial adoption value. Since the initial visual, acoustic, and semantic adoption values ​​primarily originate from the joint mapping between representation and quality values, there may still be instances where an agent outputs a higher initial adoption value in a low-quality environment. Therefore, quality correction is needed to enhance the constraint effect of environmental quality on the adoption results. When the image quality value is low, the initial visual adoption value is suppressed or downweighted to obtain a visual quality correction value. When the audio quality value is low, the initial acoustic adoption value is suppressed or downweighted to obtain an acoustic quality correction value. When the interaction stability value is low, the initial semantic adoption value is suppressed or downweighted to obtain a semantic quality correction value. This processing ensures that the adoption values ​​corresponding to each agent are consistent with the current data quality, preventing low-quality modalities from having an excessively strong impact on the final detection results.

[0082] S65. Based on the differences between visual anomaly confidence, acoustic anomaly confidence, and semantic anomaly confidence, cross-correct the visual quality correction value, acoustic quality correction value, and semantic quality correction value to obtain the visual target adoption value, acoustic target adoption value, and semantic target adoption value.

[0083] Cross-correction refers to further comparing the consistency and conflict relationships between the anomaly confidence scores output by different agents based on the quality correction values, and adjusting the adoption level of each agent according to the comparison results. If the differences between visual, acoustic, and semantic anomaly confidence scores are small, it indicates that the judgment results of different dimensions have high consistency, and the quality correction values ​​can remain relatively stable. If a certain anomaly confidence score differs significantly from other anomaly confidence scores, it is necessary to combine the quality correction value corresponding to that agent to determine whether the difference has a credible basis. When the anomaly confidence score with a large difference comes from an agent with a lower quality correction value, the target adoption value corresponding to that agent can be reduced. When the anomaly confidence score with a large difference comes from an agent with a higher quality correction value, the target adoption value corresponding to that agent can be retained or appropriately increased. Through this cross-correction process, a mutual verification relationship can be formed between different agents, reducing the impact of single-modal anomaly fluctuations on the final fusion result.

[0084] S66. Normalize the visual target adoption value, acoustic target adoption value, and semantic target adoption value to obtain the visual fusion weight, acoustic fusion weight, and semantic fusion weight.

[0085] The visual, acoustic, and semantic target adoption values, after quality and cross-correction, reflect the degree of target adoption by the visual, acoustic, and semantic agents in the current interaction segment, respectively. Since the target adoption values ​​may have different numerical ranges, directly using them as fusion weights would affect the stability of subsequent weighted fusion. Therefore, it is necessary to normalize the three target adoption values ​​to ensure that the visual, acoustic, and semantic fusion weights are within a unified numerical range and satisfy the weight constraints of subsequent fusion calculations. Normalization can employ proportional normalization, exponential normalization, or other weight normalization methods to ensure that the three fusion weights can jointly represent the relative contributions of different agents in the current interaction segment. Through this processing, the subsequent fusion of visual anomaly confidence, acoustic anomaly confidence, and semantic anomaly confidence can be adaptively weighted based on the current environmental quality and multi-agent cross-validation results.

[0086] As an example, in fintech scenarios, when target individuals undergo loan interviews, remote account openings, or online identity verification, if backlighting, background noise, or network fluctuations exist in the interaction segments, the reliability of visual analysis results can be judged based on screen illumination, the reliability of acoustic analysis results based on audio signal-to-noise ratio, and the stability of the temporal correspondence between voice content and facial behavior states based on network packet loss rate. Furthermore, by combining the differences between visual anomaly confidence, acoustic anomaly confidence, and semantic anomaly confidence, the adoption rate of different agents can be dynamically adjusted. For instance, when screen brightness is insufficient but audio is clear and semantic consistency is high, the impact of visual analysis results can be reduced, while the impact of acoustic and semantic analysis results can be increased. Conversely, when audio noise is strong but the screen is clear and facial behavior states are stable, the impact of acoustic analysis results can be reduced, while visual and semantic analysis results are adopted more frequently. In digital healthcare scenarios, when target individuals conduct remote consultations, online follow-ups, or confirm health information, the fusion weights of visual, acoustic, and semantic analysis results can be dynamically determined based on image clarity, sound clarity, and connection stability. This allows the detection process to adapt to complex interactive conditions such as home environments, mobile networks, background noise, and image occlusion.

[0087] Through steps S61 to S66, the fusion weights of the visual, acoustic, and semantic analysis results are no longer fixed, but dynamically adjusted according to the data quality of the current interaction segment and the interrelationships of the output results of each agent. This processing can reduce the impact of impaired modalities on the final judgment result when the image quality is poor, the audio noise is strong, or the network transmission is unstable, and increase the adoption degree of credible modalities, thereby reducing misjudgments caused by abnormal fluctuations in a single modality. At the same time, by cross-correcting the differences between different anomaly confidence levels, the mutual verification capability between visual, acoustic, and semantic analysis results can be enhanced, improving the stability, robustness, and credibility of behavior authenticity detection results in complex audio-visual interaction environments.

[0088] S7. Based on the visual fusion weight, acoustic fusion weight, and semantic fusion weight, the visual anomaly confidence, acoustic anomaly confidence, and semantic anomaly confidence are fused to obtain the detection result of the authenticity of the target object's behavior during the audio-visual interaction process.

[0089] The fusion processing refers to the weighted calculation or comprehensive judgment of visual, acoustic, and semantic anomaly confidence levels according to visual fusion weights, acoustic fusion weights, and semantic fusion weights, forming a detection result that can characterize the authenticity of the target object's behavior. The behavior authenticity detection result can indicate whether the target object's behavior during audio-visual interaction is credible, or it can indicate the degree of risk associated with abnormal expression, abnormal vocalization, or abnormal auxiliary participation. In fintech scenarios, this detection result can reflect the authenticity and reliability of the target object's expression during loan interviews, remote account opening, or online identity verification, helping to identify risks such as lip-syncing, others answering on behalf of the target, or off-screen guidance. In digital healthcare scenarios, this detection result can reflect the authenticity and reliability of the target object's expression during remote consultations, online follow-ups, or health information verification, helping to identify risks such as non-personalized answers or external auxiliary statements. Through the processing in step S7 above, the anomaly judgment results from the three dimensions of vision, acoustics, and semantics can be integrated into a unified behavior authenticity detection result, improving the stability, accuracy, and interpretability of the detection conclusions.

[0090] As another alternative implementation method, please continue reading. Figure 4 , Figure 4 This is a schematic diagram of the execution flow of the meta-agent for joint updating in the audio and video behavior authenticity detection method provided in this application embodiment, as shown below. Figure 4 As shown, the process includes steps S81 to S87.

[0091] S81. Obtain sample audio and video streams, and construct visual training samples, acoustic training samples, semantic training samples, and authenticity labels based on the sample audio and video streams.

[0092] The sample audio / video stream refers to historical audio / video interaction data that has been pre-collected or organized. This stream can include both real and abnormal interaction samples. Real interaction samples correspond to audio / video data of the target user naturally performing speech and behavioral interactions. Abnormal interaction samples correspond to audio / video data with abnormal speech sources, off-screen cues, assistance from others, pre-recorded audio combined with facial movements, abnormal audio-visual timing, or inconsistencies between semantics and behavior. When constructing visual training samples based on the sample audio / video stream, visual information such as facial regions, 3D facial key points, gaze changes, and facial muscle coordination can be extracted from the sample video data. When constructing acoustic training samples based on the sample audio / video stream, acoustic information such as voiceprints, sound source location, background speech components, and noise levels can be extracted from the sample audio data. When constructing semantic training samples based on the sample audio / video stream, the transcribed text of the sample speech can be associated with visual features within the corresponding time interval to form matching samples between speech content and facial behavioral states. Authenticity labels are used to mark the authenticity of the behavior corresponding to the sample audio / video stream or sample interaction segment, providing supervision for subsequent training.

[0093] S82. Perform comparative learning training on the visual agent to be trained based on visual training samples, comparative learning training on the acoustic agent to be trained based on acoustic training samples, and supervised training on the semantic agent to be trained based on semantic training samples.

[0094] The training process involves visual, acoustic, and semantic agents, each trained on different modalities to initially equip them with relatively independent single-modal or cross-modal recognition capabilities. For the visual agent undergoing comparative learning, positive and negative sample pairs are constructed from real and anomalous interactive visual samples, allowing the agent to learn the differences between natural facial muscle coordination, normal gaze changes, and anomalous lip-syncing or gaze wandering. Similarly, for the acoustic agent, positive and negative sample pairs are constructed from real and anomalous acoustic samples, enabling the agent to learn the differences between normal vocalization of the target object, stable sound source location, and responses from others, off-screen cues, and background noise. For the semantic agent undergoing supervised training, the correspondence between speech content and facial behavioral states is learned based on the speech content, visual features, and matching labels in the semantic training samples, thereby developing the ability to judge whether the semantic expressive load and facial cognitive load match.

[0095] S83. Connect the trained visual agent, acoustic agent, and semantic agent to the meta-agent to be trained, and inject image quality interference, audio quality interference, and network transmission interference into the sample audio and video streams to obtain the interference sample audio and video streams.

[0096] After the visual, acoustic, and semantic agents complete their basic training, these three agents are connected to the trainable meta-agent, enabling it to receive sample representations and sample anomaly confidence levels output by the three agents. To make the training process closely resemble a real audio-visual interaction environment, this step injects different types of environmental interference into the sample audio-visual streams. Image quality interference can include reduced brightness, partial occlusion, image blurring, image jitter, or frame rate reduction. Audio quality interference can include background noise enhancement, target speech attenuation, mixing in other people's speech, sound distortion, or short-term silence. Network transmission interference can include audio-visual asynchrony, data packet loss, stuttering, or timestamp offset. The interfered sample audio-visual streams can simulate the complex acquisition and transmission environments that may occur during actual remote interaction in fintech and digital healthcare, allowing the trainable meta-agent to learn weight adjustment methods when different modalities suffer quality impairment during the training phase.

[0097] S84. Obtain the sample anomaly confidence level of each agent's output based on the interference sample audio and video stream, and generate sample fusion weights through the to-be-trained meta-agent based on the sample anomaly confidence level and sample environment quality parameters.

[0098] In this system, interfering audio and video streams are input into the trained visual agent, acoustic agent, and semantic agent, respectively. Each agent outputs a corresponding sample anomaly confidence score. The sample anomaly confidence score reflects each agent's judgment of the risk to the authenticity of the behavior under the current interference conditions. Sample environment quality parameters reflect the image quality, audio quality, and interaction stability in the interfering audio and video streams, such as image illumination, audio signal-to-noise ratio, and network packet loss rate. After receiving the sample anomaly confidence scores and sample environment quality parameters, the trainable meta-agent does not simply average the results of each agent, but rather learns the correspondence between different environment qualities and the credibility of different agents. For example, when image quality is compromised, the trainable meta-agent needs to reduce the acceptance of the visual agent's output. When audio quality is compromised, the trainable meta-agent needs to reduce the acceptance of the acoustic agent's output. When network transmission stability is compromised, the trainable meta-agent needs to cautiously accept semantic consistency results that rely on audio-video temporal alignment. The sample fusion weight thus characterizes the relative acceptance of each agent's output under the current interference conditions.

[0099] S85. The sample anomaly confidence scores are fused according to the sample fusion weights to obtain the sample behavior authenticity detection results.

[0100] In this process, the sample fusion weight reflects the proportion of the training meta-agent's adoption of the outputs of the visual, acoustic, and semantic agents, while the sample anomaly confidence reflects the anomaly judgments given by the three agents respectively. This step weights and fuses the anomaly confidence of each sample according to the sample fusion weight to obtain the sample behavior authenticity detection result. This detection result can indicate the degree of risk of the authenticity of the behavior corresponding to the interfering audio and video stream, or it can indicate the judgment result of whether there is abnormal interaction behavior in the current sample. Since the fusion processing considers both the anomaly judgments of each agent and the credibility under the current environmental quality, this step enables the detection results during the training process to not rely on a single modality judgment, but to reflect the collaborative judgment relationship between multiple agents.

[0101] S86. Generate a global collaborative loss based on the difference between the sample behavior authenticity detection results and the authenticity label, and generate a local penalty loss based on the overconfidence penalty corresponding to each agent.

[0102] The global collaboration loss constrains the overall output of multiple agents and the meta-agent to ensure that the sample behavior detection results are close to the realism labels. When the difference between the sample behavior detection results and the realism labels is large, the global collaboration loss increases, indicating that the current collaborative judgment results need to be adjusted. The local penalty loss constrains the confidence output behavior of individual agents when the quality of their corresponding modalities is poor, preventing an agent from outputting excessively high sample anomaly confidence levels under low-quality input conditions. The overconfidence penalty can be determined jointly based on the modal quality value and the sample anomaly confidence level of the agent. When the modal quality of an agent is lower than a preset quality threshold, and the sample anomaly confidence level output by the agent is higher than the preset confidence threshold, it indicates that the agent may be making an overly strong judgment under insufficient input quality conditions, thus generating a corresponding local penalty loss. The global collaboration loss constrains the overall judgment accuracy, while the local penalty loss constrains the reasonableness of the confidence levels of each agent under impaired modalities; both together improve the stability of the training process.

[0103] S87. Based on the global cooperation loss and the local penalty loss, jointly update the visual agent, acoustic agent, semantic agent and meta-agent.

[0104] Joint update refers to the reverse adjustment of trainable parameters in the visual agent, acoustic agent, semantic agent, and meta-agent based on global collaboration loss and local penalty loss. Global collaboration loss prompts each agent and the meta-agent to collaboratively optimize around the final behavior authenticity detection result, making the fused detection result closer to the authenticity label. Local penalty loss prompts each agent to reduce unreasonably high-confidence outputs when the quality of its corresponding modality is compromised, and encourages the meta-agent to learn to dynamically adjust the fusion weights under different environmental quality conditions. After joint update, the visual agent, acoustic agent, and semantic agent can better adapt to the collaborative judgment requirements under complex interference conditions while maintaining their respective modality analysis capabilities. The meta-agent can learn the complementary and cross-verification relationships between different modality results, thereby improving the anti-interference ability and generalization ability of subsequent audio and video behavior authenticity detection.

[0105] As an example, in fintech scenarios, historical audio and video interaction data from remote account opening, loan interviews, and online identity verification can be collected. Different interaction states—such as the target person's natural responses, responses on behalf of others, off-screen prompts, and pre-recorded audio with lip-syncing—can be used as training samples. This allows the visual agent to learn the difference between normal facial coordination and abnormal lip-syncing, the acoustic agent to learn the difference between the target person's actual voice and non-target sound sources or background prompts, and the semantic agent to learn the matching relationship between speech content and facial behavior states. In digital healthcare scenarios, historical audio and video interaction data from remote consultations, online follow-ups, and health information verification can be collected. Situations such as the person naturally describing their symptoms, responses on behalf of others, supplementary prompts from bystanders, and audio / video asynchrony caused by network lag can be used as training samples. This allows each agent to learn the ability to judge authenticity in visual, acoustic, and semantic dimensions. During training, interference such as dimming, occlusion, background noise, voices from others, audio / video stuttering, or data packet loss can be added to the sample audio / video streams. This allows the meta-agents to learn to adjust the adoption ratio of visual, acoustic, and semantic analysis results under different environmental quality conditions. Furthermore, global collaborative loss constrains the overall detection results to approach the true label, while local penalty loss constrains the excessively high confidence output under low-quality modalities. This enables each agent to form collaborative judgment capabilities in complex remote interaction environments.

[0106] Through steps S81 to S87, the visual agent, acoustic agent, semantic agent, and meta-agent can develop a well-defined and collaborative judgment capability during the training phase. This process first enhances the ability of each agent to identify visual anomalies, acoustic anomalies, and semantic behavioral inconsistencies. Then, through training with interfering samples, the meta-agent learns weight adjustment strategies for different environmental qualities, thereby improving its adaptability in complex audio-visual interaction environments. Simultaneously, the global collaborative loss constrains the overall detection results to maintain consistency with the authenticity label, while the local penalty loss suppresses unreasonably high-confidence outputs under low-quality modalities. Therefore, it reduces the risk of misjudgment caused by interference conditions such as backlighting, noise, stuttering, and packet loss, and improves the stability, robustness, and generalization ability of the behavioral authenticity detection results.

[0107] As another optional implementation, the process of generating local penalty loss based on the overconfidence penalty corresponding to each agent in the above steps may also specifically include the following steps S861 to S865.

[0108] S861. Based on the sample environment quality parameters, determine the modal quality values ​​corresponding to the visual agent, acoustic agent, and semantic agent, respectively.

[0109] Among them, the sample environment quality parameters reflect the data quality status of different modalities in the sample audio and video streams. The modal quality value for a visual agent can be determined based on image illumination, image sharpness, or occlusion level; the modal quality value for an acoustic agent can be determined based on audio signal-to-noise ratio, background noise intensity, or speech intelligibility; and the modal quality value for a semantic agent can be determined based on network packet loss rate, audio and video synchronization status, or timestamp stability. A lower modal quality value indicates poorer data quality received by the corresponding agent, and a higher uncertainty in the corresponding output result.

[0110] S862. When the modal quality value corresponding to the target agent is lower than the preset quality threshold, obtain the sample anomaly confidence level output by the target agent.

[0111] S863. Compare the sample anomaly confidence level output by the target agent with the preset confidence threshold.

[0112] S864. When the confidence level of the sample anomaly output by the target agent is higher than the preset confidence threshold, an overconfidence penalty value corresponding to the target agent is generated based on the difference between the modal quality value and the preset quality threshold, and the difference between the confidence level of the sample anomaly and the preset confidence threshold.

[0113] Specifically, when the modal quality value corresponding to the target agent is low and the sample anomaly confidence level is high, it indicates that the target agent may make overly strong judgments under conditions of insufficient input quality. The difference between the modal quality value and the preset quality threshold reflects the degree of modal quality impairment, while the difference between the sample anomaly confidence level and the preset confidence threshold reflects the degree of high-confidence output. Generating an overconfidence penalty value based on these two differences allows the penalty strength to be constrained by both modal quality and confidence output strength.

[0114] S865. Generate local penalty loss based on the overconfidence penalty value corresponding to each target agent.

[0115] Specifically, the overconfidence penalty values ​​for each target agent can be summed, weighted, or averaged to obtain a local penalty loss. This local penalty loss participates in subsequent joint updates, enabling each agent to reduce unreasonably high-confidence outputs when the corresponding modality quality is poor, thereby improving the stability and anti-interference ability during training.

[0116] As an example, in fintech scenarios, when training samples for loan interviews or remote account openings, if a particular interfering sample has issues such as excessively dark images, facial occlusion, or video stuttering, the modal quality value of the visual agent may be lower than a preset quality threshold. If the visual agent still outputs a high sample anomaly confidence level under these low-quality visual input conditions, a corresponding overconfidence penalty value can be generated based on the degree of visual modal quality impairment and the extent to which the anomaly confidence level exceeds the preset confidence threshold. Similarly, when samples contain strong background noise, intrusive speech, or audio-visual asynchrony, high-confidence outputs of acoustic or semantic agents can be constrained separately. In digital healthcare scenarios, when training samples for remote consultations or online follow-ups, if samples have issues such as home environment noise, network packet loss, blurry images, or abnormal acquisition angles, and an agent still outputs an excessively high anomaly judgment based on a damaged modality, a corresponding overconfidence penalty can be triggered, making the trained agent more cautious in outputting anomaly confidence levels under low-quality input conditions.

[0117] Through steps S861 to S865, the high-confidence outputs of each agent under low-quality modal conditions can be constrained during training, enabling the visual, acoustic, and semantic agents to adjust the confidence level of anomaly judgments based on the quality of input data. This process avoids an agent making overly strong judgments when the image is unclear, the audio noise is strong, or the audio-visual timing is unstable, thereby reducing the risk of misjudgment caused by impaired modalities. Simultaneously, by transforming the overconfidence penalty into a local penalty loss, the subsequent joint update process can balance overall detection accuracy and the reliability of individual agent outputs, improving the stability and anti-interference capability of behavior authenticity detection in complex audio-visual interaction environments.

[0118] The audio-visual behavior authenticity detection method provided in this application slices and parses the audio-visual stream during the audio-visual interaction process to obtain video frame sequences, audio frame sequences, speech-transcribed text, and environmental quality parameters. It then analyzes the data from dimensions such as facial behavior, vocalization state, sound source environment, and consistency between speech content and facial behavior state using visual, acoustic, and semantic agents, respectively. This avoids relying solely on single audio features, single video features, or simple lip-sync relationships for authenticity judgment. Furthermore, by combining the representations, anomaly confidence levels, and environmental quality parameters such as screen illumination, audio signal-to-noise ratio, and network packet loss rate output by each agent through a meta-agent, the visual fusion weight, acoustic fusion weight, and semantic fusion weight are dynamically determined. This allows the analysis results of different modalities to adaptively adjust their adoption level according to the quality state of the current interaction environment. When the screen quality is poor, the audio noise is strong, or the network transmission is unstable, the impact of the damaged modality on the final detection result can be reduced, and the contribution of the reliable modality can be increased, thereby reducing misjudgments in complex environments. Meanwhile, by analyzing the consistency between semantic content and facial behavioral states, it is possible to identify situations where speech and visible behavior appear consistent but are actually aided by external assistance, non-personal expression, or abnormal guidance, thereby improving the ability to identify hidden abnormal behaviors. Therefore, the embodiments of this application can improve the accuracy of behavioral authenticity detection in fintech scenarios such as remote account opening, loan interviews, and online identity verification. They can also improve the ability to judge the authenticity of expression and the reliability of identity participation in digital healthcare scenarios such as remote consultations, online follow-ups, and health information verification, exhibiting good anti-interference capabilities, stability, and interpretability.

[0119] Please continue reading. Figure 5 , Figure 5 This is a schematic diagram of the system structure of the audio and video behavior authenticity detection device provided in the embodiments of this application, as shown below. Figure 5 As shown, the audio and video behavior authenticity detection device 50 includes: an interaction segment acquisition module 51, an interaction segment parsing module 52, a visual feature analysis module 53, an acoustic feature analysis module 54, a semantic consistency analysis module 55, a meta-agent weighting module 56, and a result fusion output module 57.

[0120] The interaction segment acquisition module 51 is specifically used to acquire the audio and video stream of the target object during the audio and video interaction process, and to slice the audio and video stream to obtain multiple interaction segments; the interaction segment parsing module 52 is specifically used to perform audio and video parsing on the interaction segments to obtain video frame sequences, audio frame sequences, speech-transcribed text, and environmental quality parameters; the visual feature analysis module 53 is specifically used to input the video frame sequence to a visual agent, extract the facial behavior features of the target object through the visual agent, and generate visual representations and visual anomaly confidence scores based on the facial behavior features; the acoustic feature analysis module 54 is specifically used to input the audio frame sequence to an acoustic agent, extract acoustic behavior features related to the target object through the acoustic agent, and generate acoustic representations and acoustic anomaly confidence scores based on the acoustic behavior features; the semantic consistency analysis module 55 is specifically used to input the audio frame sequence to the visual agent. The speech-transcribed text and the visual representation are fed into a semantic agent. The semantic agent analyzes the consistency between the speech content corresponding to the speech-transcribed text and the facial behavior state corresponding to the visual representation, and generates a semantic representation and a semantic anomaly confidence score based on the consistency analysis results. The meta-agent weighting module 56 is specifically used to input the visual representation, the acoustic representation, the semantic representation, the visual anomaly confidence score, the acoustic anomaly confidence score, the semantic anomaly confidence score, and the environmental quality parameters into the meta-agent, and determines the visual fusion weight, acoustic fusion weight, and semantic fusion weight through the meta-agent. The result fusion output module 57 is specifically used to perform fusion processing on the visual anomaly confidence score, acoustic anomaly confidence score, and semantic anomaly confidence score based on the visual fusion weight, the acoustic fusion weight, and the semantic fusion weight to obtain the behavior authenticity detection result of the target object in the audio-visual interaction process.

[0121] As an optional implementation, the visual feature analysis module 53 is further specifically used to perform face region localization and three-dimensional facial key point extraction on the video frame sequence to obtain facial key point temporal data; determine facial muscle linkage features based on the facial key point temporal data, and determine gaze-wandering features based on changes in the eye region in the video frame sequence; determine the facial behavior features of the target object based on the facial muscle linkage features and the gaze-wandering features; generate the visual representation based on the facial behavior features, and generate the visual anomaly confidence level based on the degree of deviation between the visual representation and the preset real interactive visual features.

[0122] As an optional implementation, the acoustic feature analysis module 54 is specifically used to extract voiceprint features from the audio frame sequence to determine the main voiceprint feature; perform sound source localization on the audio frame sequence to determine the sound source orientation feature corresponding to the main voiceprint feature; perform blind source separation on the audio frame sequence to obtain the target speech component and background speech component, and determine the background speech interference feature based on the background speech component; determine the acoustic behavior feature based on the main voiceprint feature, the sound source orientation feature and the background speech interference feature; generate the acoustic representation based on the acoustic behavior feature, and generate the acoustic anomaly confidence level based on the degree of deviation between the acoustic representation and the preset real interactive acoustic features.

[0123] As an optional implementation, the semantic consistency analysis module 55 is specifically used to segment the speech-transcribed text into sentences to obtain multiple text segments and determine the speech time interval corresponding to each text segment; extract facial behavior state data corresponding to each speech time interval from the visual representation; perform semantic encoding on each text segment and determine the semantic expression load value corresponding to each text segment based on the text length and corresponding speech duration of each text segment; determine the facial response intensity and facial cognitive load value based on the facial behavior state data within the corresponding speech time interval; perform matching analysis on the semantic expression load value, facial response intensity, and facial cognitive load value within the same speech time interval to obtain a consistency score corresponding to each text segment; generate the consistency analysis result based on the consistency scores corresponding to multiple text segments; generate the semantic representation based on the consistency analysis result, and generate the semantic anomaly confidence level based on the degree of deviation between the semantic representation and the preset real interaction semantic features.

[0124] As an optional implementation, the meta-agent weighting module 56 is specifically used to standardize the image illuminance, audio signal-to-noise ratio, and network packet loss rate in the environmental quality parameters to obtain image quality values, audio quality values, and interaction stability values; concatenate the image quality values ​​with the visual representation to obtain a visual weight input vector; concatenate the audio quality values ​​with the acoustic representation to obtain an acoustic weight input vector; and concatenate the interaction stability values ​​with the semantic representation to obtain a semantic weight input vector; the meta-agent performs mapping processing on the visual weight input vector, the acoustic weight input vector, and the semantic weight input vector respectively to obtain initial visual acceptance values, initial acoustic acceptance values, and initial semantic acceptance values; according to the... The image quality value, the audio quality value, and the interaction stability value are used to correct the corresponding initial visual adoption value, initial acoustic adoption value, and initial semantic adoption value, respectively, to obtain visual quality correction values, acoustic quality correction values, and semantic quality correction values. Based on the differences between the visual anomaly confidence level, the acoustic anomaly confidence level, and the semantic anomaly confidence level, the visual quality correction values, acoustic quality correction values, and semantic quality correction values ​​are cross-corrected to obtain visual target adoption values, acoustic target adoption values, and semantic target adoption values. The visual target adoption values, acoustic target adoption values, and semantic target adoption values ​​are then normalized to obtain the visual fusion weight, the acoustic fusion weight, and the semantic fusion weight.

[0125] As an optional implementation, the audio-visual behavior authenticity detection device 50 further includes a multi-agent training module. This module is specifically used to acquire sample audio-visual streams and construct visual training samples, acoustic training samples, semantic training samples, and authenticity labels based on the sample audio-visual streams. It performs comparative learning training on the visual agent to be trained based on the visual training samples, comparative learning training on the acoustic agent to be trained based on the acoustic training samples, and supervised training on the semantic agent to be trained based on the semantic training samples. The trained visual agent, acoustic agent, and semantic agent are then connected to the meta-agent to be trained, and image quality interference, audio quality interference, and network transmission interference are injected into the sample audio-visual streams. The system outputs interference to obtain interference sample audio and video streams. Based on the interference sample audio and video streams, it obtains the sample anomaly confidence scores of each agent's output, and generates sample fusion weights through the meta-agent to be trained based on the sample anomaly confidence scores and sample environment quality parameters. The sample anomaly confidence scores are fused according to the sample fusion weights to obtain the sample behavior authenticity detection results. A global collaborative loss is generated based on the difference between the sample behavior authenticity detection results and the authenticity label, and a local penalty loss is generated based on the overconfidence penalty corresponding to each agent. The visual agent, acoustic agent, semantic agent, and meta-agent are jointly updated according to the global collaborative loss and the local penalty loss.

[0126] As an optional implementation, the multi-agent training module is further configured to: determine the modal quality values ​​corresponding to the visual agent, the acoustic agent, and the semantic agent respectively based on the sample environment quality parameters; when the modal quality value corresponding to the target agent is lower than a preset quality threshold, obtain the sample anomaly confidence level output by the target agent; compare the sample anomaly confidence level output by the target agent with a preset confidence threshold; when the sample anomaly confidence level output by the target agent is higher than the preset confidence threshold, generate an overconfidence penalty value corresponding to the target agent based on the difference between the modal quality value and the preset quality threshold, and the difference between the sample anomaly confidence level and the preset confidence threshold; and generate the local penalty loss based on the overconfidence penalty values ​​corresponding to each target agent.

[0127] It should be noted that the above-mentioned audio and video behavior authenticity detection device can execute the audio and video behavior authenticity detection method provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in the embodiments of the audio and video behavior authenticity detection device can be found in the audio and video behavior authenticity detection method provided in the embodiments of this application.

[0128] Figure 6This is a schematic diagram of the hardware structure of the electronic device for performing the audio and video behavior authenticity detection method provided in the embodiments of this application, as shown below. Figure 6 As shown, the electronic device 600 includes: One or more processors 610 and memory 620, Figure 6 Take the 610 processor as an example.

[0129] The processor 610 and the memory 620 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0130] The memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the audio-visual behavior authenticity detection method in the embodiments of this application. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 620, thereby implementing the audio-visual behavior authenticity detection method in the above-described method embodiments.

[0131] The memory 620 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the audio / video behavior authenticity detection device. Furthermore, the memory 620 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include memory remotely located relative to the processor 610, and these remote memories can be connected to the audio / video behavior authenticity detection device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0132] The one or more modules are stored in the memory 620. When executed by the one or more processors 610, they perform the audio and video behavior authenticity detection method in any of the above method embodiments, for example, performing the above-described... Figure 2 Method steps S1 to S7, Figure 3 Method steps S61 to S66, Figure 4 Method steps S81 to S87 are implemented. Figure 5 The functions of modules 51-57 in the document.

[0133] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0134] This application provides a non-volatile computer-readable storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 6 One of the processors 610 can enable the one or more processors to execute the audio / video behavior authenticity detection method in any of the above method embodiments, for example, to execute the above-described... Figure 2 Method steps S1 to S7, Figure 3 Method steps S61 to S66, Figure 4 Method steps S81 to S87 are implemented. Figure 5 The functions of modules 51-57 in the document.

[0135] This application provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions, which, when executed by the electronic device, enable the electronic device to perform the audio-visual behavior authenticity detection method in any of the above method embodiments, for example, to perform the above-described method. Figure 2 Method steps S1 to S7, Figure 3 Method steps S61 to S66, Figure 4 Method steps S81 to S87 are implemented. Figure 5 The functions of modules 51-57 in the document.

[0136] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0137] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0138] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for detecting the authenticity of audio and video behavior, characterized in that, include: The audio and video streams of the target object during the audio and video interaction process are obtained, and the audio and video streams are sliced ​​to obtain multiple interaction segments; The interactive segments are analyzed to obtain video frame sequences, audio frame sequences, speech-transcribed text, and environmental quality parameters. The video frame sequence is input into a visual agent, which extracts facial behavior features of the target object and generates a visual representation and a visual anomaly confidence score based on the facial behavior features. The audio frame sequence is input into the acoustic agent, which extracts acoustic behavior features related to the target object and generates acoustic representations and acoustic anomaly confidence scores based on the acoustic behavior features. The speech-transcribed text and the visual representation are input into a semantic agent. The semantic agent analyzes the consistency between the speech content corresponding to the speech-transcribed text and the facial behavior state corresponding to the visual representation, and generates a semantic representation and a semantic anomaly confidence score based on the consistency analysis results. The visual representation, acoustic representation, semantic representation, visual anomaly confidence, acoustic anomaly confidence, semantic anomaly confidence, and environmental quality parameters are input into the meta-agent, and the visual fusion weight, acoustic fusion weight, and semantic fusion weight are determined by the meta-agent. Based on the visual fusion weight, the acoustic fusion weight, and the semantic fusion weight, the visual anomaly confidence score, the acoustic anomaly confidence score, and the semantic anomaly confidence score are fused to obtain the behavior authenticity detection result of the target object in the audio-visual interaction process.

2. The method for detecting the authenticity of audio and video behavior according to claim 1, characterized in that, The process of inputting the video frame sequence to a visual agent, extracting facial behavior features of the target object through the visual agent, and generating visual representations and visual anomaly confidence scores based on the facial behavior features includes: The video frame sequence is subjected to face region localization and three-dimensional facial key point extraction to obtain facial key point temporal data; Facial muscle linkage features are determined based on the temporal data of the facial key points, and gaze wandering features are determined based on the changes in the eye region in the video frame sequence. The facial behavioral characteristics of the target object are determined based on the facial muscle coordination characteristics and the gaze-free characteristics. The visual representation is generated based on the facial behavioral features, and the visual anomaly confidence level is generated based on the degree of deviation between the visual representation and the preset real interactive visual features.

3. The method for detecting the authenticity of audio and video behavior according to claim 1, characterized in that, The process of inputting the audio frame sequence to the acoustic agent, extracting acoustic behavioral features related to the target object through the acoustic agent, and generating acoustic representations and acoustic anomaly confidence scores based on the acoustic behavioral features includes: Voiceprint features are extracted from the audio frame sequence to determine the main voiceprint features; The audio frame sequence is subjected to sound source localization to determine the sound source directional features corresponding to the main voiceprint features; Blind source separation is performed on the audio frame sequence to obtain the target speech component and the background speech component, and background speech interference features are determined based on the background speech component; The acoustic behavior features are determined based on the main voiceprint features, the sound source location features, and the background speech interference features; The acoustic representation is generated based on the acoustic behavior features, and the acoustic anomaly confidence level is generated based on the degree of deviation between the acoustic representation and the preset real interactive acoustic features.

4. The method for detecting the authenticity of audio and video behavior according to claim 1, characterized in that, The process involves inputting the transcribed speech text and the visual representation into a semantic agent, analyzing the consistency between the speech content corresponding to the transcribed speech text and the facial behavior state corresponding to the visual representation, and generating semantic representations and semantic anomaly confidence scores based on the consistency analysis results. The transcribed speech text is segmented into sentences to obtain multiple text segments, and the speech time interval corresponding to each text segment is determined. Extract facial behavior state data corresponding to each of the speech time intervals from the visual representation; Semantic encoding is performed on each text segment, and the semantic expression load value corresponding to each text segment is determined based on the text length and corresponding speech duration of each text segment; Based on the facial behavior state data within the corresponding speech time interval, determine the facial response intensity and facial cognitive load value; Matching analysis is performed on the semantic expression load, facial response intensity, and facial cognitive load within the same speech time interval to obtain a consistency score corresponding to each text segment; The consistency analysis results are generated based on the consistency scores corresponding to the multiple text fragments. The semantic representation is generated based on the consistency analysis results, and the semantic anomaly confidence level is generated based on the degree of deviation between the semantic representation and the preset real interaction semantic features.

5. The method for detecting the authenticity of audio and video behavior according to claim 1, characterized in that, The step of inputting the visual representation, acoustic representation, semantic representation, visual anomaly confidence level, acoustic anomaly confidence level, semantic anomaly confidence level, and environmental quality parameters into a meta-agent, and determining the visual fusion weight, acoustic fusion weight, and semantic fusion weight through the meta-agent, includes: The image illuminance, audio signal-to-noise ratio and network packet loss rate in the environmental quality parameters are standardized to obtain image quality value, audio quality value and interaction stability value; The image quality value is concatenated with the visual representation to obtain a visual weight input vector; the audio quality value is concatenated with the acoustic representation to obtain an acoustic weight input vector; and the interaction stability value is concatenated with the semantic representation to obtain a semantic weight input vector. The meta-agent maps the visual weight input vector, the acoustic weight input vector, and the semantic weight input vector to obtain the visual initial acceptance value, the acoustic initial acceptance value, and the semantic initial acceptance value. Based on the image quality value, the audio quality value, and the interaction stability value, the corresponding visual initial adoption value, acoustic initial adoption value, and semantic initial adoption value are respectively corrected to obtain visual quality correction value, acoustic quality correction value, and semantic quality correction value. Based on the differences between the visual anomaly confidence, the acoustic anomaly confidence, and the semantic anomaly confidence, the visual quality correction value, the acoustic quality correction value, and the semantic quality correction value are cross-corrected to obtain the visual target adoption value, the acoustic target adoption value, and the semantic target adoption value. The visual target adoption value, the acoustic target adoption value, and the semantic target adoption value are normalized to obtain the visual fusion weight, the acoustic fusion weight, and the semantic fusion weight.

6. The method for detecting the authenticity of audio and video behavior according to claim 1, characterized in that, Before inputting the video frame sequence to the visual agent, the audio frame sequence to the acoustic agent, the speech-transcribed text, and the visual representation to the semantic agent, the method further includes: Acquire sample audio and video streams, and construct visual training samples, acoustic training samples, semantic training samples, and realism labels based on the sample audio and video streams; The visual agent to be trained is subjected to comparative learning training based on the visual training samples, the acoustic agent to be trained is subjected to comparative learning training based on the acoustic training samples, and the semantic agent to be trained is subjected to supervised training based on the semantic training samples. The trained visual agent, acoustic agent, and semantic agent are connected to the meta-agent to be trained, and the sample audio and video streams are injected with picture quality interference, audio quality interference, and network transmission interference to obtain the interference sample audio and video streams. Based on the interference sample audio and video stream, the sample anomaly confidence level output by each agent is obtained, and the sample fusion weight is generated by the meta-agent to be trained according to the sample anomaly confidence level and sample environment quality parameters. The sample anomaly confidence scores are fused according to the sample fusion weights to obtain the sample behavior authenticity detection results. A global collaborative loss is generated based on the difference between the sample behavior authenticity detection result and the authenticity label, and a local penalty loss is generated based on the overconfidence penalty corresponding to each agent. Based on the global cooperation loss and the local penalty loss, the visual agent, the acoustic agent, the semantic agent, and the meta-agent are jointly updated.

7. The method for detecting the authenticity of audio and video behavior according to claim 6, characterized in that, The generation of local penalty loss based on the overconfidence penalty corresponding to each agent includes: Based on the sample environment quality parameters, determine the modal quality values ​​corresponding to the visual agent, the acoustic agent, and the semantic agent, respectively; When the modal quality value corresponding to the target agent is lower than the preset quality threshold, the sample anomaly confidence level output by the target agent is obtained. The sample anomaly confidence level output by the target agent is compared with a preset confidence threshold. When the confidence level of the sample anomaly output by the target agent is higher than the preset confidence threshold, an overconfidence penalty value corresponding to the target agent is generated based on the difference between the modality quality value and the preset quality threshold, and the difference between the confidence level of the sample anomaly and the preset confidence threshold. The local penalty loss is generated based on the overconfidence penalty value corresponding to each target agent.

8. A device for detecting the authenticity of audio and video behavior, characterized in that, include: The interactive segment acquisition module is used to acquire the audio and video stream of the target object during the audio and video interaction process, and to slice the audio and video stream to obtain multiple interactive segments. The interactive segment parsing module is used to perform audio and video parsing on the interactive segment to obtain video frame sequence, audio frame sequence, speech-transcribed text, and environmental quality parameters; The visual feature analysis module is used to input the video frame sequence into the visual agent, extract the facial behavior features of the target object through the visual agent, and generate a visual representation and a visual anomaly confidence level based on the facial behavior features. The acoustic feature analysis module is used to input the audio frame sequence into the acoustic agent, extract acoustic behavior features related to the target object through the acoustic agent, and generate acoustic representation and acoustic anomaly confidence based on the acoustic behavior features; The semantic consistency analysis module is used to input the speech-transcribed text and the visual representation into the semantic agent, analyze the consistency between the speech content corresponding to the speech-transcribed text and the facial behavior state corresponding to the visual representation through the semantic agent, and generate semantic representation and semantic anomaly confidence based on the consistency analysis results. The meta-agent weighting module is used to input the visual representation, the acoustic representation, the semantic representation, the visual anomaly confidence, the acoustic anomaly confidence, the semantic anomaly confidence, and the environmental quality parameters into the meta-agent, and to determine the visual fusion weight, acoustic fusion weight, and semantic fusion weight through the meta-agent; The result fusion output module is used to perform fusion processing on the visual anomaly confidence, the acoustic anomaly confidence, and the semantic anomaly confidence based on the visual fusion weight, the acoustic fusion weight, and the semantic fusion weight, to obtain the behavior authenticity detection result of the target object in the audio-visual interaction process.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the audio-visual behavior authenticity detection method according to any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by an electronic device, cause the electronic device to perform the audio-visual behavior authenticity detection method according to any one of claims 1-7.