Digital human video identification method and device, computer equipment and storage medium

Abstract

The invention belongs to the field of artificial intelligence and the field of financial science and technology, and relates to a digital human video identification method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a digital human video; extracting a target video frame and a target audio signal from the digital human video; performing feature extraction on the target video frame based on a visual encoder to obtain visual features, and performing feature extraction on the target audio signal based on an audio encoder to obtain audio features; performing conversion processing on the visual features and the audio features based on the projection layer to obtain visual modal embedding and audio modal embedding; performing fusion processing on the visual modal embedding and the audio modal embedding to obtain a target feature vector; and performing prediction processing on the target feature vector based on a multi-modal large model, and generating a counterfeit identification result corresponding to the digital human video. In addition, the counterfeited identification result may be stored in the blockchain. According to the invention, based on the use of the multi-modal large model, the accuracy and reliability of digital human authentic identification are effectively improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence development technology and financial technology, and in particular to methods, devices, computer equipment and storage media for recognizing digital human videos. Background Art

[0002] With the rapid advancement of artificial intelligence technology, especially the breakthrough development of deep learning and generative adversarial networks (GANs), virtual digital human technology has transformed from a science fiction concept to an important component of real-world applications. These highly realistic virtual images have not only greatly enriched the content creation of the entertainment industry, but have also penetrated deeply into advertising marketing, distance education, medical simulation, personalized virtual assistants and other fields, bringing users an unprecedented immersive experience. However, the high-simulation virtual digital human technology has greatly lowered the threshold for digital content fraud, making forged images, videos and even audio almost indistinguishable, thus posing a severe challenge to information authenticity, copyright protection, social trust and even legal order. Therefore, how to effectively identify the authenticity of virtual digital people has become an important issue that needs to be solved urgently.

[0003] Traditional digital human authentication methods mainly rely on single-modal image analysis technology, including but not limited to: 1. Texture analysis: This method identifies forgery traces by analyzing the texture features of the image surface. Although algorithms such as local binary pattern (LBP) perform well in extracting image texture information, their authentication capabilities are often limited when faced with highly complex forgery techniques, and it is difficult to capture subtle forgery features. 2. Frequency domain analysis: By converting the image from the spatial domain to the frequency domain, the spectral characteristics of the image are analyzed using tools such as Fourier transform to distinguish between real and forged images. However, with the continuous advancement of forgery technology, the characteristics of forged images in the frequency domain are becoming increasingly difficult to distinguish from real images, which greatly reduces the authentication effect of this method. 3. Edge detection: Sobel, Canny and other algorithms are used to detect the continuity and consistency of the image edge, and this is used as a basis for judging whether the image has been tampered with. However, this method is often powerless against forged images that have been finely processed, because forgers can easily adjust the image edges to evade detection.

[0004] In summary, existing digital human authentication methods generally have the problem of low accuracy when performing virtual digital human identification. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose a method, device, computer equipment and storage medium for recognizing digital human videos, so as to solve the technical problem that the existing digital human authentication methods generally have low accuracy when performing virtual digital human recognition.

[0006] In order to solve the above technical problems, the present application embodiment provides a method for recognizing digital human videos, which adopts the following technical solutions:

[0007] Get the input digital human video;

[0008] Extracting a target video frame from the digital human video based on a preset extraction strategy;

[0009] Extracting a target audio signal from the digital human video;

[0010] Extracting features of the target video frame based on a preset visual encoder to obtain corresponding visual features, and extracting features of the target audio signal based on a preset audio encoder to obtain corresponding audio features;

[0011] The visual features and the audio features are transformed based on a preset projection layer to obtain corresponding visual modality embedding and audio modality embedding;

[0012] Fusing the visual modality embedding with the audio modality embedding to obtain a corresponding target feature vector;

[0013] The target feature vector is predicted and processed based on a pre-trained multimodal large model to generate a forged recognition result corresponding to the digital human video.

[0014] Furthermore, the step of fusing the visual modality embedding and the audio modality embedding to obtain a corresponding target feature vector specifically includes:

[0015] Converting the visual modality embedding and the audio modality embedding into the same feature space to obtain a corresponding target visual modality embedding and a target audio modality embedding;

[0016] Get the preset fusion strategy;

[0017] Based on the fusion strategy, the target visual modality embedding and the target audio modality embedding are fused to obtain a corresponding fusion feature vector;

[0018] The fused feature vector is used as the target feature vector.

[0019] Furthermore, the step of converting the visual modality embedding and the audio modality embedding into the same feature space to obtain corresponding target visual modality embedding and target audio modality embedding specifically includes:

[0020] Performing feature dimension matching processing on the visual modality embedding and the audio modality embedding to obtain a corresponding first visual modality embedding and first audio modality embedding;

[0021] Performing feature normalization processing on the first visual modality embedding and the first audio modality embedding to obtain a corresponding second visual modality embedding and a second audio modality embedding;

[0022] Obtain a preset nonlinear transformation function;

[0023] The nonlinear transformation function performs feature mapping processing on the second visual modality embedding and the second audio modality embedding in the feature space to obtain the corresponding target visual modality embedding and the target audio modality embedding.

[0024] Furthermore, the step of extracting the target video frame from the digital human video based on a preset extraction strategy specifically includes:

[0025] Extracting continuous video frames from the digital human video;

[0026] Performing a preprocessing operation on the video frame to obtain a corresponding first video frame;

[0027] Performing block segmentation processing on the first video frame to obtain a second video frame;

[0028] The second video frame is used as the target video frame.

[0029] Furthermore, the step of extracting the target audio signal from the digital human video specifically includes:

[0030] Extracting audio from the digital human video to obtain a corresponding audio signal;

[0031] Performing noise reduction processing on the audio signal to obtain a processed first audio signal;

[0032] Preprocessing the first audio signal based on a preset processing strategy to obtain a corresponding second audio signal;

[0033] The second audio signal is used as the target audio signal.

[0034] Furthermore, before the step of predicting the target feature vector based on the pre-trained multimodal large model to generate a forgery recognition result corresponding to the digital human video, the method further includes:

[0035] Obtain pre-constructed and annotated digital human video sample data;

[0036] Dividing the digital human video sample data into a training data set and a test data set;

[0037] Call the preset initial multimodal large model;

[0038] Determine the target loss function and target optimization algorithm;

[0039] Based on the target loss function and the target optimization algorithm, the initial multimodal large model is trained using the training data set to obtain a corresponding designated model;

[0040] Performing a performance evaluation on the specified multimodal large model based on the test data set;

[0041] If the specified multimodal large model meets the preset performance expectations, the specified model is used as the multimodal large model.

[0042] Furthermore, after the step of using the designated model as the multimodal large model, the method further includes:

[0043] Acquire a model storage strategy corresponding to the multimodal large model;

[0044] Calling a corresponding target storage medium based on the model storage strategy;

[0045] determining, in the target storage medium, a designated storage location corresponding to the multimodal large model;

[0046] The multimodal large model is stored in the designated storage location in the target storage medium.

[0047] In order to solve the above technical problems, the embodiment of the present application further provides a device for recognizing digital human videos, which adopts the following technical solution:

[0048] A first acquisition module, used to acquire an input digital human video;

[0049] A first extraction module, used for extracting a target video frame from the digital human video based on a preset extraction strategy;

[0050] A second extraction module, used to extract a target audio signal from the digital human video;

[0051] A third extraction module is used to extract features of the target video frame based on a preset visual encoder to obtain corresponding visual features, and to extract features of the target audio signal based on a preset audio encoder to obtain corresponding audio features;

[0052] A conversion module, used for converting the visual features and the audio features based on a preset projection layer to obtain corresponding visual modality embedding and audio modality embedding;

[0053] A fusion module, used to fuse the visual modality embedding and the audio modality embedding to obtain a corresponding target feature vector;

[0054] The prediction module is used to perform prediction processing on the target feature vector based on a pre-trained multimodal large model to generate a forgery recognition result corresponding to the digital human video.

[0055] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0056] Get the input digital human video;

[0057] Extracting a target video frame from the digital human video based on a preset extraction strategy;

[0058] Extracting a target audio signal from the digital human video;

[0059] Extracting features of the target video frame based on a preset visual encoder to obtain corresponding visual features, and extracting features of the target audio signal based on a preset audio encoder to obtain corresponding audio features;

[0060] The visual features and the audio features are transformed based on a preset projection layer to obtain corresponding visual modality embedding and audio modality embedding;

[0061] Fusing the visual modality embedding with the audio modality embedding to obtain a corresponding target feature vector;

[0062] The target feature vector is predicted and processed based on a pre-trained multimodal large model to generate a forged recognition result corresponding to the digital human video.

[0063] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0064] Get the input digital human video;

[0065] Extracting a target video frame from the digital human video based on a preset extraction strategy;

[0066] Extracting a target audio signal from the digital human video;

[0067] Extracting features of the target video frame based on a preset visual encoder to obtain corresponding visual features, and extracting features of the target audio signal based on a preset audio encoder to obtain corresponding audio features;

[0068] The visual features and the audio features are transformed based on a preset projection layer to obtain corresponding visual modality embedding and audio modality embedding;

[0069] Fusing the visual modality embedding with the audio modality embedding to obtain a corresponding target feature vector;

[0070] The target feature vector is predicted and processed based on a pre-trained multimodal large model to generate a forged recognition result corresponding to the digital human video.

[0071] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0072] The present application first obtains an input digital human video; then extracts a target video frame from the digital human video based on a preset extraction strategy; and extracts a target audio signal from the digital human video; then performs feature extraction on the target video frame based on a preset visual encoder to obtain corresponding visual features, and performs feature extraction on the target audio signal based on a preset audio encoder to obtain corresponding audio features; subsequently, the visual features and the audio features are transformed based on a preset projection layer to obtain corresponding visual modal embedding and audio modal embedding; the visual modal embedding and the audio modal embedding are further fused to obtain a corresponding target feature vector; finally, the target feature vector is predicted based on a pre-trained multimodal large model to generate a forgery recognition result corresponding to the digital human video. The present application extracts target video frames and target audio signals from the input digital human video, and then extracts features from the target video frames and target audio signals based on the use of a visual encoder and an audio encoder to obtain corresponding visual features and audio features. The visual features and the audio features are then transformed based on a projection layer to obtain visual modal embedding and audio modal embedding. The visual modal embedding and the audio modal embedding are subsequently fused to obtain a target feature vector. Finally, the target feature vector is predicted based on a pre-trained multimodal large model to generate a forgery identification result corresponding to the digital human video. In this way, the present application achieves accurate authentication of digital human data in digital human videos by comprehensively analyzing the alignment of audio features and video features in digital human videos and performing in-depth analysis and processing based on the use of a multimodal large model, and provides efficient and accurate authentication results, thereby effectively improving the accuracy and reliability of digital human authentication. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0074] Figure 1is an exemplary system architecture diagram to which the present application may be applied;

[0075] Figure 2 A flowchart of an embodiment of a method for recognizing a digital human video according to the present application;

[0076] Figure 3 is a structural schematic diagram of an embodiment of a digital human video recognition device according to the present application;

[0077] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0078] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0079] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0080] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0081] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0082] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0083] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer, a desktop computer, etc.

[0084] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0085] It should be noted that the digital human video recognition method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the digital human video recognition device is generally set in the server / terminal device.

[0086] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0087] Continue to refer Figure 2 , shows a flow chart of an embodiment of the method for identifying digital human videos according to the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The method for identifying digital human videos provided in the embodiment of the present application can be applied to any scenario that requires digital human video identification, and the method for identifying digital human videos can be applied to products in these scenarios, for example, digital human authentication in the financial insurance field. The method for identifying digital human videos includes the following steps:

[0088] Step S201, obtaining an input digital human video.

[0089] In this embodiment, the method for recognizing a digital human video is executed on an electronic device (eg, Figure 1The server / terminal device shown in the figure) can obtain the input digital human video through a wired connection or a wireless connection. It should be noted that the above-mentioned wireless connection method may include but is not limited to 3G / 4G / 5G connection, Wi-Fi connection, Bluetooth connection, Wi MAX connection, Zigbee connection, UWB (Ultra Wi DeBand) connection, and other wireless connection methods currently known or to be developed in the future. The execution subject of this application may be a digital human authentication system. In the business scenario of financial insurance product push, the above-mentioned digital human video may be a video containing virtual digital human data used in the advertising and marketing business of financial insurance.

[0090] Step S202: extracting a target video frame from the digital human video based on a preset extraction strategy.

[0091] In this embodiment, the specific implementation process of extracting the target video frame from the digital human video based on the preset extraction strategy will be further described in detail in subsequent specific embodiments of the present application, and will not be elaborated on here.

[0092] Step S203: extracting a target audio signal from the digital human video.

[0093] In this embodiment, the specific implementation process of extracting the target audio signal from the digital human video will be further described in detail in subsequent specific embodiments of the present application, and will not be elaborated on here.

[0094] Step S204: extracting features from the target video frame based on a preset visual encoder to obtain corresponding visual features, and extracting features from the target audio signal based on a preset audio encoder to obtain corresponding audio features.

[0095] In this embodiment, the visual encoder may adopt a neural network structure suitable for processing audio signals (such as RNN, CNN or Transformer) to extract features such as spectrum and fundamental frequency from the audio signal. The audio encoder may adopt a neural network structure suitable for processing image data (such as CNN) to extract visual features such as lip movement from video frames. Specifically, the target video frame may be feature extracted by using a visual encoder, and key information in the image (such as facial features, lip shape changes, etc.) may be encoded as visual modal embedding. The target audio signal may be feature extracted by using an audio encoder, specifically using audio processing technology (such as short-time Fourier transform STFT, Mel-frequency cepstral coefficient MFCC, etc.) to extract features of the audio signal, including pitch, rhythm, timbre, etc., and encode them as audio modal embedding.

[0096] Step S205: transforming the visual features and the audio features based on a preset projection layer to obtain corresponding visual modality embedding and audio modality embedding.

[0097] In this embodiment, a projection layer is pre-designed, and the visual features and audio features are converted into fixed-size visual modality embedding and audio modality embedding respectively by using the projection layer. The function of the projection layer is to standardize the feature dimension to facilitate the fusion and comparison of multimodal features.

[0098] Step S206, fusing the visual modality embedding and the audio modality embedding to obtain a corresponding target feature vector.

[0099] In this embodiment, the above-mentioned specific implementation process of fusing the visual modality embedding and the audio modality embedding to obtain the corresponding target feature vector will be further described in detail in subsequent specific embodiments of the present application and will not be elaborated in detail here.

[0100] Step S207, predicting the target feature vector based on the pre-trained multimodal large model to generate a forgery recognition result corresponding to the digital human video.

[0101] In this embodiment, the target feature vector obtained by fusing the visual modality embedding and the audio modality embedding is input into a pre-trained multimodal large model, and the large modality model automatically extracts and learns the alignment pattern between audio and video through its complex neural network structure. By analyzing the synchronization of the audio signal and the lip movement, it is detected whether there are abnormal phenomena such as time deviation or feature inconsistency between the two, thereby judging the authenticity of the audio and video. The model comprehensively analyzes the alignment of audio and video features, and further analyzes and outputs the prediction result, that is, the forgery identification result corresponding to the digital human video, which is used to judge whether the digital human data in the input digital human video is forged, and effectively provides efficient and accurate digital human forgery identification results.

[0102] Specifically, the process of using the multimodal large model to authenticate digital human data includes: Alignment pattern learning: The multimodal large model automatically learns and extracts the alignment pattern between audio and video through its complex neural network structure. This includes but is not limited to the synchronization of audio signals and lip movements, the matching of speech rhythm and video frame rate, etc. Time deviation correction: The model analyzes the possible time deviations between audio and video, and tries to correct these deviations through internal adjustments to ensure the synchronization of audio and video. This usually involves temporal translation or scaling of audio or video signals. Feature consistency detection: Based on the alignment pattern learning, the model further analyzes the consistency between audio and video features. This includes checking whether audio features (such as spectrum, base frequency) and video features (such as lip shape, motion trajectory) match each other to verify whether they come from the same source or have been tampered with. Anomaly detection: Based on the results of the above alignment and consistency analysis, the model is able to identify anomalies or inconsistencies between audio and video. These anomalies may be signs of forgery, such as lip shape not matching speech, unnatural editing or splicing in audio signals, etc. Comprehensive judgment: Ultimately, the model will combine all information (including alignment patterns, time deviations, feature consistency, etc.) to make a comprehensive judgment, that is, whether the input digital human data is forged. This judgment result will be used as the output of the model for subsequent applications or processing.

[0103] In addition, the construction process of the above-mentioned multimodal large model will be further described in detail in the subsequent specific embodiments of the present application, and will not be elaborated in detail here.

[0104] The present application first obtains an input digital human video; then extracts a target video frame from the digital human video based on a preset extraction strategy; and extracts a target audio signal from the digital human video; then performs feature extraction on the target video frame based on a preset visual encoder to obtain corresponding visual features, and performs feature extraction on the target audio signal based on a preset audio encoder to obtain corresponding audio features; subsequently, the visual features and the audio features are transformed based on a preset projection layer to obtain corresponding visual modal embedding and audio modal embedding; the visual modal embedding and the audio modal embedding are further fused to obtain a corresponding target feature vector; finally, the target feature vector is predicted based on a pre-trained multimodal large model to generate a forgery recognition result corresponding to the digital human video. The present application extracts target video frames and target audio signals from the input digital human video, and then extracts features from the target video frames and target audio signals based on the use of a visual encoder and an audio encoder to obtain corresponding visual features and audio features. The visual features and the audio features are then transformed based on a projection layer to obtain visual modal embedding and audio modal embedding. The visual modal embedding and the audio modal embedding are subsequently fused to obtain a target feature vector. Finally, the target feature vector is predicted based on a pre-trained multimodal large model to generate a forgery identification result corresponding to the digital human video. In this way, the present application achieves accurate authentication of digital human data in digital human videos by comprehensively analyzing the alignment of audio features and video features in digital human videos and performing in-depth analysis and processing based on the use of a multimodal large model, and provides efficient and accurate authentication results, thereby effectively improving the accuracy and reliability of digital human authentication.

[0105] In some optional implementations, step S206 includes the following steps:

[0106] The visual modality embedding and the audio modality embedding are transformed into the same feature space to obtain corresponding target visual modality embedding and target audio modality embedding.

[0107] In this embodiment, the above-mentioned specific implementation process of converting the visual modality embedding and the audio modality embedding into the same feature space to obtain the corresponding target visual modality embedding and target audio modality embedding will be described in further details in subsequent specific embodiments of the present application and will not be elaborated in detail here.

[0108] Get the preset fusion strategy.

[0109] In this embodiment, a fusion mechanism is pre-designed according to actual processing requirements to effectively combine audio embedding and visual embedding to form a cross-modal joint representation. Specifically, the fusion strategy includes but is not limited to strategies such as concatenation, weighted sum, and attention mechanism. Among them. Concatenation refers to simply concatenating visual embedding and audio embedding into a longer vector. Weighted Sum refers to weighted summing of visual embedding and audio embedding according to a certain weight distribution. Attention mechanism refers to using the attention mechanism to dynamically adjust the importance of visual embedding and audio embedding in the fusion process. In addition, a fusion layer for executing the above fusion strategy can be pre-designed.

[0110] Based on the fusion strategy, the target visual modality embedding and the target audio modality embedding are fused to obtain a corresponding fusion feature vector.

[0111] In this embodiment, the target visual modality embedding and the target audio modality embedding may be fused according to the fusion processing steps included in the above fusion strategy to obtain a corresponding fused feature vector, namely the above target feature vector.

[0112] The fused feature vector is used as the target feature vector.

[0113] The present application obtains the corresponding target visual modality embedding and target audio modality embedding by converting the visual modality embedding and the audio modality embedding into the same feature space; then obtains a preset fusion strategy; then fuses the target visual modality embedding and the target audio modality embedding based on the fusion strategy to obtain a corresponding fusion feature vector; and subsequently uses the fusion feature vector as the target feature vector. The present application obtains the target visual modality embedding and the target audio modality embedding by converting the visual modality embedding and the audio modality embedding into the same feature space, and then fuses the target visual modality embedding and the target audio modality embedding based on the use of the fusion strategy, so as to achieve fast and accurate completion of the fusion processing of the visual modality embedding and the audio modality embedding to obtain the target feature vector, improve the processing efficiency of the fusion processing of the multimodal embedding, ensure the data accuracy of the obtained target feature vector, and facilitate the subsequent prediction processing of the target feature vector based on the pre-trained multimodal large model, so as to achieve accurate generation of the forged recognition result corresponding to the digital human video, thereby improving the accuracy and reliability of the forged recognition of the video person contained in the digital human video.

[0114] In some optional implementations of this embodiment, converting the visual modality embedding and the audio modality embedding into the same feature space to obtain corresponding target visual modality embedding and target audio modality embedding includes the following steps:

[0115] The visual modality embedding and the audio modality embedding are processed by feature dimension matching to obtain a corresponding first visual modality embedding and first audio modality embedding.

[0116] In this embodiment, if it is detected that the dimensions of the visual modality embedding and the audio modality embedding are different, their dimensions can be adjusted by adding a fully connected layer (FC layer) or an embedding layer to ensure that the dimensions of the visual modality embedding and the audio modality embedding are the same or compatible. Exemplarily, if the dimension of the visual modality embedding is 1024 and the dimension of the audio modality embedding is 512, an additional FC layer can be added after the audio modality embedding to expand the dimension of the audio modality embedding to 1024.

[0117] Perform feature normalization processing on the first visual modality embedding and the first audio modality embedding to obtain a corresponding second visual modality embedding and a second audio modality embedding.

[0118] In this embodiment, the embeddings of the two modalities (the first visual modality embedding and the first audio modality embedding) can be normalized by using a normalization method to ensure that they have similar distribution and range in value, which helps the model to better process data of different modalities. Among them, the normalization method can use Min-Max normalization, Z-Score normalization and other methods.

[0119] Get the preset nonlinear transformation function.

[0120] In this embodiment, the nonlinear transformation function may specifically adopt activation functions such as ReLU and Tanh.

[0121] The nonlinear transformation function performs feature mapping processing on the second visual modality embedding and the second audio modality embedding in the feature space to obtain the corresponding target visual modality embedding and the target audio modality embedding.

[0122] In this embodiment, the embedding is further mapped by using nonlinear transformation functions to increase the expressiveness of the features. These transformations help map features of different modalities into more similar feature spaces, making subsequent fusion easier.

[0123] The present application performs feature dimension matching processing on the visual modality embedding and the audio modality embedding to obtain the corresponding first visual modality embedding and the first audio modality embedding; then performs feature normalization processing on the first visual modality embedding and the first audio modality embedding to obtain the corresponding second visual modality embedding and the second audio modality embedding; then obtains a preset nonlinear transformation function; and subsequently performs feature mapping processing on the second visual modality embedding and the second audio modality embedding by the nonlinear transformation function in the feature space to obtain the corresponding target visual modality embedding and the target audio modality embedding. The present application performs feature dimension matching processing, feature normalization processing, and feature mapping processing on the visual modality embedding and the audio modality embedding, so as to quickly and accurately convert the visual modality embedding and the audio modality embedding into the same feature space, thereby obtaining the corresponding target visual modality embedding and the target audio modality embedding, thereby improving the accuracy and intelligence of the conversion process. It is beneficial for the target visual modality embedding and the target audio modality embedding to be able to interact and merge seamlessly in the subsequent model.

[0124] In some optional implementations, step S202 includes the following steps:

[0125] Continuous video frames are extracted from the digital human video.

[0126] In this embodiment, video frames are extracted from the digital human video to obtain continuous video frames.

[0127] A preprocessing operation is performed on the video frame to obtain a corresponding first video frame.

[0128] In this embodiment, the above preprocessing operations at least include denoising, cropping, scaling and other operations to ensure that the image quality meets the model input requirements.

[0129] The first video frame is segmented into blocks to obtain a second video frame.

[0130] In this embodiment, the above-mentioned block segmentation processing refers to patching the first video frame, dividing each frame into several small blocks (patches) to facilitate subsequent feature extraction. Through the block segmentation processing, the video data will be converted into small blocks that are easier to analyze, which is conducive to enabling the model to extract and use visual information more efficiently.

[0131] The second video frame is used as the target video frame.

[0132] The present application extracts continuous video frames from the digital human video, and then performs preprocessing operations on the video frames to obtain the corresponding first video frame; then performs block segmentation processing on the first video frame to obtain the second video frame; and subsequently uses the second video frame as the target video frame. The present application extracts continuous video frames from the digital human video, and then performs preprocessing operations and block segmentation processing on the video frames to obtain the required target video frame, which effectively ensures that the image quality of the obtained target video frame meets the model input requirements, and the use of the target video frame is conducive to enabling the model to extract and utilize visual information more efficiently.

[0133] In some optional implementations, step S203 includes the following steps:

[0134] Audio is extracted from the digital human video to obtain a corresponding audio signal.

[0135] In this embodiment, the corresponding audio signal can be extracted from the audio stream contained in the digital human video.

[0136] The audio signal is subjected to noise reduction processing to obtain a processed first audio signal.

[0137] In this embodiment, the obtained audio signal is subjected to noise reduction processing to remove background noise and retain clear speech features. Noise reduction processing helps to improve the quality of audio features, so that the model can more accurately analyze and recognize audio information in subsequent steps.

[0138] The first audio signal is preprocessed based on a preset processing strategy to obtain a corresponding second audio signal.

[0139] In this embodiment, the processing strategy includes resampling, denoising, voice activity detection (VAD) and other processing to extract a pure voice signal from the audio signal to obtain the second audio signal.

[0140] The second audio signal is used as the target audio signal.

[0141] The present application obtains the corresponding audio signal by performing audio extraction on the digital human video; then performs noise reduction processing on the audio signal to obtain a processed first audio signal; then pre-processes the first audio signal based on a preset processing strategy to obtain a corresponding second audio signal; and subsequently uses the second audio signal as the target audio signal. The present application obtains the corresponding audio signal by performing audio extraction on the digital human video, and then performs noise reduction processing and pre-processing on the audio signal to obtain the required target audio signal, thereby ensuring the accuracy of the obtained target audio signal and retaining clear speech features, which helps to improve the quality of the audio features and facilitates the model to more accurately analyze and identify audio information in subsequent steps.

[0142] In some optional implementations of this embodiment, before step S207, the electronic device may further perform the following steps:

[0143] Obtain pre-built and annotated digital human video sample data.

[0144] In this embodiment, the digital human video sample data includes a large number of real digital human videos and forged digital human videos.

[0145] The digital human video sample data is divided into a training data set and a test data set.

[0146] In this embodiment, the digital human video sample data can be divided into a training data set and a test data set according to a preset division ratio. There is no specific limitation on the selection of the division ratio, which can be set according to actual training requirements, for example, 7:3.

[0147] Call the preset initial multimodal large model.

[0148] In this embodiment, the initial multimodal large model is a pre-built neural network structure capable of receiving multimodal input. The neural network structure may include one or more fusion layers for combining embeddings of different modalities. Specifically, the initial multimodal large model may adopt any one of BL IP-2, LLaVA, Mini GPT-4, etc.

[0149] Determine the target loss function and target optimization algorithm.

[0150] In this embodiment, the target loss function may specifically adopt a cross entropy loss function. The target optimization algorithm may specifically adopt a back propagation algorithm.

[0151] Based on the target loss function and the target optimization algorithm, the initial multimodal large model is trained using the training data set to obtain a corresponding designated model.

[0152] In this embodiment, the initial multimodal large model is trained by using the labeled training data set to learn the natural alignment pattern between the audio modality and the visual modality and the potential forgery pattern. During the training process, the model gradually learns the natural alignment pattern between the audio signal and the lip movement, as well as the abnormal features in the forged data. Then, the model parameters are optimized through the back propagation algorithm and the loss function to improve the performance of the model in the digital human forgery detection task, thereby obtaining a trained designated model.

[0153] A performance evaluation is performed on the specified multimodal large model based on the test data set.

[0154] In this embodiment, the authentication performance of the specified model is evaluated on the test data set, including performance indicators such as accuracy and recall.

[0155] If the specified multimodal large model meets the preset performance expectations, the specified model is used as the multimodal large model.

[0156] In this embodiment, if the obtained performance indicators such as accuracy and recall rate all meet the corresponding performance indicator thresholds, it is determined that the specified multimodal large model meets the performance expectations, and then the specified model is used as the multimodal large model. Among them, the above-mentioned performance indicator thresholds at least include accuracy thresholds and recall thresholds. There is no specific limitation on the value of the performance indicator threshold, which can be set according to actual business needs. In addition, if the obtained performance indicators such as accuracy and recall rate do not all meet the corresponding performance indicator thresholds, the model structure and parameters are adjusted according to the evaluation results to further improve the accuracy and robustness of the model.

[0157] The present application obtains pre-constructed and annotated digital human video sample data; and divides the digital human video sample data into a training data set and a test data set; then calls a preset initial multimodal large model; then determines the target loss function and the target optimization algorithm; subsequently, based on the target loss function and the target optimization algorithm, uses the training data set to train the initial multimodal large model to obtain the corresponding designated model; further based on the test data set, the performance of the designated multimodal large model is evaluated; if the designated multimodal large model meets the preset performance expectations, the designated model is used as the multimodal large model. The present application uses pre-constructed digital human video sample data to train and evaluate the performance of the called initial multimodal large model based on the predetermined target loss function and target optimization algorithm, thereby obtaining a multimodal large model that meets the performance expectations, so as to realize the rapid and intelligent completion of the construction of the multimodal large model, improve the construction efficiency of the multimodal large model, and ensure the accuracy and robustness of the obtained multimodal large model.

[0158] In some optional implementations of this embodiment, after taking the designated model as the multimodal large model, the electronic device may further perform the following steps:

[0159] A model storage strategy corresponding to the multimodal large model is obtained.

[0160] In this embodiment, there is no specific limitation on the selection of the above-mentioned model storage strategy, which can be determined according to the actual business storage requirements. Exemplarily, the model storage strategy can adopt any one of local database storage, cloud server storage, network disk storage, blockchain storage, etc. Among them, blockchain storage is preferably used as the above-mentioned model storage strategy. By using blockchain to store and manage multimodal large models, the security and immutability of multimodal large models can be effectively guaranteed.

[0161] The corresponding target storage medium is called based on the model storage strategy.

[0162] In this embodiment, for storage strategies such as local database storage, cloud server storage, network disk storage, and blockchain storage, the corresponding storage media are local database, cloud server, network disk, and blockchain, respectively.

[0163] A designated storage location corresponding to the multimodal large model is determined in the target storage medium.

[0164] In this embodiment, a designated storage location for specifically storing model data of a model type may be predefined in the target storage medium.

[0165] The multimodal large model is stored in the designated storage location in the target storage medium.

[0166] In this embodiment, after determining the designated storage location corresponding to the multimodal large model, the multimodal large model can be stored in the designated storage location in the target storage medium to complete the standardized storage of the multimodal large model.

[0167] The present application obtains the model storage strategy corresponding to the multimodal large model; then calls the corresponding target storage medium based on the model storage strategy; then determines the designated storage location corresponding to the multimodal large model in the target storage medium; and subsequently stores the multimodal large model in the designated storage location in the target storage medium. After completing the construction of the multimodal large model, the present application will also intelligently obtain the model storage strategy corresponding to the multimodal large model, and call the corresponding target storage medium based on the model storage strategy, and then store the multimodal large model in the corresponding designated storage location in the target storage medium, so as to realize the intelligent storage and standardized storage of the multimodal large model, and effectively ensure the data security and non-tamperability of the multimodal large model.

[0168] In some optional implementations, the user information obtained is subject to the user's consent and complies with relevant laws and policies.

[0169] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0170] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned forged identification results, the above-mentioned forged identification results can also be stored in a node of a blockchain.

[0171] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0172] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0173] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0174] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0175] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0176] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a digital human video recognition device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0177] like Figure 3 As shown, the digital human video recognition device 300 described in this embodiment includes: a first acquisition module 301, a first extraction module 302, a second extraction module 303, a third extraction module 304, a conversion module 305, a fusion module 306 and a prediction module 307. Among them:

[0178] The first acquisition module 301 is used to acquire an input digital human video;

[0179] A first extraction module 302, configured to extract a target video frame from the digital human video based on a preset extraction strategy;

[0180] The second extraction module 303 is used to extract the target audio signal from the digital human video;

[0181] A third extraction module 304 is used to extract features of the target video frame based on a preset visual encoder to obtain corresponding visual features, and to extract features of the target audio signal based on a preset audio encoder to obtain corresponding audio features;

[0182] A conversion module 305 is used to convert the visual features and the audio features based on a preset projection layer to obtain corresponding visual modality embedding and audio modality embedding;

[0183] A fusion module 306 is used to fuse the visual modality embedding and the audio modality embedding to obtain a corresponding target feature vector;

[0184] The prediction module 307 is used to perform prediction processing on the target feature vector based on the pre-trained multimodal large model to generate a forgery recognition result corresponding to the digital human video.

[0185] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the digital human video recognition method in the above-mentioned implementation mode, and will not be described in detail here.

[0186] In some optional implementations of this embodiment, the fusion module 306 includes:

[0187] A conversion submodule, used to convert the visual modality embedding and the audio modality embedding into the same feature space to obtain a corresponding target visual modality embedding and a target audio modality embedding;

[0188] The acquisition submodule is used to obtain the preset fusion strategy;

[0189] A fusion submodule, used for fusing the target visual modality embedding and the target audio modality embedding based on the fusion strategy to obtain a corresponding fusion feature vector;

[0190] The first determining submodule is used to use the fused feature vector as the target feature vector.

[0191] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the digital human video recognition method in the above-mentioned implementation mode, and will not be described in detail here.

[0192] In some optional implementations of this embodiment, the conversion submodule includes:

[0193] A matching unit, configured to perform feature dimension matching processing on the visual modality embedding and the audio modality embedding to obtain a corresponding first visual modality embedding and a first audio modality embedding;

[0194] A processing unit, configured to perform feature normalization processing on the first visual modality embedding and the first audio modality embedding to obtain a corresponding second visual modality embedding and a second audio modality embedding;

[0195] An acquisition unit, used for acquiring a preset nonlinear transformation function;

[0196] A mapping unit is used to perform feature mapping processing on the second visual modality embedding and the second audio modality embedding by the nonlinear transformation function in the feature space to obtain the corresponding target visual modality embedding and the target audio modality embedding.

[0197] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the digital human video recognition method in the above-mentioned implementation mode, and will not be described in detail here.

[0198] In some optional implementations of this embodiment, the first extraction module 302 includes:

[0199] A first extraction submodule, used to extract continuous video frames from the digital human video;

[0200] An operation submodule, used for performing a preprocessing operation on the video frame to obtain a corresponding first video frame;

[0201] A segmentation submodule, configured to perform a block segmentation process on the first video frame to obtain a second video frame;

[0202] The second determining submodule is configured to use the second video frame as the target video frame.

[0203] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the digital human video recognition method in the above-mentioned implementation mode, and will not be described in detail here.

[0204] In some optional implementations of this embodiment, the second extraction module 303 includes:

[0205] The second extraction submodule is used to extract audio from the digital human video to obtain a corresponding audio signal;

[0206] a noise reduction submodule, configured to perform noise reduction processing on the audio signal to obtain a processed first audio signal;

[0207] a preprocessing submodule, configured to preprocess the first audio signal based on a preset processing strategy to obtain a corresponding second audio signal;

[0208] The third determining submodule is configured to use the second audio signal as the target audio signal.

[0209] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the digital human video recognition method in the above-mentioned implementation mode, and will not be described in detail here.

[0210] In some optional implementations of this embodiment, the digital human video recognition device further includes:

[0211] The second acquisition module is used to acquire pre-constructed and annotated digital human video sample data;

[0212] A division module, used for dividing the digital human video sample data into a training data set and a test data set;

[0213] A first calling module is used to call a preset initial multimodal large model;

[0214] A first determination module is used to determine a target loss function and a target optimization algorithm;

[0215] A training module, used to train the initial multimodal large model using the training data set based on the target loss function and the target optimization algorithm to obtain a corresponding designated model;

[0216] An evaluation module, configured to perform a performance evaluation on the specified multimodal large model based on the test data set;

[0217] The second determination module is used to use the designated model as the multimodal large model if the designated multimodal large model meets the preset performance expectations.

[0218] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the digital human video recognition method in the above-mentioned implementation mode, and will not be described in detail here.

[0219] In some optional implementations of this embodiment, the digital human video recognition device further includes:

[0220] A third acquisition module is used to acquire a model storage strategy corresponding to the multimodal large model;

[0221] A second calling module, used to call the corresponding target storage medium based on the model storage strategy;

[0222] A third determination module, configured to determine a designated storage location corresponding to the multimodal large model in the target storage medium;

[0223] A storage module is used to store the multimodal large model in the designated storage location in the target storage medium.

[0224] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the digital human video recognition method in the above-mentioned implementation mode, and will not be described in detail here.

[0225] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0226] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.

[0227] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0228] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Med ia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the digital human video recognition method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0229] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the computer-readable instructions stored in the memory 41 or process data, such as running the computer-readable instructions of the digital human video recognition method.

[0230] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0231] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the digital human video recognition method as described above.

[0232] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0233] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for recognizing a digital human video, characterized in that: The steps include: Get the input digital human video; Extracting a target video frame from the digital human video based on a preset extraction strategy; Extracting a target audio signal from the digital human video; Extracting features of the target video frame based on a preset visual encoder to obtain corresponding visual features, and extracting features of the target audio signal based on a preset audio encoder to obtain corresponding audio features; The visual features and the audio features are transformed based on a preset projection layer to obtain corresponding visual modality embedding and audio modality embedding; Fusing the visual modality embedding with the audio modality embedding to obtain a corresponding target feature vector; The target feature vector is predicted and processed based on a pre-trained multimodal large model to generate a forged recognition result corresponding to the digital human video.

2. The method for recognizing a digital human video according to claim 1, characterized in that: The step of fusing the visual modality embedding and the audio modality embedding to obtain a corresponding target feature vector specifically includes: Converting the visual modality embedding and the audio modality embedding into the same feature space to obtain a corresponding target visual modality embedding and a target audio modality embedding; Get the preset fusion strategy; Based on the fusion strategy, the target visual modality embedding and the target audio modality embedding are fused to obtain a corresponding fusion feature vector; The fused feature vector is used as the target feature vector.

3. The method for recognizing a digital human video according to claim 2, characterized in that: The step of converting the visual modality embedding and the audio modality embedding into the same feature space to obtain corresponding target visual modality embedding and target audio modality embedding specifically includes: Performing feature dimension matching processing on the visual modality embedding and the audio modality embedding to obtain a corresponding first visual modality embedding and first audio modality embedding; Performing feature normalization processing on the first visual modality embedding and the first audio modality embedding to obtain a corresponding second visual modality embedding and a second audio modality embedding; Obtain a preset nonlinear transformation function; The nonlinear transformation function performs feature mapping processing on the second visual modality embedding and the second audio modality embedding in the feature space to obtain the corresponding target visual modality embedding and the target audio modality embedding.

4. The method for recognizing a digital human video according to claim 1, characterized in that: The step of extracting the target video frame from the digital human video based on a preset extraction strategy specifically includes: Extracting continuous video frames from the digital human video; Performing a preprocessing operation on the video frame to obtain a corresponding first video frame; Performing block segmentation processing on the first video frame to obtain a second video frame; The second video frame is used as the target video frame.

5. The method for recognizing a digital human video according to claim 1, characterized in that: The step of extracting the target audio signal from the digital human video specifically includes: Extracting audio from the digital human video to obtain a corresponding audio signal; Performing noise reduction processing on the audio signal to obtain a processed first audio signal; Preprocessing the first audio signal based on a preset processing strategy to obtain a corresponding second audio signal; The second audio signal is used as the target audio signal.

6. The method for recognizing a digital human video according to claim 1, characterized in that: Before the step of predicting the target feature vector based on the pre-trained multimodal large model to generate a forgery recognition result corresponding to the digital human video, the method further includes: Obtain pre-constructed and annotated digital human video sample data; Dividing the digital human video sample data into a training data set and a test data set; Call the preset initial multimodal large model; Determine the target loss function and target optimization algorithm; Based on the target loss function and the target optimization algorithm, the initial multimodal large model is trained using the training data set to obtain a corresponding designated model; Performing a performance evaluation on the specified multimodal large model based on the test data set; If the specified multimodal large model meets the preset performance expectations, the specified model is used as the multimodal large model.

7. The method for recognizing a digital human video according to claim 6, characterized in that: After the step of using the designated model as the multimodal large model, the method further includes: Acquire a model storage strategy corresponding to the multimodal large model; Calling a corresponding target storage medium based on the model storage strategy; determining, in the target storage medium, a designated storage location corresponding to the multimodal large model; The multimodal large model is stored in the designated storage location in the target storage medium.

8. A digital human video recognition device, characterized in that: include: A first acquisition module, used to acquire an input digital human video; A first extraction module, used for extracting a target video frame from the digital human video based on a preset extraction strategy; A second extraction module, used to extract a target audio signal from the digital human video; A third extraction module is used to extract features of the target video frame based on a preset visual encoder to obtain corresponding visual features, and to extract features of the target audio signal based on a preset audio encoder to obtain corresponding audio features; A conversion module, used for converting the visual features and the audio features based on a preset projection layer to obtain corresponding visual modality embedding and audio modality embedding; A fusion module, used to fuse the visual modality embedding and the audio modality embedding to obtain a corresponding target feature vector; The prediction module is used to perform prediction processing on the target feature vector based on a pre-trained multimodal large model to generate a forgery recognition result corresponding to the digital human video.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the method for recognizing a digital human video as claimed in any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the method for recognizing a digital human video according to any one of claims 1 to 7 are implemented.

Images