Virtual anchor recognition method based on emotion perception and semi-supervised contrastive learning

Abstract

The present invention discloses a virtual anchor recognition method based on emotion perception and semi-supervised contrastive learning, which relates to the technical field of virtual video recognition and comprises the following steps: randomly sampling continuous video frames to obtain sequence frames; performing facial recognition on the pre-processed sequence frames, and cropping the recognized faces; inputting the obtained facial image sequence into an emotion perception and recognition module for emotion recognition, and marking the unlabeled facial image sequence with pseudo labels; constructing a contrastive learning data set; passing the contrastive learning data set through a virtual anchor representation learning network, passing the facial representation through a designed virtual anchor representation contrastive learning network, and then passing the learned feature representation through a virtual anchor recognition network to obtain a prediction result; the virtual anchor recognition method and system based on emotion perception and semi-supervised contrastive learning provided by the present invention can recognize virtual anchors.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual video recognition, and in particular to a virtual anchor recognition method based on emotion perception and semi-supervised contrastive learning. Background Art

[0002] As a revolutionary vehicle for the digital cultural industry, virtual anchors' technological evolution is strongly coupled with breakthroughs in artificial intelligence. Driven by deep generative technologies like generative adversarial networks (GANs) and neural radiance fields (NeRFs), virtual anchors have transcended the limitations of traditional 3D modeling, achieving realistic micro-expression modeling with 0.1mm accuracy and speech synthesis naturalness scores exceeding 4.5 points. Currently, my country's virtual anchor industry has surpassed 100 billion yuan in scale, penetrating deeply into e-commerce live streaming, news broadcasting, and educational science programs. Its intelligent production and broadcasting systems significantly improve content production efficiency.

[0003] The rapid development of virtual anchors has become a focus of public attention, but it also faces several challenges. First, the iterative upgrades of deep synthesis technology have led to a blurring of the perceptual boundaries between virtual anchors and real-life anchors. Traditional recognition methods based on unimodal feature analysis are no longer effective against multimodally coupled deep virtual content. Second, the cognitive confusion effect caused by high-fidelity virtual anchors has exceeded the critical threshold of human visual perception, posing a significant risk to social trust mechanisms. Consequently, virtual anchors may spread inaccurate information and be used to manipulate public opinion or engage in other illegal activities such as fraud and defamation. Finally, existing supervised learning frameworks exhibit algorithmic vulnerabilities when dealing with dynamic adversarial attacks, particularly in the context of rapidly evolving generative models, which suffer from severe generalization deficiencies. Therefore, constructing a semi-supervised contrastive learning system that integrates emotion perception mechanisms has become a key technological breakthrough for the credible identification of virtual anchors. No relevant technologies for virtual anchor identification exist in the existing art. Summary of the Invention

[0004] The purpose of the present invention is to provide a virtual anchor recognition method based on emotion perception and semi-supervised contrastive learning, which can determine whether the anchor in the video is a virtual anchor.

[0005] The technical solutions of the present invention are as follows:

[0006] The virtual anchor recognition method based on emotion perception and semi-supervised contrastive learning includes the following steps:

[0007] Sampling continuous video frames to obtain sequence frames;

[0008] The facial region of the sequence frames is extracted through the facial recognition model to obtain a sequence of facial images;

[0009] The unlabeled face image sequence is input into the emotion perception and recognition model to perform emotion recognition to obtain pseudo labels;

[0010] The facial image sequence is input into the virtual anchor representation learning network to obtain feature representation; the virtual anchor representation learning network is trained using a comparative learning dataset, which includes labeled facial image sequences and pseudo-labeled facial image sequences;

[0011] The feature representation is input into the virtual anchor recognition network to obtain the recognition result.

[0012] Furthermore, before extracting facial regions from the sequence frames using a facial recognition model to obtain a facial image sequence, the method further includes performing an image enhancement processing step on the sequence frames.

[0013] Specifically, the sampling process of continuous video frames to obtain sequence frames includes:

[0014] In a live video stream, a continuous sequence of original frames of fixed length is generated by sampling at random starting points;

[0015] Random frame sampling is performed on the continuous original frame sequence according to a preset discard rate.

[0016] Specifically, extracting facial regions from sequence frames using a facial recognition model to obtain a facial image sequence includes:

[0017] Scaling and / or normalizing the sequence frames to obtain input sequence frames;

[0018] Input the input sequence frames into the facial recognition model to obtain the face bounding box;

[0019] The facial region is cropped using the face bounding box, and all facial regions are adjusted to the same size to obtain a face image sequence.

[0020] Specifically, inputting the unlabeled facial image sequence into the emotion perception and recognition model to perform emotion recognition to obtain pseudo labels includes:

[0021] The global feature vector is obtained by extracting the global feature of the whole face image. , is the global feature vector dimension;

[0022] The whole face image is divided into n*n local area blocks, and local feature vectors are obtained after local feature extraction for each local area block. ;

[0023] Calculate the similarity scores between the local feature vector and the global feature vector to obtain the position weight score of each local area block;

[0024] After guiding the local vector through the position weight score, the local vector is fused with the global feature vector to generate a fused feature map;

[0025] A clustering algorithm is used to perform unsupervised clustering on the fused feature maps, and the obtained classification results are used as pseudo labels for the unlabeled face image sequence.

[0026] Preferably, the similarity score calculation between the local feature vector and the global feature vector is specifically performed by using dot product similarity and normalization operation to calculate the similarity score, and the formula is:

[0027] ,

[0028] in, is the similarity score between the i-th local feature vector and the global feature vector, is the global eigenvector, is the i-th local eigenvector, is a globally learnable weight parameter, is a locally learnable weight parameter, and T is a transposition operation.

[0029] Specifically, after guiding the local vector by the position weight score, the local vector is fused with the global feature vector to generate a fused feature map, which includes:

[0030] Weight each local eigenvector to obtain the weighted local eigenvector ,

[0031]

[0032] in, is the weighted i-th local eigenvector;

[0033] According to the original divided n×n grid position, the n×n weighted local feature vectors are arranged at the corresponding positions to form a low-resolution feature map. The channel dimension is adjusted through 1×1 convolution to generate a recombined feature map. ;

[0034] The global eigenvector Transformed into and reorganized feature maps through fully connected layers Updated global feature map of the same dimension ;

[0035] Reorganize the feature map and update the global feature map Splice by channel dimension to generate fusion feature map ;

[0036] Through multi-layer deconvolution, the fusion feature map is gradually Upsample to the original face image resolution.

[0037] Preferably, the virtual anchor representation learning network adopts a deep 3D convolutional neural network, including a first convolutional layer, a second convolutional layer, a third convolutional layer, an Inception module and a Softmax layer connected in sequence; the first convolutional layer adopts a 7×7×7 convolution kernel; the second convolutional layer adopts a 1×1×1 convolution kernel; the third convolutional layer uses a 3×3×3 convolution kernel; a MaxPooling layer is provided after the first convolutional layer and the third convolutional layer; the Inception module has three convolution branches, the first convolution branch is a 1×1×1 convolution, the second convolution branch is a 1×1×1 convolution followed by a 3×3×3 convolution, and the third convolution branch is a 1×1×1 convolution followed by a 5×5×5 convolution. Finally, the outputs of the three branches are spliced ​​and merged to form a single feature representation; the feature representation output by the Inception module is normalized by the Softmax function to obtain an empirical distribution.

[0038] Preferably, the virtual anchor representation learning network is trained by comparing learning data sets, including:

[0039] Construct a contrastive learning dataset, where the labeled face image sequence is denoted as , the pseudo-labeled face image sequence is recorded as ;exist and In each pair of sequence data , if the sequence x i and the sequence x j The facial image sequences of both real anchors or virtual anchors are marked as 1, which is a positive sample pair, recorded as , if the sequence x i and the sequence x j The face image sequence of a real anchor and the face image sequence of a virtual anchor are marked as 0, which are negative sample pairs, recorded as ;

[0040] Input the contrastive learning dataset into the virtual anchor representation learning network to obtain the empirical distribution of each pair of sequence data. The virtual anchor represents the data pair input to the learning network, sharing the same parameters; after training the contrastive learning network, the parameters of the deep 3D convolutional neural network are fixed;

[0041] Specifically, the face image sequence The empirical distribution of , It means that the virtual anchor represents the learning network;

[0042] Paired sequence data Calculate the KL divergence of the empirical distribution:

[0043]

[0044] in, represents the KL divergence value, represents the edge parameter, whose random value range is (0,1); is a positive sample pair, represents a negative sample pair.

[0045] Specifically, the loss function of the virtual anchor representation learning network is:

[0046]

[0047] in, express The logarithm of the sequence data P in express The logarithm of the sequence data P in .

[0048] After adopting the above scheme, the beneficial effects of the present invention are as follows:

[0049] The technology of the present invention can identify virtual anchors and real anchors in video frames, filling the gap in this field and being used in the fields of anti-fraud, anti-defamation, etc. Compared with existing virtual video recognition technology, the present invention focuses on the different changes in facial muscles caused by emotions between virtual anchors and real anchors when assigning pseudo-labels. It learns the facial features of anchors through a position-based autoencoder without using manual features. In this way, local feature differences can be amplified, making pseudo-label classification more accurate. The KL divergence and loss function designed by the present invention enable the model to continuously optimize the discriminant features of data with pseudo-labels, guided by contrast loss, to improve discrimination performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flow chart of the core steps in a specific embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of the core steps in a specific embodiment of the present invention;

[0052] Figure 3 Schematic diagram of the random sampling process in S1 of the specific embodiment of the present invention;

[0053] Figure 4 This is the structural diagram of the Inception module in the present invention;

[0054] To avoid infringement, Figure 2 The portraits used in the video block out the face area with a special shape. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0056] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the specified features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0057] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or illustration". Any embodiment of the present invention described as "for example" is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any person skilled in the art to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed herein.

[0058] In order to automatically identify virtual anchors, this specific implementation provides a virtual anchor recognition method based on emotion perception and semi-supervised contrastive learning. After randomly sampling live videos or historical videos to form a sequence of frames and preprocessing, the pre-trained SSD face detection network extracts the facial representation of each frame. By constructing a contrastive learning data set, the facial representation is passed through a designed virtual anchor representation learning network, and then the learned feature representation is passed through the virtual anchor recognition network to obtain a prediction result, thereby determining whether the anchor is a virtual anchor. Through semi-supervised processing, the present invention can perform better training based on partially labeled data, thereby obtaining recognition results.

[0059] The method of the present invention includes a training part and a testing part. For the training part, Figure 1 and Figure 2As shown, the following steps are included:

[0060] S1. Randomly sample continuous video frames to obtain sequence frames; the continuous video frames include virtual anchor video frames and real anchor video frames.

[0061] S1.1. In a live video stream, generate a continuous sequence of raw frames of fixed length by sampling at random starting points.

[0062] S1.2. For the continuous original frame sequence, random frame sampling is performed according to the preset drop rate, and redundant video frames are removed to reduce the computational complexity. For the live broadcast video (frame rate 30fps, resolution 720*1280), Figure 3 As shown in the figure, a continuous sequence with a fixed length of 10 seconds is randomly sampled as the original sequence, and then some frames with a length of 2 seconds in the original sequence are randomly discarded to form a final sequence frame with a size of 240*720*1280, that is, the size of a sample is 240*720*1280.

[0063] The above S1.1 and S1.2 represent the sub-steps of S1, and the following text is similar.

[0064] S2. Perform image enhancement on the sequence of frames; image enhancement can improve the robustness of features. Common image enhancement processes include noise injection, sharpening, cropping, translation, flipping, etc. This specific implementation uses common image enhancement processes;

[0065] S3. The facial region of the sequence frame is extracted by a pre-trained facial recognition model, and the extracted facial region is resized to obtain a sequence of face images; this specific implementation of the facial recognition model is a two-dimensional model, so when processing, each image in the sequence frame is resized, and then the adjusted images are spliced ​​to obtain a sequence of face images; S3 specifically comprises the following steps:

[0066] S31. Scale and normalize the preprocessed sequence frames to obtain input sequence frames to construct input features that meet the input specifications of the deep neural network.

[0067] Specifically, the input sequence frames are uniformly scaled to the standard size of 720×720 pixels, with an overall size of 240*720*720. Then, normalization is performed to normalize the pixel values ​​to the interval [0, 1]. Then, the average value of the current frame image is subtracted from each pixel value of each frame image.

[0068] S32. Input the input sequence frames to the facial recognition model to obtain the face bounding box. The facial recognition model uses the existing SSD model (using the VGG16 backbone network). The input of the SSD model should be a two-dimensional image, specifically an input size of 720*720. The sequence frames of an overall size of 240*720*720 obtained in S31 need to be split along the first dimension. That is, the 240-frame sequence is split into multiple sub-batches, i.e., set the batch size to 16, and then input them into the model in sequence to reduce video memory usage. The SSD model in this specific implementation has been pre-trained on a large-scale face dataset. The large-scale face dataset can be either public or private datasets, such as the public datasets CelebA and WIDER FACE. In the SSD model, a multi-layer feature map is generated through the VGG16 backbone network. These feature maps have sizes of 38×38, 19×19, 10×10, 5×5, 3×3, and 1×1, respectively. These multi-layer feature maps can capture facial features of different scales. Based on a preset aspect ratio (set to 1:1 in this implementation) and scale parameter, a priori bounding boxes are generated for each layer of feature maps, and their coordinate offsets, confidence levels, and category labels are predicted. Detection boxes with confidence levels below a dynamic threshold (set to 0.7) are then removed. Detection boxes within a single frame are deduplicated using the intersection-over-union (IoU) ratio (set to ≥ 0.5), retaining the highest confidence results to obtain a concise and accurate set of face bounding boxes. Specifically, the SSD model is an existing technology, and the above description only describes the key technical points. The model details and code are publicly available and readily understood by those skilled in the art.

[0069] S33. Crop the facial region using the face bounding box and adjust all facial regions to the same size to obtain a facial image sequence with a size of 240*300*300.

[0070] The sequence objects processed by the above S1-S3 include labeled sequence data and unlabeled sequence data; the labeled sequence data comes from the virtual anchor video frames and the real anchor video frames; similarly, the unlabeled sequence data comes from the virtual anchor video frames and the real anchor video frames; for the unlabeled face image sequence, pseudo labels need to be obtained, that is, enter S4, for the labeled face image sequence directly enter S5.

[0071] S4. Input the obtained unlabeled face image sequence into the emotion perception and recognition model for emotion recognition to obtain pseudo labels.

[0072] The facial muscles of virtual anchors and real anchors change differently when expressing emotions, so a position-based autoencoder is used to learn the anchor's facial features. S4 includes the following operations:

[0073] S41. Obtain the global feature vector for the entire face image through the global feature extraction model ; The volume global feature extraction model here includes multiple layers of 3D convolution. Through 3D convolution, the spatiotemporal dimensions are gradually compressed, and the global feature vector dimension generated is finally 1*1*1*128. The settings of each convolution layer are: the first 3D convolution layer, the first 3D maximum pooling layer, the second 3D convolution layer, the second 3D maximum pooling layer, the third 3D convolution layer and the first adaptive 3D average pooling layer. The convolution kernel of the first 3D convolution layer and the second 3D convolution layer is 3*3*3, with a step size of 2 and a padding of 1. The convolution kernel of the first 3D maximum pooling layer and the second 3D maximum pooling layer is 2*2*2, with a step size of 2. The convolution kernel of the third 3D convolution layer is 3*3*3, with a step size of 1 and a padding of 1.

[0074] S42. The whole face image is divided into n*n local area blocks. In this specific embodiment, n is 3, and 3*3 local area blocks are used for illustration; local feature vectors are obtained after performing local feature extraction model on each local area block. The 3*3 here refers to each image in the face image sequence, so the divided local area is also a sequence as a whole. The settings of each convolution layer are: the fourth 3D convolution layer, the third 3D maximum pooling layer, the fifth 3D convolution layer, the sixth 3D convolution layer and the second adaptive 3D average pooling. The convolution kernels of the fourth 3D convolution layer, the fifth 3D convolution layer and the sixth 3D convolution layer are 3*3*3, the step size is 2, and the padding is 1. The convolution kernel of the third 3D maximum pooling layer is 2*2*2, and the step size is 2.

[0075] S43. Calculate the similarity score between the local feature vector and the global feature vector to obtain the position weight score of each local area block. Specifically, the similarity score calculation is performed using dot product similarity and normalization operations, and the formula is:

[0076] ,

[0077] in, is the similarity score between the i-th local feature vector and the global feature vector, is the global eigenvector, is the i-th local eigenvector, is a globally learnable weight parameter, is a locally learnable weight parameter, and T is a transposition operation.

[0078] S44. After guiding the local vector through the position weight score, the local vector is fused with the global feature vector to generate a fused feature map ; The S44 specifically includes:

[0079] S441. Weight each local feature vector to obtain a weighted local feature vector , specifically:

[0080]

[0081] in, is the weighted i-th local feature vector; this operation can amplify the local area features that are highly correlated with the global features and suppress irrelevant areas.

[0082] S442. Arrange the 3×3 weighted local feature vectors in the corresponding positions according to the original 3×3 grid positions to form a low-resolution feature map. Adjust the channel dimension through 1×1×1 convolution to generate a reconstructed feature map. .

[0083] S443. The global feature vector Convert to and restructure feature maps Updated global feature map of the same dimension .

[0084] S444. Reorganize the feature map and update the global feature map Splice by channel dimension to generate fusion feature map ; The fusion feature map combines global information and local information, which can improve the reconstruction quality.

[0085] The fusion feature map is gradually upsampled to the original face image resolution through multi-layer deconvolution; in this specific implementation, 5 layers of 3D deconvolution layers are used, including the first 3D deconvolution layer, the second 3D deconvolution layer, the third 3D deconvolution layer, the fourth 3D deconvolution layer and the fifth 3D deconvolution layer. The convolution kernel of the second 3D deconvolution layer of the first 3D deconvolution layer is 1*4*4, the step size is 1*2*2, and the padding is 0*1*1. The convolution kernel of the third 3D deconvolution layer is 1*5*5, the step size is 1*3*3, and the padding is 0*1*1. The convolution kernels of the fourth and fifth 3D deconvolution layers are 1*6*6, the step size is 1*5*5, and the padding is 0*1*1; multi-layer deconvolution gradually expands the resolution, and finally adjusts the number of channels to 3, and the last layer uses The activation function constrains the output value to the range of [0,1], that is, restores the high-resolution face image by gradually upsampling.

[0086] The loss function of the S41-S44 process is: , constrained fusion feature map sequence Pixel and semantic consistency with the original face image sequence g.

[0087] S45. Use clustering algorithm to fusion feature map Unsupervised clustering is performed, and the classification results obtained are used as pseudo-labels for the unlabeled facial image sequence. The clustering algorithm here can be any commonly used unsupervised clustering algorithm. This specific implementation uses the KMeans clustering algorithm, with two clustered categories, "0" or "1" respectively. In this way, each clustered category is assigned a pseudo-label. Because the virtual anchor and the real anchor are very similar visually, it is not possible to directly use binary clustering to distinguish the entire image. The emotional changes in facial muscles of the virtual anchor and the real anchor are different. Therefore, operations S41-S44 are performed to learn the anchor's facial features through a position-based autoencoder. This can amplify local feature differences.

[0088] S5. Constructing a contrastive learning dataset The entire contrastive learning dataset contains some labeled real anchor sequences and some unlabeled virtual anchor sequences, specifically, labeled real anchor face sequences, unlabeled real anchor face sequences, labeled virtual anchor face sequences and unlabeled virtual anchor face sequences; the labeled face image sequences are denoted as (including the real anchor face sequence with labels and the virtual anchor face sequence with labels), the original unlabeled face image sequence obtained by S4 is recorded as (including the original unlabeled real anchor face sequence and the original unlabeled virtual anchor face sequence); and In each pair of sequence data , if the sequence x i and the sequence x j The facial image sequences of both real anchors or virtual anchors are marked as 1, which is a positive sample pair, recorded as , if the sequence x i and the sequence x j The face image sequence of a real anchor and the face image sequence of a virtual anchor are marked as 0, which are negative sample pairs, recorded as , it should be noted that the sequence x i and the sequence x j Either from , or both come from .

[0089] S6. Input the contrastive learning dataset into the virtual anchor representation learning network to obtain the empirical distribution of each pair of sequence data. It is a virtual anchor that represents the data pair input to the learning network and shares the same parameters.

[0090] The virtual anchor representation learning network uses a deep 3D convolutional neural network, including the first convolutional layer, the second convolutional layer, the third convolutional layer, the Inception module and the Softmax layer connected in sequence. The first convolutional layer uses a 7×7×7 convolution kernel to extract low-level information; the second convolutional layer uses a 1×1×1 convolution kernel to reduce feature size and total parameters; the third convolutional layer uses a 3×3×3 convolution kernel to extract high-level information; MaxPooling layers are set after the first and third convolutional layers; the Inception module is an existing module, such as Figure 4 As shown in the figure, it is used to extract temporal features at different scales and enhance the model's representation capabilities. The Inception module consists of three convolutional branches. The first convolutional branch is a 1×1×1 convolution. The second convolutional branch performs a 1×1×1 convolution followed by a 3×3×3 convolution. The third convolutional branch performs a 1×1×1 convolution followed by a 5×5×5 convolution. The outputs of these three branches are then concatenated and merged to form a single feature representation. The Inception module is used to extract temporal features at different scales and enhance the model's representation capabilities.

[0091] The feature representation output by the Inception module is normalized by the Softmax function to obtain a face image sequence. Experience distribution , Indicates that the virtual anchor represents the learning network.

[0092] Paired sequence data Calculate the KL divergence of the empirical distribution:

[0093]

[0094] in, represents the KL divergence value, Indicates that the edge parameter takes a random value in the range (0,1); is a positive sample pair, Represents a negative sample pair; a positive sample pair represents a similar sequence frame data set, and a negative sample pair represents a dissimilar sequence frame data set, that is, the representation value of similar sequence frames is made larger, and the representation of dissimilar values ​​is made smaller.

[0095] The virtual anchor represents the loss function of the learning network:

[0096]

[0097] in, express The logarithm of the sequence data P in express The logarithm of the sequence data P in the

[0098] After training the contrastive learning network for the virtual anchor representation, the parameters of the deep 3D convolutional neural network are fixed. The virtual anchor representation learning network obtains a better feature representation through contrastive learning, and then directly classifies the obtained better feature representation.

[0099] S7. The output of the virtual anchor representation learning network is recognized by the virtual anchor recognition network; after the training of the virtual anchor representation comparison learning network in S6, the parameters of the deep 3D convolutional neural network are fixed; specifically, the virtual anchor recognition network adopts a feedforward neural network followed by a Function, the virtual anchor recognition network is used to classify the output of the representation learning network. A conventional classification network can be used. In this specific implementation, a two-layer feedforward neural network is used for classification, and the loss function uses cross entropy loss to obtain the distribution of different categories, which can automatically determine whether the anchor is a virtual anchor.

[0100] In the present invention, the SSD facial recognition model, the virtual anchor representation learning network and the virtual anchor recognition network are trained separately.

[0101] The testing part (i.e., the real-time identification part) of the present invention includes the following steps:

[0102] S100. Sampling the continuous video frames to obtain sequence frames; the continuous video frames here are unknown frames, that is, it is unknown whether they are virtual anchor video frames or real anchor video frames.

[0103] S200. Extract facial regions from sequence frames using a trained facial recognition model to obtain a facial image sequence.

[0104] S300. Input the facial image sequence into the trained virtual anchor representation learning network to obtain feature representation.

[0105] S400: Input the feature representation into the virtual anchor recognition network to obtain a recognition result.

[0106] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0107] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A virtual anchor recognition method based on emotion perception and semi-supervised contrastive learning, characterized by: The steps include: Sampling continuous video frames to obtain sequence frames; The facial region of the sequence frames is extracted through the facial recognition model to obtain a sequence of facial images; The unlabeled face image sequence is input into the emotion perception and recognition model to perform emotion recognition to obtain pseudo labels; The facial image sequence is input into the virtual anchor representation learning network to obtain feature representation; the virtual anchor representation learning network is trained using a comparative learning dataset, which includes labeled facial image sequences and pseudo-labeled facial image sequences; Input the feature representation into the virtual anchor recognition network to obtain the recognition result; The virtual anchor representation learning network is trained by comparing learning data sets, including: Construct a contrastive learning dataset, where the labeled face image sequence is denoted as , the pseudo-labeled face image sequence is recorded as ;exist and In each pair of sequence data , if the sequence x i and the sequence x j The facial image sequences of both real anchors or virtual anchors are marked as 1, which is a positive sample pair, recorded as , if the sequence x i and the sequence x j The face image sequence of a real anchor and the face image sequence of a virtual anchor are marked as 0, which are negative sample pairs, recorded as ; Input the contrastive learning dataset into the virtual anchor representation learning network to obtain the empirical distribution of each pair of sequence data. The virtual anchor represents the data pair input to the learning network, sharing the same parameters; after training the contrastive learning network, the parameters of the deep 3D convolutional neural network are fixed; Specifically, the face image sequence The empirical distribution of , It means that the virtual anchor represents the learning network; Paired sequence data Calculate the KL divergence of the empirical distribution: ； in, represents the KL divergence value, represents the edge parameter, whose random value range is (0,1); is a positive sample pair, represents a negative sample pair; The step of inputting the unlabeled facial image sequence into the emotion perception and recognition model to perform emotion recognition to obtain pseudo labels comprises: The global feature vector is obtained by extracting the global feature of the whole face image. , is the global feature vector dimension; The whole face image is divided into n*n local area blocks, and local feature vectors are obtained after local feature extraction for each local area block. ; Calculate the similarity scores between the local feature vector and the global feature vector to obtain the position weight score of each local area block; After guiding the local vector through the position weight score, the local vector is fused with the global feature vector to generate a fused feature map; A clustering algorithm is used to perform unsupervised clustering on the fused feature maps, and the obtained classification results are used as pseudo labels for the unlabeled face image sequence.

2. The virtual anchor recognition method based on emotion perception and semi-supervised contrastive learning according to claim 1 is characterized in that: Before extracting facial regions from the sequence frames using the facial recognition model to obtain a sequence of facial images, the method further includes performing an image enhancement process on the sequence frames.

3. The virtual anchor recognition method based on emotion perception and semi-supervised contrastive learning according to claim 1 is characterized in that: The sampling process of continuous video frames to obtain sequence frames includes: In a live video stream, a continuous sequence of original frames of fixed length is generated by sampling at random starting points; Random frame sampling is performed on the continuous original frame sequence according to a preset discard rate.

4. The virtual anchor recognition method based on emotion perception and semi-supervised contrastive learning according to claim 1 is characterized in that The facial region extraction of the sequence frames by the facial recognition model to obtain the facial image sequence includes: Scaling and / or normalizing the sequence frames to obtain input sequence frames; Input the input sequence frames into the facial recognition model to obtain the face bounding box; The facial region is cropped using the face bounding box, and all facial regions are adjusted to the same size to obtain a face image sequence.

5. The virtual anchor identification method based on emotion perception and semi-supervised contrastive learning according to claim 1 is characterized in that: The similarity score calculation between the local feature vector and the global feature vector is specifically performed by using dot product similarity and normalization operation to calculate the similarity score, and the formula is: ； in, is the similarity score between the i-th local feature vector and the global feature vector, is the global eigenvector, is the i-th local eigenvector, is a globally learnable weight parameter, is a locally learnable weight parameter, and T is a transposition operation.

6. The virtual anchor identification method based on emotion perception and semi-supervised contrastive learning according to claim 1 is characterized in that: After the local vector is guided by the position weight score, the local vector is fused with the global feature vector to generate a fused feature map, which includes: Weight each local eigenvector to obtain the weighted local eigenvector , ； in, is the weighted i-th local eigenvector; According to the original divided n×n grid position, the n×n weighted local feature vectors are arranged at the corresponding positions to form a low-resolution feature map. The channel dimension is adjusted through 1×1 convolution to generate a recombined feature map. ; The global eigenvector Transformed into and reorganized feature maps through fully connected layers Updated global feature map of the same dimension ; Reorganize the feature map and update the global feature map Splice by channel dimension to generate fusion feature map ; Through multi-layer deconvolution, the fusion feature map is gradually Upsample to the original face image resolution.

7. The virtual anchor identification method based on emotion perception and semi-supervised contrastive learning according to claim 1 is characterized in that: The virtual anchor representation learning network adopts a deep 3D convolutional neural network, including a first convolutional layer, a second convolutional layer, a third convolutional layer, an Inception module and a Softmax layer connected in sequence; the first convolutional layer adopts a 7×7×7 convolution kernel; the second convolutional layer adopts a 1×1×1 convolution kernel; the third convolutional layer uses a 3×3×3 convolution kernel; a MaxPooling layer is provided after the first convolutional layer and the third convolutional layer; the Inception module has three convolution branches, the first convolution branch is a 1×1×1 convolution, the second convolution branch is a 1×1×1 convolution followed by a 3×3×3 convolution, and the third convolution branch is a 1×1×1 convolution followed by a 5×5×5 convolution. Finally, the outputs of the three branches are spliced ​​and merged to form a single feature representation; The feature representation output by the Inception module is normalized by the Softmax function to obtain the empirical distribution.

8. The virtual anchor recognition method based on emotion perception and semi-supervised contrastive learning according to claim 1 is characterized in that ,The virtual anchor represents the loss function of the learning network as follows: ； in, express The logarithm of the sequence data P in express The logarithm of the sequence data P in .

Images