An interactive face voice labeling method, device, equipment and storage medium

Abstract

The application provides an interactive face voice labeling method and device, equipment and storage medium, first, receiving a face image, and using a convolutional neural network to extract features of the face image to generate face features; then, obtaining a voice signal, and using a recurrent neural network and a fully connected layer to extract features of the voice signal to generate voice features; then, according to identity label information of the face image and the voice signal, calling data in a training data batch as an identity prototype feature to perform an alignment operation on the voice features and the face features; finally, labeling the face image and the voice signal based on the alignment operation, effectively accurately mapping different modal features to a common feature space, realizing comparability between heterogeneous modal information, thereby improving the accuracy of interactive face voice labeling.

Description

Technical Field

[0001] This invention relates to the field of biometric matching in pattern recognition, and particularly to an interactive face and voice annotation method, apparatus, device, and storage medium. Background Technology

[0002] With the further development of cross-modal matching technology, researchers have focused on addressing the imbalance in the amount of label information between different modalities by utilizing cross-modal interactive annotation techniques. The core of this research lies in leveraging the inherent modal information correlation between face and speech, and the rich label information of one modality to annotate another modality with scarce label information. This allows the supplemented modal information to be effectively applied to the training of downstream tasks. The core of this task is to construct the correlation between different modal information, using various high-level semantic constraints to map the feature information of different modalities into a common high-dimensional feature space, thereby achieving comparability of heterogeneous modal data in the high-dimensional feature space. Since the amount of data and data structure contained in image and audio data are inconsistent, setting reasonable high-level semantic constraints to enable the feature encoder to accurately and effectively map information into the feature space is a key technical challenge in achieving this task. Currently, face-speech annotation methods are still in a relatively preliminary exploratory stage and face many practical problems, specifically: 1) Indirect alignment within the same modality is not conducive to the formation of a cross-modal common high-dimensional feature space; 2) The weighting strategy of model methods on training data only remains at a coarse-grained level of sample identity. These problems lead to inaccuracies in the modal common feature space, weak perception performance for data with large distribution differences, and poor model generalization performance.

[0003] In view of the above, this application is hereby submitted. Summary of the Invention

[0004] This invention discloses an interactive face and voice annotation method, apparatus, device, and storage medium, which aims to effectively and accurately map different modal features to a common feature space, realize the comparability between heterogeneous modal information, and thus improve the accuracy of interactive face and voice annotation.

[0005] The first embodiment of the present invention provides an interactive face and voice annotation method, including:

[0006] The system receives a face image and uses a convolutional neural network to extract features from the face image to generate face features.

[0007] The speech signal is acquired, and a recurrent neural network and a fully connected layer are used to extract features from the speech signal to generate speech features;

[0008] Based on the identity label information of the face image and the voice signal, data from the training data batch is used as an approximation of the identity prototype features to perform an alignment operation on the voice features and the face features;

[0009] The face image and the voice signal are labeled based on the alignment operation.

[0010] Preferably, the step of receiving a face image and extracting features from the face image using a convolutional neural network to generate face features specifically involves:

[0011] Acquire a face image, and after detecting and locating the face region, crop and scale the face image to a preset image size;

[0012] After normalizing the cropped face image, a convolutional neural network is used for feature extraction. The face feature representation is learned through end-to-end training, and a 512-dimensional face feature is output. The convolutional neural network includes convolutional layers, pooling layers, and fully connected layers.

[0013] Preferably, the step of acquiring the speech signal and using a recurrent neural network and a fully connected layer to extract features from the speech signal to generate speech features specifically involves:

[0014] Acquire a speech signal, and perform speech activity detection to segment the speech signal into speech segments;

[0015] The segmented speech segments are uniform in duration and normalized in volume. A recurrent neural network is used to perform temporal modeling of the speech segments, and then a fully connected layer is connected to extract speech features, outputting 128-dimensional speech features.

[0016] Preferably, the alignment operation includes:

[0017] Intramodal features are directly perceived and semantically aligned through prototype-independent intramodal contrastive loss, and intermodal features with the same identity are semantically aligned through intermodal contrastive loss.

[0018] Preferably, it further includes: constructing a loss function for the alignment operation, specifically:

[0019] In the alignment task within a single modality, sample data is randomly selected based on identity labels and substituted into the contrastive learning to generate a prototype-independent contrastive learning loss function within the modality.

[0020] In a training batch of paired data, an intermodal contrast loss function is constructed by minimizing the distance between feature information of the same identity while maximizing the distance between feature information of different identities.

[0021] The alignment operation loss function is generated based on the intra-modal prototype-independent contrastive learning loss function and the inter-modal contrastive loss function.

[0022] Preferably, it further includes:

[0023] During model training, the distance between features of sample data pairs is continuously measured to calculate the approximate distance value of the data pairs. This value is then used to determine the degree of deviation of the sample data pairs from their corresponding identity categories. Based on the degree of deviation, the sample data pairs are divided into biased pairs and over-biased pairs.

[0024] Preferably, it also includes constraining the bias pairs during the training process using a triplet loss training method.

[0025] A second embodiment of the present invention provides an interactive face and voice annotation device, comprising:

[0026] A face feature generation unit is used to receive a face image and extract features from the face image using a convolutional neural network to generate face features;

[0027] A speech feature generation unit is used to acquire speech signals and extract features from the speech signals using a recurrent neural network and a fully connected layer to generate speech features;

[0028] The alignment operation unit is used to perform an alignment operation on the voice features and the face features by calling data from the training data batch as approximate identity prototype features based on the identity label information of the face image and the voice signal;

[0029] The annotation unit is used to annotate the face image and the speech signal based on the alignment operation.

[0030] The third embodiment of the present invention provides an interactive face and voice annotation device, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to implement an interactive face and voice annotation method as described in any of the above embodiments.

[0031] The fourth embodiment of the present invention provides a computer-readable storage medium, characterized in that it stores a computer program, which can be executed by the processor of the device where the computer-readable storage medium is located, to implement an interactive face and voice annotation method as described in any of the above claims.

[0032] Based on the interactive face and voice annotation method, apparatus, device, and storage medium provided by this invention, a face image is first received, and a convolutional neural network is used to extract features from the face image to generate face features. Next, a voice signal is acquired, and a recurrent neural network and a fully connected layer are used to extract features from the voice signal to generate voice features. Then, based on the identity label information of the face image and the voice signal, data from a training data batch is used as approximate identity prototype features to perform an alignment operation on the voice features and the face features. Finally, the face image and the voice signal are annotated based on the alignment operation, effectively mapping different modal features accurately to a common feature space, achieving comparability between heterogeneous modal information, thereby improving the accuracy of interactive face and voice annotation. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating an interactive face and voice annotation method provided in the first embodiment of the present invention;

[0034] Figure 2 This invention provides an interactive face-speech annotation framework that combines differential training and prototype-independent methods.

[0035] Figure 3 This is a schematic diagram of an interactive face and voice annotation device provided in the second embodiment of the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0038] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0039] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0040] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0041] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0042] The terms "first" and "second" used in the embodiments are merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permissible. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein.

[0043] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0044] This invention discloses an interactive face and voice annotation method, apparatus, device, and storage medium, which aims to effectively and accurately map different modal features to a common feature space, realize the comparability between heterogeneous modal information, and thus improve the accuracy of interactive face and voice annotation.

[0045] The first embodiment of the present invention provides an interactive face and voice annotation method, which can be executed by an interactive face and voice annotation device (hereinafter referred to as the annotation device), specifically, by one or more processors within the annotation device, to at least implement the following steps:

[0046] S101: Receive a face image and use a convolutional neural network to extract features from the face image to generate face features;

[0047] In this embodiment, the labeling device can be a desktop computer, laptop computer, server, workstation, or other terminal with data processing and analysis capabilities. The labeling device can be equipped with a corresponding operating system and application software, and the functions required in this embodiment are achieved through the combination of the operating system and application software.

[0048] Specifically, in this embodiment, a face image is acquired, and the face image is then detected, the face region is located, and the image is cropped and scaled to a preset image size.

[0049] After normalizing the cropped face image, a convolutional neural network is used for feature extraction. The face feature representation is learned through end-to-end training, and a 512-dimensional face feature is output. The convolutional neural network includes convolutional layers, pooling layers, and fully connected layers.

[0050] It should be noted that, based on the identity labels of the modal data, different modal data with the same identity semantics are combined into data pairs for input training the model. For the face images in the data pairs, mean and normalization preprocessing is performed. To avoid loss of generality, the overall face-speech training dataset is as follows: ,in Represents the first in the face dataset Zhang's facial image data, This indicates that the first [item] in the audio dataset... One audio segment data, This represents the number of face-speech sample pairs in the dataset. Let... Identity labels representing the sample set, , This represents the total number of identity labels in the dataset;

[0051] Define a high-level semantic feature extractor for different modal information as a face network encoder. and voice network encoder , for the Face-Voice Data Pair Feature extraction can be represented as:

[0052] ,

[0053] Face network encoder A ResNet-50 network encoder is used. For each input face image, the initial features extracted by the face image sub-network are 512-dimensional, and then a fully connected mapping layer is applied. Generate 128-dimensional facial features .

[0054] S102, Acquire the speech signal, and use a recurrent neural network and a fully connected layer to extract features from the speech signal to generate speech features;

[0055] Specifically, in this embodiment, a speech signal is acquired, and speech activity detection is performed on the speech signal to segment speech segments.

[0056] The segmented speech segments are uniform in duration and normalized in volume. A recurrent neural network is used to perform temporal modeling of the speech segments, and then a fully connected layer is connected to extract speech features, outputting 128-dimensional speech features.

[0057] Voice Subnetwork Encoder The pre-trained speech feature extraction network ThinResNet34 is used. For each input speech sample, the initial features extracted by the speech sub-network are 128-dimensional, and then further processed by a fully connected mapping layer. Generate 128-dimensional speech features .

[0058] S103, based on the identity label information of the face image and the voice signal, call the data in the training data batch as an approximation of the identity prototype feature to perform an alignment operation on the voice feature and the face feature;

[0059] Specifically, in this embodiment,

[0060] The alignment operation includes: performing direct mutual perception and semantic alignment of intramodal features through prototype-independent intramodal contrastive loss, and performing semantic alignment of intermodal features with the same identity through intermodal contrastive loss.

[0061] It should be noted that the implementation of interactive face and speech annotation requires that data from different modalities be mapped to a common feature space to ensure comparability. At the same time, it is necessary to align features from different modalities to minimize the distance between features of different modalities with the same identity and maximize the distance between features of different modalities with different identities, so that features with the same identity semantics can be aligned and correspond across different modalities.

[0062] Traditional intra-modal alignment strategies achieve feature alignment by pre-defining identity prototype features and then aligning intra-modal features to their corresponding identity prototype features. However, this method prevents features from perceiving each other, resulting in poor intra-modal feature alignment and ultimately indirectly leading to inaccurate construction of the common feature space.

[0063] Based on the above research, this embodiment proposes a prototype-independent feature alignment strategy, which aims to discard the prototype features in traditional methods, directly select existing data to extract corresponding features to approximate the identity feature prototype, and then constrain the sample feature data to be close to the corresponding approximate prototype, thereby achieving the effect of single modality feature alignment.

[0064] To achieve a prototype-independent alignment strategy, this embodiment defines the model's input data set as follows: ,in and These represent facial features respectively. and speech features Features of other samples with the same identity Representation of sample pairs The same set of features for the same identity.

[0065] Compared to traditional methods that rely on pre-defined identity prototype features, this embodiment uses features generated by sub-networks to approximate and replace the prototype features of traditional methods, thereby enabling features to perceive each other. Specifically, in the alignment task within a single modality, this embodiment selects sample data pairs based on identity labels and randomization. Because in a training batch Since the data represents different identities, this embodiment can approximate this batch of data as identity prototype features, and thus perform alignment operations within a single modality.

[0066] S104, the face image and the voice signal are labeled based on the alignment operation.

[0067] In this embodiment, it further includes: constructing a loss function for the alignment operation, specifically:

[0068] In the alignment task within a single modality, sample data is randomly selected based on identity labels and substituted into the contrastive learning to generate a prototype-independent contrastive learning loss function within the modality.

[0069] In a training batch of paired data, an intermodal contrast loss function is constructed by minimizing the distance between feature information of the same identity while maximizing the distance between feature information of different identities.

[0070] The alignment operation loss function is generated based on the intra-modal prototype-independent contrastive learning loss function and the inter-modal contrastive loss function.

[0071] It should be noted that substituting the sample data randomly selected based on identity labels as identity prototype features into the traditional contrastive learning method yields the following intramodal prototype-independent contrastive learning loss function:

[0072]

[0073]

[0074]

[0075] in This is the temperature hyperparameter in metric learning. In this embodiment, it can be set to 0.07, but it is not limited to this.

[0076] Inter-modal feature alignment strategies can be achieved by minimizing the distance between feature information of the same identity while maximizing the distance between feature information of different identities. In a training batch... In the data, different modalities of the same identity can be constructed into a data pair. Data from different identities and different modalities are then used to construct negative pairs. The corresponding intermodal contrast loss can be expressed as:

[0077]

[0078]

[0079] Therefore, cross-face-voice contrastive learning in the common subspace is formalized as follows:

[0080]

[0081] Therefore, combining the two feature alignment losses—one within a single modality and the other across modalities—the cross-face-voice prototype-independent contrastive learning in the common subspace can be represented as:

[0082] .

[0083] In this embodiment, it also includes:

[0084] During model training, the distance between features of sample data pairs is continuously measured to calculate the approximate distance value of the data pairs. This value is then used to determine the degree of deviation of the sample data pairs from their corresponding identity categories. Based on the degree of deviation, the sample data pairs are divided into biased pairs and over-biased pairs.

[0085] It should be noted that prototype-independent feature alignment methods rely on the accuracy of the sample data. This embodiment continuously measures the distance between features of sample data pairs during model training to calculate approximate distance values ​​for the data pairs, and uses this to determine the degree of deviation of the sample data pairs relative to their corresponding identity categories. Based on the degree of deviation, the sample data pairs are further divided into biased pairs and over-biased pairs. Generally, sample pairs that are negatively correlated are considered biased pairs if the semantic features they correspond to have high similarity. Therefore, this embodiment uses the following similarity scores to filter out biased pairs in the sample data pairs:

[0086]

[0087] in The threshold hyperparameters are set to 0.5 in this invention. Based on the similarity scores mentioned above, this invention can filter out biased data pairs from the training samples in real time. .

[0088] In this embodiment, the method of training with triplet loss is also included to constrain the bias pairs during the training process.

[0089] It should be noted that, for the deviation pairs, this embodiment employs further constraints to enable them to be repositioned to the correct relative position. For each face-voice sample pair... A positive sample pair can be constructed. , This involves removing biased pairs from a feature set containing identical identities, along with a negative sample. , Since bias pairs have greater training value compared to other training data, they can provide more effective guidance for model training. Therefore, this embodiment further constrains bias pairs through a triplet loss training method, with the loss defined as follows:

[0090]

[0091]

[0092] For discrete cosine distance, To constrain constants, Represents taking a set The number of Show set If the number of elements is such that the overall deviation constrains the joint triplet loss as follows:

[0093]

[0094] The overall training loss function can then be defined as follows: .

[0095] The above embodiments are further described below:

[0096] Dataset Introduction: This dataset is extracted from the large-scale audiovisual dataset AVSpeech, which contains 290,000 YouTube videos. The audio information in each video segment belongs to only one visible speaker in the video. Based on the dataset's gender balance criterion, this embodiment selected 550 high-quality data points with different identities to form a new face-speech dataset, named AVSpeech-CM. The parameters used in this embodiment's experiments are set as follows: stochastic gradient optimization uses the Adam optimizer, and the learning rate is... The value was set to 0.001, and the batch size was set to 20. To verify the effectiveness of this embodiment, experiments were conducted on interactive face-speech verification and interactive 1:2 annotation tasks, and some representative benchmark methods in the field were selected for experimental comparison.

[0097] Interactive Verification Task: This task focuses on interactive face-speech verification, aiming to determine whether a given face image and audio recording belong to the same person. It is a binary classification problem. The evaluation metric used for this task is AUC (area under the ROC curve).

[0098] Interactive 1:2 annotation task. Given a face image sample, the 1:2 annotation task requires selecting a speech sample (FV) from two speech data sets that matches the given sample's identity. Similarly, given a speech sample, it requires selecting a face sample (VF) from two face samples that matches the given sample's identity. The 1:2 annotation task has two scenarios: "FV" and "VF". The evaluation metric used for the interactive 1:2 annotation task is accuracy. To verify the effectiveness of this invention, representative methods are selected for comparison:

[0099] The experimental results are shown in Table 1, where "G" indicates that the input face-speech pair during testing comes from people of the same gender, while "U" indicates that the data has no gender restriction and is obtained through random selection. The experimental results show that this embodiment achieves better results than the benchmark method on different test tasks. The SVHF method, because it only constrains triplet data pairs, cannot simultaneously perceive the relative relationships of multiple different feature data in each batch training, resulting in a weak perception of the entire feature space and poor performance. DIMNet focuses on handling cases where data cannot be input in pairs, leading to a fundamental difference in data input compared to dual-stream paired data input, resulting in performance deficiencies. Although the SSS method uses contrastive learning for inter-modal feature alignment, it still uses the traditional multi-class prototype feature alignment method for single-modal feature alignment, resulting in poor model training performance. In contrast, this embodiment fully considers the interrelationship between intramodal and intermodal feature alignment and proposes a prototype-independent alignment method to fundamentally solve the problem of differences in the common feature space caused by indirect feature alignment. Furthermore, it utilizes a differentiated training strategy to further improve data utilization and the model's generalization performance. As shown in Table 1, this embodiment demonstrates a significant improvement over the benchmark method on all test tasks, and experiments verify the effectiveness of this method.

[0100] Table 1 Comparison of test results of the present invention and various methods.

[0101]

[0102] Please see Figure 3 The second embodiment of the present invention provides an interactive face and voice annotation device, comprising:

[0103] The face feature generation unit 201 is used to receive a face image and extract features from the face image using a convolutional neural network to generate face features;

[0104] The speech feature generation unit 202 is used to acquire speech signals and use a recurrent neural network and a fully connected layer to extract features from the speech signals to generate speech features;

[0105] Alignment operation unit 203 is used to perform alignment operation on the voice feature and the face feature by calling data from the training data batch as approximate identity prototype features based on the identity label information of the face image and the voice signal;

[0106] The annotation unit 204 is used to annotate the face image and the speech signal based on the alignment operation.

[0107] The third embodiment of the present invention provides an interactive face and voice annotation device, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to implement an interactive face and voice annotation method as described in any of the above embodiments.

[0108] The fourth embodiment of the present invention provides a computer-readable storage medium, characterized in that it stores a computer program, which can be executed by the processor of the device where the computer-readable storage medium is located, to implement an interactive face and voice annotation method as described in any of the above claims.

[0109] Exemplary examples show that the computer program described in the third and fourth embodiments of the present invention can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in implementing an interactive face and voice annotation device. For example, the apparatus described in the second embodiment of the present invention.

[0110] The processor referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the interactive face and voice annotation method, connecting various parts of the interactive face and voice annotation method through various interfaces and lines.

[0111] The memory can be used to store the computer program and / or modules. The processor implements various functions of an interactive face and voice annotation method by running or executing the computer program and / or modules stored in the memory and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, text conversion function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, text message data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD card), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0112] If the implemented module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0113] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0114] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An interactive face speech annotation method, characterized in that, The method comprises the following steps: receiving a face image, and extracting features of the face image by using a convolutional neural network to generate face features; obtaining a speech signal, and extracting features of the speech signal by using a recurrent neural network and a fully connected layer to generate speech features; According to the identity label information of the face image and the speech signal, the data in the training data batch is approximated as an identity prototype feature to perform an alignment operation on the speech features and the face features; wherein the alignment operation comprises: through the prototype-independent intra-modal contrast loss, the intra-modal features are directly perceived and semantically aligned with each other, and through the inter-modal contrast loss, the inter-modal features of the same identity are semantically aligned, specifically: On the alignment task in a single mode, according to the identity label and randomly selecting sample data as the identity prototype feature, the intra-modal prototype-independent contrast learning loss function is generated; in a training batch of data, the inter-modal contrast loss function is constructed by minimizing the distance between the feature information of the same identity and maximizing the distance between the feature information of different identities; the loss function of the alignment operation is generated based on the intra-modal prototype-independent contrast learning loss function and the inter-modal contrast loss function; During the model training process, the distance between the features of the sample data pairs is continuously measured, the approximate distance value of the data pairs is calculated, and the deviation degree of the sample data pairs relative to the corresponding identity category is judged according to the approximate distance value, and the sample data pairs are divided into deviation pairs and super deviation pairs according to the deviation degree; at the same time, the deviation pairs are constrained through the training mode of the triplet loss in the training process; Based on the alignment operation, the face image and the speech signal are labeled.

2. The interactive face speech annotation method of claim 1, wherein, The receiving face image, and extracting features of the face image by using a convolutional neural network to generate face features, specifically comprises: obtaining a face image, and detecting and positioning the face region of the face image, then cropping and scaling the face image to a preset image size; after normalizing the cropped face image, extracting features by using a convolutional neural network, learning face feature representation through end-to-end training, and outputting 512-dimensional face features, wherein the convolutional neural network comprises a convolutional layer, a pooling layer, and a fully connected layer.

3. The interactive face speech annotation method of claim 1, wherein, The obtaining speech signal, and extracting features of the speech signal by using a recurrent neural network and a fully connected layer to generate speech features, specifically comprises: obtaining a speech signal, and performing speech activity detection on the speech signal to segment out speech segments; after unifying the time length and normalizing the volume of the segmented speech segments, modeling the time sequence of the speech segments by using a recurrent neural network, and then inputting the fully connected layer to extract speech features, outputting 128-dimensional speech features.

4. An interactive face speech annotation apparatus, characterized by, The method comprises the following steps: a face feature generation unit for receiving a face image, and extracting features of the face image by using a convolutional neural network to generate face features; a speech feature generation unit for obtaining a speech signal, and extracting features of the speech signal by using a recurrent neural network and a fully connected layer to generate speech features; An alignment operation unit is configured to perform an alignment operation on the speech feature and the face feature according to identity label information of the face image and the speech signal by calling data in a training data batch as an identity prototype feature; wherein the alignment operation comprises: directly perceiving and aligning the features in the same modal through a prototype-independent modal intra-contrast loss, and aligning the features between different modal through an inter-modal contrast loss, specifically: In the alignment task in a single modal, identity labels are used to randomly select sample data as identity prototype features to be substituted into contrast learning to generate an intra-modal prototype-independent contrast learning loss function; in a training batch, an inter-modal contrast loss function is constructed by minimizing the distance between the feature information of the same identity and maximizing the distance between the feature information of different identities; a loss function of the alignment operation is generated based on the intra-modal prototype-independent contrast learning loss function and the inter-modal contrast loss function; During the model training process, the distance between the features of the sample data pairs is continuously measured, the approximate distance value of the data pairs is calculated, and the deviation degree of the sample data pairs relative to the corresponding identity categories is judged according to the approximate distance value, and the sample data pairs are divided into deviation pairs and super-deviation pairs according to the deviation degree; meanwhile, the deviation pairs are constrained through a training method of a ternary loss in the training process; A labeling unit is configured to label the face image and the speech signal based on the alignment operation.

5. An interactive face speech annotation device, characterized by, A computer program is stored in a memory, and the computer program can be executed by a processor to implement the interactive face and speech labeling method according to any one of claims 1 to 3.

6. A computer readable storage medium characterized by, A computer program is stored in a computer readable storage medium, and the computer program can be executed by a processor of a device where the computer readable storage medium is located to implement the interactive face and speech labeling method according to any one of claims 1 to 3.

Images