Digital human identity generation method and device

The generation of digital human identity identifiers through facial and voice feature encoding solves the problem of inconvenient generation of digital human identity identifiers and realizes the convenience and accuracy of identity confirmation and retrieval.

CN119007272APending Publication Date: 2024-11-22BEIJING MUNICIPAL PUBLIC SECURITY BUREAU
View PDF 16 Cites 0 Cited by

Patent Information

Application Number
CN202411492330.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-24
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Digital people lack reliable means of generating identity identification, resulting in inconvenience and uncertainty in their application and management.

Method used

Determine facial feature vectors by detecting videos and/or pictures of digital people, detecting speech signals to determine speech feature vectors, and encode both to generate the identity of digital people, and extract facial features using multi-task convolutional neural network and VGGFace model. The phonological features are extracted by the cepspectral coefficient of the el frequency, and finally the identity identifier is generated by the fusion of low-rank multimodal features.

Benefits of technology

It realizes reliable confirmation, re-identification and comparison of digital person identities, simplifies digital person search query and identification verification, and improves the efficiency and accuracy of digital person management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007272A_ABST
    Figure CN119007272A_ABST
Patent Text Reader

Abstract

The invention discloses a digital person identity generation method, which comprises the following steps: detecting a video and / or a picture of a digital person, and determining a facial feature vector of the digital person; detecting a voice signal of the digital human, and determining a voice feature vector; and encoding the facial feature vector and the voice feature vector to generate an identity label of the digital person. The invention further discloses a digital human identity generation device. After the facial features and the voice features of the digital person are fused and coded, the most obvious features of the digital person are reserved, so that convenience is brought to confirmation, re-recognition and comparison of the digital person; in addition, retrieval, query, identification and verification of the digital person can be realized through the identity label. According to the invention, the problem that the digital person lacks reliable identity identification at present is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method and device for generating a digital human identity. Background Art

[0002] With the development of artificial intelligence and digital technologies, digital humans have appeared in various industries, aroused widespread attention, and become an investment hotspot. As an important application that carries multimodality after GPT4, digital humans are considered to be the entrance to the metaverse and an application direction with broad development potential.

[0003] Digital human is an artificial being driven by a computer program with its own consciousness and identity. It can show the same physiological, psychological and social performance as real humans in the digital world. Compared with traditional image animation, digital human technology can achieve more complex animation effects, including verbal dialogue, gestures, expression changes, body language, etc. Its technical goal is to create a digital image close to the human image through computer graphics technology, and give it a specific character identity setting, visually shorten the psychological distance with humans, and bring more realistic emotional interaction to humans.

[0004] At present, digital human theory and technology are becoming more and more mature, the scope of application is constantly expanding, the industry is gradually forming and enriching, and the corresponding business model is also continuously evolving and diversifying. Digital human technology combines actual application scenarios and fields, cuts into various types, forms industry application solutions, and empowers the industry. According to different application scenarios or industries, there have been entertainment digital humans such as virtual anchors and virtual idols; educational digital humans such as virtual teachers; assistant digital humans such as virtual customer service, virtual tour guides, and smart assistants; film and television digital humans such as stand-ins or virtual actors, etc. Virtual digital humans with different appearances and functions empower film and television, media, games, finance, culture and tourism, and provide customized services to users according to their needs.

[0005] As an emerging technology field, digital humans still lack standardized means of dissemination, application and management. How to generate reliable identity for digital humans is the technical problem to be solved in this application. Summary of the invention

[0006] The present application proposes a method and device for generating a digital human identity. In order to solve the problem that a digital human lacks a reliable identity, the digital human's identity is generated by determining a facial feature vector and a voice feature vector of the digital human, encoding the facial feature vector and the voice feature vector.

[0007] In a first aspect, the present application provides a method for generating a digital human identity, comprising the following steps: Detecting the video and / or picture of the digital person and determining the facial feature vector of the digital person; Detect the voice signal of the digital person and determine the voice feature vector; The facial feature vector and the voice feature vector are encoded to generate the identity of the digital person.

[0008] In a second aspect, an embodiment of the present application further provides a digital human identity generation device, which is used to implement the method of any one of the embodiments of the present application, including: a facial feature processing module, a voice feature processing module and an encoding module.

[0009] The facial feature processing module is used to detect the video and / or picture of the digital human and determine the facial feature vector of the digital human.

[0010] The speech feature processing module is used to detect the speech signal of the digital human and determine the speech feature vector.

[0011] The encoding module is used to encode the facial feature vector and the voice feature vector to generate the identity of the digital person.

[0012] In a third aspect, an embodiment of the present application also provides a digital human identity generation system, including a client and a server.

[0013] The client is used to provide the digital human video and / or picture to the server.

[0014] The server is used to receive the digital human video and / or picture provided by the client to implement the method described in the first aspect.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the method described in any embodiment provided in the present application is implemented.

[0016] In a fifth aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in any embodiment provided in the present application is implemented.

[0017] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: the present application determines the facial feature vector and the voice feature vector of the digital person, and encodes the facial feature vector and the voice feature vector to generate the identity of the digital person. By fusing and encoding the facial features and voice features of the digital person, the most obvious features of the digital person are retained, thereby facilitating the confirmation, re-identification, and comparison of the digital person; in addition, the search query and identification verification of the digital person can be realized through the identity. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A schematic diagram of a digital human management system and client provided in an embodiment of the present application; Figure 2 A flow chart of a method for generating a digital human identity provided in an embodiment of the present application; Figure 3 A P-net processing block diagram provided in an embodiment of the present application; Figure 4 An R-net processing block diagram provided for an embodiment of the present application; Figure 5 An O-net processing block diagram provided for an embodiment of the present application; Figure 6 A VGGface network structure block diagram provided in an embodiment of the present application; Figure 7 A Mel-frequency cepstral coefficient calculation process provided in an embodiment of the present application; Figure 8 A schematic diagram of a low-rank multimodal feature fusion process provided in an embodiment of the present application; Fig. 9 A digital human identity retrieval flow chart provided in an embodiment of the present application; Fig.10 A framework structure diagram of a digital human identity library provided in an embodiment of the present application; Fig.11 A block diagram of a digital human identity authentication and retrieval system provided in an embodiment of the present application; Fig.12 A schematic diagram of a method and device for generating a digital human identity is provided for an embodiment of the present application; Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0020] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0021] Figure 1 A schematic diagram of a digital human management system and client provided in an embodiment of the present application. Digital human creators, users and auditing users interact with the digital human identity management system through the client to complete digital human creation, use and audit. In the digital human identity management system, multiple servers are deployed, including: digital human generation server, digital human application server, digital human detection server and digital human identity identification server. Through the interaction between the above-mentioned client and server, the digital human service needs of different users are met.

[0022] In one embodiment of the present application, when the user of the client is a digital human creator, the client is used to provide the digital human video and / or picture to the server. In other embodiments of the present application, when the user of the client is a digital human user, the client is used to read the digital human video and / or picture from the server; when the user of the client is an audit user, the client is used to read the digital human video and / or picture from the server, further identify the digital human video and / or picture, and further implement the server-permitted editing function on the digital human.

[0023] In one embodiment of the present application, any server receives the digital human video and / or picture provided by the client to perform identity recognition of the digital human; in particular, the digital human generation server can be used to generate the identity identifier of the digital human and store it in a set storage device; the digital human application server can generate the identity identifier of the digital human in real time, and compare the stored identity identifier of the digital human with the identity identifier of the digital human generated in real time to determine whether the real-time running digital human and the original digital human belong to the same digital object.

[0024] Figure 2 A flowchart of a method for generating a digital human identity provided in an embodiment of the present application includes steps 210 to 230: Step 210: Detect the video and / or picture of the digital human and determine the facial feature vector of the digital human.

[0025] In one embodiment, detecting the digital human video and / or picture and determining the digital human facial feature vector includes: using a multi-task convolutional neural network to detect images in the digital human video and / or the picture, and using VGGFace to determine the digital human facial feature vector.

[0026] The Multi-task Convolutional Neural Network (MTCNN) algorithm is a face recognition and alignment technology based on deep learning, which can perform face recognition and alignment tasks simultaneously. The MTCNN algorithm consists of three sub-networks: the proposal network (P-Net), the refinement network (R-Net), and the output network (O-Net). These three networks process faces in order from coarse to fine.

[0027] Before using these three sub-networks, the original images are scaled to different scales, and then the images of different scales are sent to these three sub-networks for training. The purpose is to detect faces of different sizes, thereby realizing multi-scale object detection.

[0028] Step 210A: The P-Net network detects and locates the face and generates a candidate face region and a bounding box.

[0029] The Proposal Network P-Net, full name Proposal Network, is the first sub-network in the MTCNN model. P-Net is a fully convolutional network (FCN) that consists of a series of convolutional layers without fully connected layers. The advantage of fully convolutional networks is that they can accept input images of any size and produce outputs of corresponding sizes, and can handle faces of various sizes. P-Net works by sliding on the input image, and each slide generates a candidate window. This window is a small area that may contain a face. The size and step size of the sliding window can be adjusted to control the accuracy of the detection and the complexity of the calculation. Usually, the network runs at multiple scales so that faces of different sizes can be detected. The main task of P-Net is to perform rough face detection and bounding box regression. For each candidate window, the network outputs a binary face classification (face / not face). P-Net is the first sub-network in the MTCNN algorithm, and its main task is to generate candidate windows and bounding box regression vectors.

[0030] For example, Figure 3As shown in the figure, the P-Net input is an RGB image with a set size of 12×12×3. During the training phase, P-Net needs to determine whether this 12×12 image area contains a face, and output the regression of the face box and the location of the key points of the face. It should be noted that 12×12 here is just an example of a numerical value. The regression of this face box is a four-element vector that represents the position and size of the face in the window. These values ​​are used to fine-tune the position and size of the detected face to more accurately locate the face. During the testing phase, the responsibility of P-Net is to output the 4 coordinate information of N bounding boxes and a score. These 4 coordinate information have been corrected by the network's face box regression, and the score can be regarded as the output of the classification, that is, the probability that the window contains a face. The first part of the network output is used to determine whether the image contains a face. The size of the output vector is 1×1×2, that is, two values, representing the probability that the image is a face and not a face.

[0031] The cross entropy loss function used in training the model is as follows: (1) in It is the probability inferred by the network that the sample is a face. This symbol represents the true label.

[0032] Step 210B: The R-Net network classifies faces.

[0033] R-Net, the full name of Refine Network, is the second stage of the Multi-Task Cascade Convolutional Neural Network (MTCNN) model. After P-Net generates a series of candidate windows, the job of R-Net is to further screen these windows.

[0034] For example, Figure 4 As shown in the figure, the input of R-Net is a candidate window of, for example, 24x24. After convolution, pooling, and full connection, three results are output: one is the face classification result, one is the bounding box regression value, and the other is the face key point alignment. The face classification result is a two-element vector that indicates the probability of whether there is a face in the window. The bounding box regression value is a four-element vector that is used to fine-tune the bounding box position and size of the face to more accurately locate the face.

[0035] The target box uses Euclidean distance to calculate the loss function: (2) in represents the coordinates of the bounding box corrected after the network output, is the ground-truth bounding box of the object.

[0036] R-Net is a deeper convolutional neural network, which aims to eliminate a large number of non-face windows to reduce the computational burden of subsequent stages and make more detailed adjustments to the remaining candidate windows. Through the above process, R-Net can not only improve the accuracy of face detection, but also effectively reduce the amount of calculation and improve the efficiency of the entire face detection process.

[0037] Step 210C: O-Net network determines the location of facial key points.

[0038] Output O-Net, full name Output Network, is the third network of the MTCNN model. The main task of O-Net is to perform final face detection, bounding box regression, and positioning of the five points of the face (two eyes, nose, and two corners of the mouth). It receives the candidate window from R-Net, and then uses a deeper convolutional neural network for more accurate face detection and bounding box regression. In addition, O-Net also outputs the coordinates of the five points of the face through regression, which is very important for subsequent tasks such as face alignment and recognition. The five key points correspond to the center of the left eye, the center of the right eye, the tip of the nose, the left corner of the mouth, and the right corner of the mouth.

[0039] For example, Figure 5 As shown in Figure 1, O-Net receives the candidate regions output by R-Net and scales them to a target size of, for example, 48×48×3, which enables O-Net to perform face detection and key point localization at a higher resolution. The structure of O-Net is similar to that of P-Net, including convolutional layers, fully connected layers, and nonlinear activation functions.

[0040] The output of O-Net consists of three parts: ① Bounding box coordinate information: O-Net will perform more precise bounding box regression on each candidate window and output more accurate coordinates of the face bounding box.

[0041] ②Face score: O-Net will output the probability score of whether each candidate window contains a face, which helps to further screen and determine the final face detection result.

[0042] ③ Positions of five key points: O-Net will output the positions of the five key points of the face (left eye center, right eye center, nose tip, left mouth corner, and right mouth corner). Each key point is represented by a 2D coordinate, so the output of the key point is a 1×1×10 vector, where 10 is the product of the number of key points and the coordinate dimension. The positions of these key points are very useful for subsequent face alignment and recognition tasks.

[0043] When training O-Net, the Euclidean distance is usually used as the loss function for the location of key points to optimize the model's key point positioning ability:

[0044] in Represents the coordinates of the key points obtained after the network output, are the real coordinates of the key points.

[0045] Steps 210A~C are optionally implemented. It should be noted that any sub-network of P-Net, R-Net and O-Net can be used for calculation. It is understandable that when only the P-Net network is used, non-face images are preliminarily screened out, and the range of faces obtained may be misjudged, but the processing speed is faster. It is also understandable that the use of the R-Net network to further determine the face probability improves the processing accuracy. However, the processing speed is slow when only R-Net is used. Further, it is also understandable that O-Net can provide accurate recognition of key facial features, but the processing speed is the slowest when only the O-Net network is used. Therefore, the combination of any two of the three can simultaneously improve the processing speed and processing accuracy.

[0046] Optimally, the MTCNN model uses three cascaded networks, P-Net, R-Net, and O-Net, each of which is responsible for different tasks, and at each stage, the candidate regions are filtered and refined, thereby improving the accuracy and efficiency of face detection. P-Net is the first stage of MTCNN, which uses a smaller convolutional neural network to quickly scan the entire image and generate candidate face regions and bounding boxes. Since the network structure of P-Net is relatively simple, it runs the fastest. R-Net is the second stage of MTCNN, which receives the candidate regions from P-Net and further filters and refines them. R-Net uses a larger convolutional neural network for more accurate face detection and bounding box regression, so it runs slightly slower than P-Net. O-Net is the third and final stage of MTCNN, which receives the candidate regions from R-Net and performs the final face detection and bounding box regression. In addition, O-Net is also responsible for outputting the locations of the five key points of the face. O-Net uses a deeper convolutional neural network for face detection and key point localization, so it runs the slowest.

[0047] Through this cascade method, the MTCNN model can reduce the number of candidate regions that need to be processed at each stage, thereby effectively reducing the calculation time while ensuring the accuracy of face detection. This is also the main reason why MTCNN can achieve high-precision and high-efficiency face detection.

[0048] Step 210D: Use VGGFace to determine the digital human facial feature vector.

[0049] The VGGFace model (Visual Geometry Group-Face) is a deep learning model specifically designed for face recognition and verification tasks. VGGFace uses the classic VGGNet network structure to handle face recognition problems, such as Figure 6 As shown in the figure, the network architecture has a total of 22 layers, including 13 convolutional layers, 5 pooling layers, 3 fully connected layers and the last Softmax classification layer. The convolution kernel size used in each convolutional layer is 3×3, the step size is 1, and the padding is 1. The nonlinearity is introduced in the input ReLU activation function to improve the expressiveness of the model; at specific positions in the convolutional layer sequence, there will be 5 maximum pooling layers (Max Pooling), the pooling window is 2x2, and the step size is 2, which is used to reduce the spatial dimension of the features and enhance the abstraction ability of the features. In the network architecture of VGGFace, the features output by the convolutional layer tend to focus on local features and texture details, and the fully connected layer integrates all the features extracted by the convolutional layer to form a more comprehensive and advanced feature representation. Therefore, the output of the second fully connected layer (Full Connection, FC) in VGGFace is used as the extracted facial features, that is, Figure 6 The second 1×1×4096 vector matrix in contains highly condensed and abstract facial information.

[0050] Further, the method of using VGGFace to determine the digital human facial feature vector includes: optimizing the triplet loss function according to the median of the Euclidean distance between the anchor image and the positive example image and the Euclidean distance between the anchor image and the negative example image.

[0051] Specifically, in order to further improve the accuracy of facial recognition, the present application uses a triplet loss function to optimize the training process of the VGGFace model. The triplet loss function strengthens the model's discrimination ability by minimizing the distance between facial images from the same person (positive examples) and the anchor image, while maximizing the distance between facial images from different people (negative examples) and the anchor image. Specifically, several frames are first extracted from the video of each digital person. These frames represent the appearance of the digital person at different time points, different expressions or different angles, increasing the model's ability to adapt to changes. A random one is selected from the extracted frames as an anchor image, which will be used as a benchmark for comparison. Afterwards, other frames from the same video as the anchor image are used as positive examples, and frames captured from videos of other digital people are used as negative examples, together with the anchor image to form multiple triplets. Such a data structure helps the model learn to distinguish different states of the same digital person from completely different digital people.

[0052] This application adjusts the model weights by optimizing the following triple loss function:

[0053] Chinese and Represents the Euclidean distance set between the anchor image and a series of positive and negative images, respectively, and is a hyperparameter used to adjust the minimum distance difference between positive and negative samples. It represents the median value in the extracted set. Considering that when there are multiple positive or negative examples, the median can effectively avoid the abnormal loss function caused by individual outliers, thereby providing a smoother and more consistent training signal. In this way, the VGGFace model can more effectively learn how to distinguish different faces, especially when the samples vary greatly, thereby improving the overall performance and reliability of the face verification system.

[0054] Step 220: Detect the voice signal of the digital human and determine the voice feature vector.

[0055] In one embodiment, the detecting the speech signal of the digital person and determining the speech feature vector includes: detecting the speech signal of the digital person using Mel-frequency cepstral coefficients and determining the speech feature vector.

[0056] The Mel frequency scale is an audio processing scale based on human auditory perception. It is used to measure the frequency of sound to make it more consistent with human auditory characteristics. The Mel scale was originally designed to bridge the difference between physical frequency (Hz) and human perceived frequency, and is widely used in speech recognition, music analysis, and other audio processing fields. The core principle of the Mel frequency scale is to convert actual frequency into Mel frequency. This conversion is based on the human ear's ability to perceive sounds of different frequencies. The human ear's perception of frequency is not linear, especially the perception of low-frequency sounds is more sensitive than high-frequency sounds. This perception curve shows that the increase in frequency perceived by the human ear is not proportional to the increase in actual frequency. The Mel frequency scale achieves the conversion from Hertz (Hz) to Mel (Mel) through the following formula:

[0057] Where f is the actual frequency (Hz), M( f ) corresponds to the Mel frequency. This formula reflects an important auditory phenomenon: as the frequency increases, the same frequency increment makes less and less difference in human auditory perception. In audio signal processing, the Mel scale is used to construct filter banks, which simulate the auditory characteristics of the human ear and are used to analyze the spectrum of audio signals. In this way, a feature called Mel-frequency cepstral coefficients (MFCCs) can be created to represent the basic quality and content of the sound.

[0058] Mel-frequency cepstral coefficients (MFCCs) are a form of feature representation used in speech recognition and audio processing, and are very effective in tasks such as speech recognition, speaker identification, and music information retrieval. MFCCs are designed to mimic the way the human ear processes sound to provide a compact feature representation that contains both pitch information and timbre information.

[0059] Figure 7 The figure shows the calculation process of Mel frequency cepstral coefficients. Figure 7 The modules in the figure represent related processing functions or operating devices. They include: 1. Pre-emphasize the output of the continuous speech 710 module The purpose of pre-emphasis 720 is to enhance the high frequency part, make the spectrum of the signal flat, keep it in the whole frequency band from low frequency to high frequency, and use the same signal-to-noise ratio to find the spectrum. At the same time, it is also to eliminate the effect of vocal cords and lips during the process, to compensate for the high frequency part of the speech signal suppressed by the pronunciation system, and to highlight the high frequency resonance peak. Pre-emphasize the speech signal through the high-pass filter:

[0060] In the formula, is the pre-emphasized signal, is the original signal, is the pre-emphasis coefficient.

[0061] 2. Framing and windowing Framing 730: Due to the non-stationary and short-term stationary characteristics of the speech signal, the speech signal is framed. One frame has N sampling points, such as the value of N is 256 or 512, and the time covered is about 20~30ms. In order to avoid excessive changes between two adjacent frames and smooth transitions, there is an overlapping area between two adjacent frames. This overlapping area contains M sampling points, and the value of M is usually about 1 / 2 or 1 / 3 of N.

[0062] Windowing 740: To mitigate spectral leakage, each frame is multiplied by a window function (such as a Hamming window) to reduce the discontinuity effect at the frame edge. The formula for the Hamming window is

[0063] 3. Fast Fourier Transform 750 Since it is usually difficult to see the characteristics of the signal in the time domain, it is usually converted into energy distribution in the frequency domain. Different energy distributions can represent the characteristics of different voices. Therefore, after multiplying the Hamming window, each frame must be subjected to a fast Fourier transform to obtain the energy distribution in the spectrum. The fast Fourier transform of each frame signal after framing and windowing is performed to obtain the spectrum of each frame:

[0064] 4. Mel filter bank 760 According to formula (5), the energy spectrum is converted to the Mel frequency scale through a set of Mel-scale triangular filter banks:

[0065] in is the position of the filter's center frequency in the Fourier spectrum.

[0066] 5. Logarithmic Operations 770 Output energy of each Mel-scale triangular filter is the square sum of the filtered signal, and the logarithmic expression is:

[0067] 6. Discrete Cosine Transform 780 Final logarithmic energy Perform discrete cosine transform to get MFCCs:

[0068] in is the nth cepstral coefficient, M is the number of filters. Through a series of processing steps, MFCCs can extract features that describe the basic properties of sound from complex audio signals. These features are extremely valuable for various audio analysis and processing tasks.

[0069] Step 230: Encode the facial feature vector and the voice feature vector to generate an identity identifier of the digital person.

[0070] In one embodiment, encoding the facial feature vector and the speech feature vector comprises: fusing the facial feature vector and the speech feature vector through low-rank multimodal features to generate fused features and encoding them.

[0071] Low-rank multimodal feature fusion is an efficient multimodal data fusion technology that simplifies the interaction representation between modalities through low-rank factors. By defining a set of low-rank weights for each modality, and then calculating the linear combination of these weights and the features of each modality, the fused multimodal representation is finally generated through the element-by-element product and accumulation of these combinations.

[0072] In one embodiment, Figure 8 As shown, the facial feature vectors are obtained according to the method described in the aforementioned steps 210 and 220. and the audio feature vector Later, assuming that there are low-rank factors, defining the first order of facial feature vectors and audio feature vectors The low-rank factors are and , then the fusion features The calculation formula is as follows:

[0073] in is a nonlinear transformation, is the outer product of the tensor.

[0074] The low-rank fusion method combines information from different modalities while reducing the parameters and computational complexity of the model. , which can balance the performance and efficiency of the model and maintain the effective capture of the complex interactions between modalities, making it particularly effective in dealing with tasks that require combining information from different sensors or data sources.

[0075] Preferably, the format of the identity identifier may be a digital string of a specified length, and the identity identifier is generated by sequentially expanding each dimension of the fusion feature and then performing a hash operation.

[0076] In one embodiment, after the facial feature vector and the speech feature vector are fused through low-rank multimodal features, the method includes: calculating a feature distance between the fused feature and a preset fused feature, and comparing the feature distance with a set threshold.

[0077] Specifically, preset fusion features have been saved in the digital human identity library. When the user enters new digital human information, the fusion features are generated after processing in steps 210, 220 and 230. The cosine similarity between the fusion feature and the saved preset fusion feature is calculated, that is, the fusion feature stored in the digital human identity library is calculated using the cosine similarity to determine the degree of matching. Cosine similarity measures the angular relationship between two vectors and can be used to compare multi-dimensional vectors. It is suitable for high-dimensional data generated in deep learning. The formula used for cosine similarity is as follows:

[0078] According to the set threshold, it is determined whether the digital person identity has been stored in the database. If the calculated cosine similarity does not exceed the set threshold, it means that the digital person identity has not been found and has not been stored in the database; otherwise, the corresponding digital person information with the largest cosine similarity is output.

[0079] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: the present application determines the facial feature vector and the voice feature vector of the digital person, and encodes the facial feature vector and the voice feature vector to generate the identity of the digital person. By fusing and encoding the facial features and voice features of the digital person, the most obvious features of the digital person are retained, thereby facilitating the confirmation, re-identification, and comparison of the digital person; in addition, the search query and identification verification of the digital person can be realized through the identity.

[0080] Fig. 9 A digital human identity retrieval flow chart provided in an embodiment of the present application includes: Step 910: input the video of the digital human; Step 911: extract video frames and use MTCNN to perform face detection and positioning; Step 912: Use VGGFace to extract the facial feature vector of the digital human; Step 921: extracting audio data of the video; Step 922, resampling, denoising preprocessing; Step 923: advance the digital human audio feature vector using the Mel frequency scale; Step 930: extracting low-rank multimodal fusion features; Step 940: Calculate the feature distance between the fused feature and the fused feature stored in the digital human identity database; In one embodiment, for the digital human video input into the retrieval system, the low-rank fusion features of the digital human to be retrieved are finally extracted according to the method described in steps 210 and 220, and then the fusion features stored in the digital human identity library are matched using cosine similarity. Cosine similarity measures the angular relationship between two vectors, can be used to compare multi-dimensional vectors, and is suitable for high-dimensional data generated in deep learning.

[0081] The cosine similarity measure can be calculated using formula (13).

[0082] Finally, based on the set threshold, it is determined whether the digital person identity has been stored in the database. If the calculated cosine similarities do not exceed the set threshold, it means that the digital person identity has not been found and has not been stored in the database; otherwise, the corresponding digital person information with the largest matched cosine similarity is output.

[0083] Step 950, determine whether it is greater than a set threshold: if so, execute step 960; otherwise, execute step 970; Step 960: output the found digital person identity information and terminate the execution; Step 970: Not found, terminate execution.

[0084] Fig.10A digital human identity library framework structure diagram provided in an embodiment of the present application. The modules in the figure represent corresponding operating devices for data processing, and the modules are marked with processing result data, including: digital human facial image 1010, digital human facial feature vector 1011, digital human audio data 1020, digital human audio feature vector 1021, low-rank multimodal fusion feature 1030, digital human identity 1040 and digital human additional identity information 1050.

[0085] In one embodiment, the digital human facial image 1010 is analyzed according to step 210 to obtain a digital human facial feature vector 1011 .

[0086] In one embodiment, the digital human audio data 1020 is analyzed according to step 220 to obtain a digital human audio feature vector 1021 .

[0087] In one embodiment, the digital human facial feature vector 1011 and the digital human audio feature vector 1021 in the aforementioned embodiment are processed according to step 230 to obtain a low-rank multimodal fusion feature 1030, which is encoded to generate a digital human identity 1040, and then the digital human additional identity information 1050 and the digital human identity 1040 are stored in a database.

[0088] Fig.11 A block diagram of a digital human identity authentication and retrieval system provided in an embodiment of the present application, wherein the digital human identity authentication and retrieval system 1100 includes: a digital human identity library 1110 and a digital human identity retrieval system 1120.

[0089] In one embodiment, the digital human identity library 1110 uses Fig.10 The framework structure diagram of the digital human identity library is shown.

[0090] The digital human identity retrieval system 1120 includes a user terminal, a digital human management system server, and a network connecting the user terminal and the digital human management system server. After the user terminal inputs query information (such as a digital human identity, a digital human facial image, or a digital human audio data), the input query information is transmitted to the digital human management system server through the network. The server is configured with the digital human identity generation method provided in the embodiment of the present application, and the server retrieves and returns the search results to the user terminal.

[0091] Fig.12 A digital human identity generation device provided in an embodiment of the present application is used to implement the method described in the first aspect of the present application, including: a facial feature processing module 1210, a voice feature processing module 1220 and an encoding module 1230. The identity generation device includes processing software and a software running module, which is loaded on Figure 1In any client or any server as shown.

[0092] The facial feature processing module 1210 is used to detect the video and / or picture of the digital human and determine the facial feature vector of the digital human; the functions implemented are as described in step 210 of the embodiment of the present application.

[0093] The speech feature processing module 1220 is used to detect the speech signal of the digital human and determine the speech feature vector; the functions implemented are as described in step 220 of the embodiment of the present application.

[0094] The encoding module 1230 is used to encode the facial feature vector and the voice feature vector to generate the identity of the digital person. The function implemented is as described in step 230 of the embodiment of the present application.

[0095] In one embodiment, the facial feature processing module 1210 is further used to: use a multi-task convolutional neural network to detect images and / or the pictures in the digital human video, as described in steps 210A~C, and use VGGFace to determine the digital human facial feature vector, as described in step 210D.

[0096] In one embodiment, the speech feature processing module 1220 is further used to: detect the speech signal of the digital human using Mel-frequency cepstral coefficients to determine a speech feature vector.

[0097] In one embodiment, the encoding module 1230 is further used to: encode the facial feature vector and the speech feature vector by low-rank multimodal feature fusion.

[0098] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] Therefore, the present application also proposes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in any embodiment of the first aspect of the present application is implemented.

[0100] Furthermore, in order to generate the device described in the embodiment of the second aspect of this application, the present application also proposes an electronic device (or computing device), comprising a processor and a memory storing program instructions, wherein the processor is configured to implement the method described in the embodiment of the first aspect of this application when running the program instructions.

[0101] Fig.13 The electronic device 1300 shown in the figure is only an example and should not bring any limitation to the functions and application scope of the embodiment of the present application.

[0102] It includes: one or more processors 1320; a storage device 1310, used to store one or more programs. When the one or more programs are run by the one or more processors 1320, the one or more processors 1320 implement the digital human identity generation method provided in the embodiments of the present application, and the method includes at least one step described in each embodiment of the present application.

[0103] The number of processors 1320 in the electronic device may be one or more. Fig.13 A processor 1320 is taken as an example; the processor 1320, storage device 1310, input device 1330 and output device 1340 in the electronic device can be connected via a bus or other methods, and the figure takes the connection via bus 1350 as an example.

[0104] The storage device 1310, as a computer-readable storage medium, can be used to store software programs, computer executable programs, and module units, such as program instructions corresponding to the digital human identity generation method in the embodiment of the present application. The storage device 1310 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the terminal, etc. In addition, the storage device 1310 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the storage device 1310 may further include a memory remotely arranged relative to the processor 1320, and these remote memories may be connected via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0105] It can be understood that the processor 1320 of this embodiment includes at least one module among the facial feature processing module 1210, the voice feature processing module 1220 and the encoding module 1230 of the embodiment of the present application, which is used to implement at least one part of the functions of the above-mentioned embodiment, and can be an entity composed of one or more components. The specific functions of each module will not be repeated here.

[0106] The input device 1330 may be used to receive input numbers, character information or voice information, and to generate key signal inputs related to user settings and function controls of the electronic device.

[0107] The output device 1340 may include electronic devices such as a display screen and a speaker.

[0108] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0109] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for generating a digital human identity, characterized in that: The following steps are involved: Detecting the video and / or picture of the digital person and determining the facial feature vector of the digital person; Detect the voice signal of the digital person and determine the voice feature vector; The facial feature vector and the voice feature vector are encoded to generate the identity of the digital person.

2. The method for generating a digital human identity according to claim 1, characterized in that: The detecting the video and / or picture of the digital human and determining the facial feature vector of the digital human comprises: using a multi-task convolutional neural network to detect the image in the video of the digital human and / or the picture, and using VGGFace to determine the facial feature vector of the digital human.

3. The method for generating a digital human identity according to claim 2, characterized in that: The method of using VGGFace to determine the digital human facial feature vector includes: optimizing the triplet loss function according to the median of the Euclidean distance between the anchor image and the positive example image and the Euclidean distance between the anchor image and the negative example image.

4. The method for generating a digital human identity according to claim 1, characterized in that: Mel-frequency cepstral coefficients are used to detect the speech signal of the digital human and determine the speech feature vector.

5. The method for generating a digital human identity according to claim 1, characterized in that: The encoding of the facial feature vector and the speech feature vector includes: fusing the facial feature vector and the speech feature vector through low-rank multimodal features to generate fused features and encoding them.

6. The method for generating a digital human identity according to claim 5, characterized in that: After the facial feature vector and the speech feature vector are fused through low-rank multimodal features, the method includes: calculating a feature distance between the fused feature and a preset fused feature, and comparing the feature distance with a set threshold.

7. A digital human identity generation device, used to implement the digital human identity generation method according to any one of claims 1 to 6, characterized in that: include: Facial feature processing module, speech feature processing module and encoding module; The facial feature processing module is used to detect the video and / or picture of the digital human and determine the facial feature vector of the digital human; The speech feature processing module is used to detect the speech signal of the digital person and determine the speech feature vector; The encoding module is used to encode the facial feature vector and the voice feature vector to generate the identity of the digital person.

8. A digital human identity generation system, characterized in that: Including client and server; The client is used to provide digital human videos and / or pictures to the server; The server is used to receive the digital human video and / or picture provided by the client, and implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Large-scale highly-efficient face recognition method based on Hamming distance

    CN106778526A

  • Unique Identity Recognition and Control Method of Digital Human Artificial Intelligence

    CN109063627A

  • Face analysis method and system combining multi-task and multi-scale convolution neural network

    CN109508654A

  • Facial image identity derecognition method and device and electronic equipment

    CN112101087A

  • Double-record quality inspection method and device, server and storage medium

    CN112328999A