Digital human identity authentication method, device and system

The generation of digital human identity identifiers through the fusion of multimodal information and tensor pooling technology has solved the difficulty of digital human identity authentication, improved the security and reliability of identity authentication, and realized effective authentication and retrieval of digital human identity.

CN119004428APending Publication Date: 2024-11-22BEIJING MUNICIPAL PUBLIC SECURITY BUREAU
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411492520.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-24
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The lack of standardized dissemination, application and management methods for digital person identity authentication, which leads to difficulty in identity authentication.

Method used

By obtaining multimodal information of digital people, including image features, audio features and semantic features, multimodal tensor pooling fusion generates fusion features and encodes them, generates identity identifiers, and uses pre-generated digital people identity identifiers to generate them Compare to determine whether it has passed the certification.

Benefits of technology

It improves the security, reliability and stability of digital person identity authentication, ensures the security of interaction between digital person generation users, application users and third parties, and realizes the search query of digital persons.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119004428A_ABST
    Figure CN119004428A_ABST
Patent Text Reader

Abstract

The invention discloses a digital person identity authentication method. The method comprises the following steps: acquiring multi-modal information of a digital person; through multi-modal tensor pooling fusion, generating fusion features and encoding the fusion features, and generating an identity label of the digital person; and comparing the identity label with a pre-generated digital human identity label, and determining whether authentication is passed or not. The invention further comprises a digital human identity authentication device and system for implementing the method. According to the technical scheme, multi-modal information is fused to realize digital human identity authentication, security and secrecy of interaction among a digital human generation user, a digital human application user and a third party are guaranteed, and security, reliability and stability of digital human identity authentication are improved; in addition, retrieval and query of the digital person can be realized through the identity label.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method, device and system for digital human identity authentication. Background Art

[0002] With the development of artificial intelligence and digital technologies, digital humans have appeared in various industries, aroused widespread attention, and become an investment hotspot. As an important application that carries multimodality after GPT4, digital humans are considered to be the entrance to the metaverse and an application direction with broad development potential.

[0003] Digital human is an artificial being driven by a computer program with its own consciousness and identity. It can show the same physiological, psychological and social performance as real humans in the digital world. Compared with traditional image animation, digital human technology can achieve more complex animation effects, including verbal dialogue, gestures, expression changes, body language, etc. Its technical goal is to create a digital image close to the human image through computer graphics technology, and give it a specific character identity setting, visually shorten the psychological distance with humans, and bring more realistic emotional interaction to humans.

[0004] At present, digital human theory and technology are becoming more and more mature, the scope of application is constantly expanding, the industry is gradually forming and enriching, and the corresponding business model is also continuously evolving and diversifying. Digital human technology combines actual application scenarios and fields, cuts into various types, forms industry application solutions, and empowers the industry. According to different application scenarios or industries, there have been entertainment digital humans such as virtual anchors and virtual idols; educational digital humans such as virtual teachers; assistant digital humans such as virtual customer service, virtual tour guides, and smart assistants; film and television digital humans such as stand-ins or virtual actors, etc. Virtual digital humans with different appearances and functions empower film and television, media, games, finance, culture and tourism, and provide customized services to users according to their needs.

[0005] As an emerging technology field, digital humans still lack standardized means of dissemination, application and management. How to authenticate the identity of digital humans is a technical problem that this application aims to solve. Summary of the invention

[0006] The digital human identity authentication method, device and system proposed in this application combine multimodal information such as digital human video, pictures, audio, application scenarios, user roles, etc., to solve the problem of difficulty in digital human identity authentication.

[0007] In a first aspect, an embodiment of the present application provides a method for digital human identity authentication, comprising the following steps: Obtain multimodal information of digital humans; By multi-modal tensor pooling fusion, fusion features are generated and encoded to generate the identity of the digital person; The identity is compared with a pre-generated digital human identity to determine whether the authentication is successful.

[0008] Preferably, the multimodal information includes multiple types of the following information: image features, audio features, semantic features; the source of the digital human's multimodal information includes one or more of the following: digital human electronic information, digital human generation tools, identity data, authorization code, application scenario data.

[0009] Preferably, at least a part of the multimodal information is obtained from an authorization code, and the authorization code is set by at least one of the following clients: a digital human generation user client, a digital human application user client, and a third-party client.

[0010] Preferably, at least a part of the multimodal information is obtained from application scenario data, and the application scenario data is set by at least one of the following clients: a digital human generation user client, a digital human application user client, and a third-party client.

[0011] Preferably, at least a portion of the multimodal information is obtained from identity data, and the identity data includes at least one of the following: identity data of one or more digital human-generated users, identity data of one or more digital human-applied users, and identity data of one or more third parties.

[0012] Preferably, the identity identifier is generated by sequentially expanding each dimension of the fusion feature and then performing a hash operation.

[0013] In a second aspect, an embodiment of the present application further provides a digital human identity authentication device, which is used to implement the digital human identity authentication method described in any one of the first aspects, including: an acquisition module, a generation module and a determination module.

[0014] The acquisition module is used to obtain the multimodal information of the digital human.

[0015] The generation module is used to generate and encode fusion features through multi-modal tensor pooling fusion to generate the identity of the digital person.

[0016] The determination module is used to compare the identity with a pre-generated digital human identity to determine whether the authentication is passed.

[0017] In a third aspect, an embodiment of the present application further provides a digital human identity authentication system for implementing any of the digital human identity authentication methods described in the first aspect, comprising: multiple clients and at least one server.

[0018] Any one of the clients is used for at least one of the following functions: generating, identifying or editing the electronic data of the digital person.

[0019] Any one of the servers is used to generate the identity of the digital person.

[0020] Any of the servers is used to compare the identity with a pre-generated digital human identity to determine whether the authentication is successful.

[0021] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the method described in any embodiment provided in the present application is implemented.

[0022] In a fifth aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in any embodiment provided in the present application is implemented.

[0023] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: the present application obtains the multimodal information of the digital person, generates and encodes the fusion features through multimodal tensor pooling fusion, generates the identity of the digital person, compares the identity with the pre-generated digital person identity, and determines whether it is authenticated. The fusion of multimodal information realizes the identity authentication of the digital person, ensures the security and confidentiality of the interaction between the digital person generating user, the digital person application user and the third party, and improves the security, reliability and stability of the digital person identity authentication; in addition, the retrieval query of the digital person can be realized through the identity. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A schematic diagram of a digital human management system and client provided in an embodiment of the present application; Figure 2 A flow chart of a digital human identity authentication method provided in an embodiment of the present application; Figure 3 A multi-task convolutional neural network provided in an embodiment of the present application; Figure 4 A VGGface network structure block diagram provided in an embodiment of the present application; Figure 5 A Mel-frequency cepstral coefficient calculation process provided in an embodiment of the present application; Figure 6 A block diagram of a digital human identity authentication system provided in an embodiment of the present application; Figure 7A schematic diagram of a digital human identity authentication device is provided for an embodiment of the present application; Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0026] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0027] Figure 1 A schematic diagram of a digital human management system and client provided in an embodiment of the present application. Digital human generation users, application users, audit users, authentication users, etc. interact with the digital human identity authentication system through the client to complete digital human creation, use, audit and authentication. In the digital human identity management system, multiple servers are deployed, including: digital human generation server 110, digital human application server 120, digital human detection server 130, authentication server 140 and digital human identity identification server 150. Through the interaction between the above-mentioned client and server, the digital human service needs of different users are met.

[0028] Preferably, the digital human authentication server can access the digital human identity server to retrieve the specified digital human identity.

[0029] In one embodiment of the present application, when the user of the client is a digital human generating user, the client 111 is used to provide the digital human video and / or picture to the server, including the digital human generating server. In other embodiments of the present application, when the user of the client is a digital human using user, the client 121 is used to read the digital human video and / or picture from the server, including the application digital human server; when the user of the client is an auditing user, the client 131 is used to read the digital human video and / or picture from the server, including the application digital human server, and further, identify the digital human video and / or picture by detecting the digital human server, and further, implement the editing function of the server permission on the digital human. The detection digital human server is used to detect according to the multimodal information source and obtain multimodal information. It can be understood that when the client 111, 121, 131, 141 supported by the detection digital human server issues a detection instruction, the detection is performed according to the indicated multimodal information source to obtain multimodal information.

[0030] In one embodiment of the present application, any server receives the digital human video and / or picture provided by the client to perform identity recognition of the digital human; in particular, the digital human generation server 110 can be used to generate the digital human and store it in a set storage device; the digital human authentication server 140 can receive and detect the fusion features provided by the digital human server, generate the identity of the digital human in real time, and compare the identity of the digital human generated in real time with the identity mark of the digital human stored in the digital human identity server 150 to determine whether it has the right to further process or apply the digital human.

[0031] Figure 2 A flowchart of a digital human identity authentication method provided in an embodiment of the present application includes steps 210 to 230: Step 210: Obtain multimodal information of the digital human.

[0032] Multimodal refers to the use of multiple sensory modes (such as vision, hearing, touch, etc.) to convey information or communicate. In the field of computer science and human-computer interaction, multimodality usually refers to the use of multiple different input and output methods to achieve human-computer interaction. For example, a multimodal application can use multiple input methods such as voice, gestures, and touch screens at the same time, and use multiple output methods such as audio, images, and text to interact with users.

[0033] In the present application, the multimodal information includes multiple types of the following information: image features, audio features, and semantic features; the source of the multimodal information of the digital human includes one or more of the following: digital human electronic information, digital human generation tools, identity data, authorization code, and application scenario data.

[0034] Among them, the electronic information of digital humans includes but is not limited to at least one of the following: digital human videos, digital human pictures, digital human audio, and other electronic data describing the characteristics of digital humans.

[0035] In one embodiment of the present application, the multimodal information includes digital human facial features. The video and / or picture of the digital human is detected to determine the digital human facial feature vector. Preferably, it includes: using a multi-task convolutional neural network to detect the picture in the digital human video and / or the picture, and using VGGFace to determine the digital human facial feature vector.

[0036] In one embodiment of the present application, at least a portion of the multimodal information is obtained from an authorization code, and the authorization code is set by at least one of the following clients: a digital human generation user client, a digital human application user client, and a third-party client.

[0037] In one embodiment of the present application, at least a portion of the multimodal information is obtained from application scenario data, and the application scenario data is set by at least one of the following clients: a digital human generation user client, a digital human application user client, and a third-party client. Typical application scenarios include: news propaganda, entertainment, and commercial advertising.

[0038] In one embodiment of the present application, at least a portion of the multimodal information is obtained from identity data, and the identity data includes at least one of the following: one or more digital human generated user identity data, one or more digital human application user identity data, and one or more third-party identity data.

[0039] In one embodiment of the present application, at least a portion of the multimodal information is obtained from a digital human generation tool, such as a digital human generation algorithm, digital human facial model parameters, and digital human voice model parameters.

[0040] In one embodiment of the present application, a digital human generation tool can be obtained by detecting the digital human electronic information. For example, the digital human generation algorithm can be determined by obtaining the digital human configuration file. For another example, the digital human generation algorithm can be determined by analyzing the digital human facial features.

[0041] Step 220: Generate fusion features through multi-modal tensor pooling and encode them to generate the identity of the digital person.

[0042] For example, the facial feature vector and the voice feature vector are fused through multimodal tensor pooling to generate fused features and encode them to generate the identity of the digital person.

[0043] In one embodiment, the facial feature vector is obtained and the audio feature vector After that, first concatenate them together, and construct the polynomial feature tensor by performing a P-order tensor product on the concatenated feature vectors , define a pooling weight tensor W, which is used to control the influence of different high-order terms. Each pooling result Calculated by the following formula:

[0044] The original input of this formula is the concatenation of the feature vectors of the digital human face and audio modalities. The multi-layer linear layer fusion can dynamically evaluate the expressiveness of the two modalities in the digital human identity identification during the training process. That is, when each digital human is used for authentication, the face may be more recognizable than the audio. In this case, the multi-layer linear layer will highlight and pay more attention to the facial features in the feature fusion, and vice versa. This method can dynamically balance the importance of the two modal information for digital human identity recognition through the weight tensor, and has better robustness for the situation where one of the facial or audio modalities is missing or has less information.

[0045] Preferably, the format of the identity identifier can be a digital string of a specified length, and the identity identifier is generated by sequentially expanding each dimension of the fusion feature and performing a hash operation. Specific hash algorithms can use SHA-256, SHA-512, etc.

[0046] Preferably, the format of the identity identifier can be a digital string of a specified length, and the identity identifier is generated by sequentially expanding each dimension of the fusion feature and performing a hash operation on some dimensions. Specific hash algorithms can use SHA-256, SHA-512, etc.

[0047] In one embodiment, other sources of multimodal information may include: digital human electronic data, digital human image generation algorithm (tool features), generating user identity features, regulatory role, and may also include application scenario features, authorization code features, etc., where the authorization code may come from the generating user, the using user, or a third party, for example, the third party is a digital human regulatory authority.

[0048] Specifically, select the facial feature vector and the audio feature vector , and other sources of the multimodal information to obtain feature data, such as application scenario features, after splicing the three parts of information, a new polynomial feature tensor is constructed by performing a P-order tensor product on the spliced ​​feature vectors , define a pooling weight tensor W 2 , which is used to control the influence of different high-order terms. Formula (1) is used to calculate the fusion feature and encode it to generate the identity of the digital person.

[0049] In one embodiment of the present application, the identity identifier is generated by sequentially expanding each dimension of the fusion feature and performing a hash operation.

[0050] Any one or more of the other sources of the above multimodal information, the original information or the video features, audio features and / or semantic features generated by artificial intelligence extraction, can be combined with the selected facial feature vector and the audio feature vector The splicing is performed, and the fusion features are calculated by determining the tensor weights, and the encoding is performed to generate the identity of the digital person.

[0051] Step 230: Compare the identity with the pre-generated digital human identity to determine whether the authentication is successful.

[0052] In one embodiment, comparing the identity with a pre-generated digital human identity to determine whether the authentication is passed includes: calculating the feature distance between the fusion feature corresponding to the identity and the preset fusion feature corresponding to the pre-generated digital human identity, and comparing the feature distance with a set threshold.

[0053] Specifically, in the digital human identity database, pre-generated digital human identity identifiers and preset fusion features have been saved. When the user inputs new digital human information, after processing in steps 210 and 220, the fusion features corresponding to the digital human identity identifier are generated. The cosine similarity between the fusion feature and the preset fusion feature corresponding to the saved digital human identity identifier is calculated, that is, the fusion features stored in the digital human identity database are calculated using the cosine similarity to determine the degree of matching. Cosine similarity measures the angular relationship between two vectors and can be used to compare multi-dimensional vectors. It is suitable for high-dimensional data generated in deep learning.

[0054] According to the set threshold, it is determined whether the digital human identity has been stored in the database. If the calculated cosine similarity exceeds the set threshold, it means that a pre-generated digital human identity similar to or identical to the identity identifier is found and the authentication is successful; otherwise, the authentication fails.

[0055] In one embodiment, a designated role is associated with the pre-generated digital human identity, such as a designated supervisor, to access the pre-generated digital human identity and its related information. During authentication, after processing in steps 210 and 220, when the identity is compared with the pre-generated digital human identity in step 230, in addition to comparing the feature distance between the fused features (the specific comparison method includes: calculating the feature distance between the fused feature corresponding to the identity and the preset fused feature corresponding to the pre-generated digital human identity, and comparing the feature distance with the set threshold), the user identity of the client is also compared with the designated role associated with the pre-generated digital human identity to see if they are consistent. If they are inconsistent, the authentication fails; if they are consistent, the authentication is judged based on the comparison of the feature distance with the preset threshold to see if the authentication is passed.

[0056] In one embodiment, the facial feature vector is selected and the audio feature vector , and feature information from other sources of the multimodal information, such as the identity feature of the generating user, after splicing the three parts of information, a new polynomial feature tensor weight is constructed by performing a P-order tensor product on the spliced ​​feature vector, and the fusion feature is calculated. Through the method, a digital person B can only be created and edited by a designated generating user, while other generating users cannot create, edit or use it, so other people cannot obtain authentication, or distinguish the identity of the generating user during authentication and perform authorization after authentication.

[0057] In one embodiment, the facial feature vector is selected and the audio feature vector , and feature information from other sources of the multimodal information, such as video, audio or semantic feature vectors of application scenario description data; the method described in this embodiment will limit a digital person A to be used only in specific application scenarios, such as news broadcasting, and cannot be authenticated in other application scenarios.

[0058] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: the present application obtains the multimodal information of the digital person, generates and encodes the fusion features through multimodal tensor pooling fusion, generates the identity of the digital person, compares the identity with the pre-generated digital person identity, and determines whether the authentication is passed. The fusion of multimodal information is used to realize the identity authentication of the digital person, which ensures the security of the interaction between the digital person generating user, the digital person application user and the third party, and improves the security, reliability and stability of the digital person identity authentication; in addition, the retrieval query of the digital person can be realized through the identity.

[0059] Figure 3Represents a multi-task convolutional neural network. The multi-task convolutional neural network (MTCNN) algorithm is a face recognition and alignment technology based on deep learning. To implement step 210, it can perform face recognition and alignment tasks simultaneously. The MTCNN algorithm consists of three sub-networks: a proposal network (P-Net) 310, a refinement network (R-Net) 320, and an output network (O-Net) 330. These three networks process faces in a coarse to fine order.

[0060] It should be noted that in order to obtain the video features in the multimodal information of the digital human, any sub-network of P-Net, R-Net and O-Net can be used for calculation. It is understandable that when only the P-Net network is used, non-face images are preliminarily screened out, and the range of faces obtained may be misjudged, but the processing speed is faster. It is also understandable that the use of the R-Net network to further determine the face probability improves the processing accuracy. However, the processing speed is slow when only R-Net is used. Furthermore, it is also understandable that O-Net can provide accurate recognition of key facial features, but the processing speed is the slowest when only the O-Net network is used. Therefore, the combination of any two of the three can simultaneously improve the processing speed and processing accuracy.

[0061] Before using these three sub-networks, the original images are scaled to different scales, and then the images of different scales are sent to these three sub-networks for training. The purpose is to detect faces of different sizes, thereby realizing multi-scale object detection.

[0062] The P-Net network 310 detects and locates the face and generates candidate face regions and bounding boxes. The proposal network P-Net, full name Proposal Network, is the first sub-network in the MTCNN model. P-Net is a fully convolutional network (FCN) that consists of a series of convolutional layers without fully connected layers. The advantage of fully convolutional networks is that they can accept input images of any size and produce outputs of corresponding sizes, and can handle faces of various sizes. P-Net works by sliding on the input image, and each slide generates a candidate window. This window is a small area that may contain a face. The size and step size of the sliding window can be adjusted to control the accuracy of the detection and the complexity of the calculation. Usually, the network runs at multiple scales so that faces of different sizes can be detected. The main task of P-Net is to perform rough face detection and bounding box regression. For each candidate window, the network outputs a binary face classification (face / not face). P-Net is the first sub-network in the MTCNN algorithm, and its main task is to generate candidate windows and bounding box regression vectors.

[0063] R-Net network 320 classifies faces. R-Net, the full name of Refine Network, is the second stage of the Multi-Task Cascade Convolutional Neural Network (MTCNN) model. After P-Net generates a series of candidate windows, the job of R-Net is to further screen these windows. After the input candidate window of R-Net undergoes operations such as convolution, pooling, and full connection, it outputs three results: one is the face classification result, one is the bounding box regression value, and the other is the alignment of face key points. The face classification result is a two-element vector indicating the probability of whether there is a face in the window. The bounding box regression value is a four-element vector used to fine-tune the bounding box position and size of the face to locate the face more accurately.

[0064] R-Net is a deeper convolutional neural network, which aims to eliminate a large number of non-face windows to reduce the computational burden of subsequent stages and make more detailed adjustments to the remaining candidate windows. Through the above process, R-Net can not only improve the accuracy of face detection, but also effectively reduce the amount of calculation and improve the efficiency of the entire face detection process.

[0065] The O-Net network 330 determines the location of facial key points. Output O-Net, full name Output Network, is the third network of the MTCNN model. The main task of O-Net is to perform final face detection, bounding box regression, and positioning of five facial points (two eyes, nose, and two mouth corners). It receives the candidate window from R-Net, and then uses a deeper convolutional neural network for more accurate face detection and bounding box regression. In addition, O-Net also outputs the coordinates of the five facial points through regression, which is very important for subsequent tasks such as face alignment and recognition. The five key points correspond to the center of the left eye, the center of the right eye, the tip of the nose, the left corner of the mouth, and the right corner of the mouth.

[0066] Optimally, the MTCNN model uses three cascaded networks, P-Net, R-Net, and O-Net, each of which is responsible for different tasks, and at each stage, the candidate regions are filtered and refined, thereby improving the accuracy and efficiency of face detection. P-Net is the first stage of MTCNN, which uses a smaller convolutional neural network to quickly scan the entire image and generate candidate face regions and bounding boxes. Since the network structure of P-Net is relatively simple, it runs the fastest. R-Net is the second stage of MTCNN, which receives the candidate regions from P-Net and further filters and refines them. R-Net uses a larger convolutional neural network for more accurate face detection and bounding box regression, so it runs slightly slower than P-Net. O-Net is the third and final stage of MTCNN, which receives the candidate regions from R-Net and performs the final face detection and bounding box regression. In addition, O-Net is also responsible for outputting the locations of the five key points of the face. O-Net uses a deeper convolutional neural network for face detection and key point localization, so it runs the slowest.

[0067] Through this cascade method, the MTCNN model can reduce the number of candidate regions that need to be processed at each stage, thereby effectively reducing the calculation time while ensuring the accuracy of face detection. This is also the main reason why MTCNN can achieve high-precision and high-efficiency face detection.

[0068] Figure 4 A VGGface network structure block diagram provided in an embodiment of the present application uses VGGFace to determine the facial feature vector of a digital human.

[0069] The VGGFace model (Visual Geometry Group-Face) is a deep learning model specifically designed for face recognition and verification tasks. VGGFace uses the classic VGGNet network structure to handle face recognition problems, for example Figure 4As shown in the figure, the network architecture has a total of 22 layers, including 13 convolutional layers, 5 pooling layers, 3 fully connected layers and the last Softmax classification layer. The convolution kernel size used in each convolutional layer is 3×3, the step size is 1, and the padding is 1. The nonlinearity is introduced in the input ReLU activation function to improve the expressiveness of the model; at specific positions in the convolutional layer sequence, there will be 5 maximum pooling layers (Max Pooling), the pooling window is 2x2, and the step size is 2, which is used to reduce the spatial dimension of the features and enhance the abstraction ability of the features. In the network architecture of VGGFace, the features output by the convolutional layer tend to focus on local features and texture details, and the fully connected layer integrates all the features extracted by the convolutional layer to form a more comprehensive and advanced feature representation. Therefore, the output of the second fully connected layer (Full Connection, FC) in VGGFace is used as the extracted facial features, that is, Figure 4 The second 1×1×4096 vector matrix in contains highly condensed and abstract facial information.

[0070] Further, the method of using VGGFace to determine the digital human facial feature vector includes: optimizing the triplet loss function according to the median of the Euclidean distance between the anchor image and the positive example image and the Euclidean distance between the anchor image and the negative example image.

[0071] Specifically, in order to further improve the accuracy of facial recognition, the present application uses a triplet loss function to optimize the training process of the VGGFace model. The triplet loss function strengthens the model's discrimination ability by minimizing the distance between facial images from the same person (positive examples) and the anchor image, while maximizing the distance between facial images from different people (negative examples) and the anchor image. Specifically, several frames are first extracted from the video of each digital person. These frames represent the appearance of the digital person at different time points, different expressions or different angles, increasing the model's ability to adapt to changes. A random one is selected from the extracted frames as an anchor image, which will be used as a benchmark for comparison. Afterwards, other frames from the same video as the anchor image are used as positive examples, and frames captured from videos of other digital people are used as negative examples, together with the anchor image to form multiple triplets. Such a data structure helps the model learn to distinguish different states of the same digital person from completely different digital people.

[0072] This application adjusts the model weights by optimizing the following triple loss function:

[0073] Chinese and Represents the Euclidean distance set between the anchor image and a series of positive and negative images, respectively, and is a hyperparameter used to adjust the minimum distance difference between positive and negative samples. It represents the median value in the extracted set. Considering that when there are multiple positive or negative examples, the median can effectively avoid the abnormal loss function caused by individual outliers, thereby providing a smoother and more consistent training signal. In this way, the VGGFace model can more effectively learn how to distinguish different faces, especially when the samples vary greatly, thereby improving the overall performance and reliability of the face verification system.

[0074] The multi-task convolutional neural network is used to detect the pictures in the digital human video and / or the pictures, and the facial feature vector of the digital human is determined using VGGFace.

[0075] In one embodiment of the present application, the multimodal information includes speech features. The speech signal of the digital person is detected to determine the speech feature vector. Preferably, the method includes: using Mel frequency cepstral coefficients to detect the speech signal of the digital person to determine the speech feature vector.

[0076] Figure 5 A Mel-frequency cepstral coefficient calculation process provided in an embodiment of the present application detects a digital human's voice signal and determines a voice feature vector.

[0077] The Mel frequency scale is an audio processing scale based on human auditory perception. It is used to measure the frequency of sound to make it more consistent with human auditory characteristics. The Mel scale was originally designed to bridge the difference between physical frequency (Hz) and human perceived frequency, and is widely used in speech recognition, music analysis, and other audio processing fields. The core principle of the Mel frequency scale is to convert actual frequency into Mel frequency. This conversion is based on the human ear's ability to perceive sounds of different frequencies. The human ear's perception of frequency is not linear, especially the perception of low-frequency sounds is more sensitive than high-frequency sounds. This perception curve shows that the increase in frequency perceived by the human ear is not proportional to the increase in actual frequency. The Mel frequency scale achieves the conversion from Hertz (Hz) to Mel (Mel) through the following formula:

[0078] Where f is the actual frequency (Hz), M( f ) corresponds to the Mel frequency. This formula reflects an important auditory phenomenon: as the frequency increases, the same frequency increment makes less and less difference in human auditory perception. In audio signal processing, the Mel scale is used to construct filter banks, which simulate the auditory characteristics of the human ear and are used to analyze the spectrum of audio signals. In this way, a feature called Mel-frequency cepstral coefficients (MFCCs) can be created to represent the basic quality and content of the sound.

[0079] Mel-frequency cepstral coefficients (MFCCs) are a form of feature representation used in speech recognition and audio processing, and are very effective in tasks such as speech recognition, speaker identification, and music information retrieval. MFCCs are designed to mimic the way the human ear processes sound to provide a compact feature representation that contains both pitch information and timbre information.

[0080] Figure 5 The modules in the figure represent related processing functions or operating devices. They include: (1) Pre-emphasize the continuous speech 510 signal The purpose of pre-emphasis 520 is to enhance the high frequency part, make the spectrum of the signal flat, keep it in the whole frequency band from low frequency to high frequency, and use the same signal-to-noise ratio to find the spectrum. At the same time, it is also to eliminate the effect of the vocal cords and lips during the process, to compensate for the high frequency part of the speech signal suppressed by the pronunciation system, and to highlight the high frequency resonance peak. The speech signal is pre-emphasized through a high-pass filter.

[0081] (2) Framing and windowing Frame division 530: Due to the non-stationary and short-term stationary characteristics of the speech signal, the speech signal is divided into frames. One frame has N sampling points. If the value of N is 256 or 512, the time covered is about 20~30ms. In order to avoid excessive changes between two adjacent frames and smooth transitions, there will be an overlapping area between two adjacent frames. This overlapping area contains M sampling points. Usually the value of M is about 1 / 2 or 1 / 3 of N.

[0082] Windowing 540: To mitigate spectral leakage, each frame is multiplied by a window function (such as a Hamming window) to reduce the discontinuity effect at the frame edge.

[0083] (3) Fast Fourier Transform 550 Since it is usually difficult to see the characteristics of the signal in the time domain, it is usually converted into energy distribution in the frequency domain. Different energy distributions can represent the characteristics of different speech. Therefore, after multiplying the Hamming window, each frame must be subjected to a fast Fourier transform to obtain the energy distribution in the spectrum.

[0084] (4) Mel filter bank 560 The energy spectrum is converted to the Mel frequency scale through a set of Mel-scaled triangular filters.

[0085] (5) Logarithmic operations 570 Output energy of each Mel-scale triangular filter is the logarithm of the sum of squares of the filtered signal.

[0086] (6) Discrete cosine transform 580 Final logarithmic energy Perform discrete cosine transform to get MFCCs:

[0087] in is the nth cepstral coefficient, M is the number of filters. Through a series of processing steps, MFCCs can extract features that describe the basic properties of sound from complex audio signals. These features are extremely valuable for various audio analysis and processing tasks.

[0088] Figure 6 A block diagram of a digital human identity authentication system provided for an embodiment of the present application, wherein the digital human identity authentication system 600 includes: a digital human identity library 610, a digital human identity authentication server 620 and a client 630. The digital human identity authentication system 600 is used to implement the method described in any embodiment of the present application, and the digital human identity authentication system includes multiple clients, at least one digital human identity library 610 and at least one digital human identity authentication server (as a server). Among them, any one of the clients is used for at least one of the following functions: generating, identifying or editing the electronic data of the digital human. Any one of the servers is used to generate the identity of the digital human. Any one of the servers is used to compare the identity with the pre-generated digital human identity to determine whether the authentication is passed. Any one of the digital human identity libraries is used to store and update the multimodal information of the digital human, including multiple of the following information: image features, audio features, semantic features; the source of the multimodal information of the digital human includes one or more of the following: digital human electronic information, digital human generation tools, identity data, authorization code, application scenario data.

[0089] In one embodiment, the digital human identity library 610 is used to store the digital human identity identifier and the fusion features of the digital human; the digital human identity authentication server 620 or the client 630 is used to execute steps 210 to 230 described in any embodiment of the present application; at least one of the clients 630 is used to input video, pictures, voice and other information related to the digital human. The client 630, the server 620 and the digital human identity library 610 are connected by a network. After the user terminal inputs the query information, the input query information is transmitted to the digital human identity authentication server through the network. The server is configured with the digital human identity authentication method provided in the embodiment of the present application, and compares it with the pre-generated digital human identity identifier stored in the digital human identity library 610 to determine whether the authentication is passed, and returns the result to the client 630.

[0090] Combined with Figure 1 , 6 Further examples: In one embodiment of the present application, when a generating user issues a license to a specific application user, the generating user of the digital person authorizes and the application user applies for authentication. The generating user provides at least a portion of the multimodal information or the source of the multimodal information to the application user, and the generating end and the application end generate an identity (in the embodiment of the present application, the generating end refers to the generating user client and / or the generating server, and the application end refers to the application user client and / or the application server). The first client 111 sends an instruction to the generating server 110 to generate a digital human video or picture according to the set algorithm. The first client 111 or the generating server 110 generates the identity of the digital person according to the method of steps 210 to 220 of the embodiment of the present application, and stores it in the digital person identity library 610 as a pre-generated identity, which is managed by the digital person identity server 160. Among them, the identity of the digital person is generated based on the multimodal information. When the application user sends an instruction to the application server through the second client 121 to retrieve the digital human video or picture, the second client or application server needs to generate the identity of the digital human according to the multimodal information from the first client and according to steps 210~220 of the embodiment of the present application. When the digital human identity server obtains the identity, it compares it with the pre-generated identity according to step 230 of the embodiment of the present application. After the authentication is passed, the application server can access the generation server and obtain the video or picture of the digital human. Here, it should be noted that the generator and the application end use the same algorithm to generate the identifier, and the generator sends the indication information of the algorithm for generating the identity identifier to the application end. The multimodal information based on the generator and the application end is also the same. As an authorization method, the generator sends the multimodal information or the source of the multimodal information for generating the identity identifier to the application end. For example, when the generator sends the multimodal information for generating the identity identifier to the application end, the digital human identifier is generated according to the multimodal information; when the generator sends the source of the multimodal information for generating the identity identifier to the application end, including the digital human electronic information and the authorization code, the multimodal information is obtained according to the digital human electronic information and the authorization code detection, and then the digital human identifier is generated. In this embodiment, the first server sends an indication of the algorithm for generating the identity identifier to the second server to determine how to generate the digital human identifier. Alternatively, the first client sends an indication of the algorithm for generating the identity identifier to the second client to determine how to generate the digital human identifier. In this embodiment, the first server sends multimodal information or an indication of the source of multimodal information to the second server to determine how to generate the digital human identifier. Alternatively, the first client sends multimodal information or an indication of a source of multimodal information to the second client to determine how to generate a digital human identification. The indicated source of multimodal information includes at least a portion of the following information: digital human electronic information, digital human generation tool, identity data, authorization code, and application scenario data.

[0091] In one embodiment of the present application, when the application user issues a license to the generating user, the application user authorizes and the generating user applies for authentication. The application user provides at least a portion of the multimodal information or the source of the multimodal information to the generating user, and the generating end and the application end generate an identity. For example, the generating user provides the electronic information of the digital person, the application user provides the application scenario data, and the generating end or any end of the application end pre-generates the identity, and the generating user generates the identity in real time when applying for authentication. The first client 111 sends an instruction to the generating server 110 to generate a video or picture of the digital person according to the set algorithm. The second client 121 or the application server 120 generates the identity of the digital person according to the method of steps 210 to 220 of the embodiment of the present application, and stores it in the digital person identity library as a pre-generated identity, which is managed by the digital person identity server 150. Among them, the identity of the digital person is generated based on the multimodal information. When the generating user sends an instruction to the generating server through the first client 111 to generate a video or picture of a digital human, the first client needs to generate an identity of the digital human according to the multimodal information from the second client and according to steps 210 to 220 of the embodiment of the present application. When the digital human identity server obtains the identity, it compares it with the pre-generated identity according to step 230 of the embodiment of the present application. After the authentication is passed, the application server can access the generating server and obtain the video or picture of the digital human. Here, it should be noted that the generator and the application end use the same algorithm to generate the identifier, the application end sends the instruction information of the algorithm for generating the identity identifier to the generator end, and the multimodal information based on the generator and the application end is also the same. As an authorization method, the application end sends the multimodal information or the source of multimodal information for generating the identity identifier to the generator end. For example, the application end sends the multimodal information for generating the identity identifier to the generator end, and the digital human identity information is generated based on the multimodal information; the application end sends the source of multimodal information for generating the identity identifier to the generator end, including the digital human electronic information and the application scenario data, and the multimodal information is generated based on the digital human electronic information and the application scenario data detection, and then the digital human identifier is generated. In this embodiment, the second server sends an instruction containing the algorithm for generating the identity identifier to the first server to determine how to generate the digital human identifier. Alternatively, the second client sends an instruction containing the algorithm for generating the identity identifier to the first client to determine how to generate the digital human identifier. In this embodiment, the second server sends multimodal information or an instruction containing the source of multimodal information to the first server to determine how to generate the digital human identifier. Alternatively, the second client sends multimodal information or an indication of a source of multimodal information to the first client to determine how to generate a digital human identification. The indicated source of multimodal information includes at least a portion of the following information: digital human electronic information, digital human generation tool, identity data, authorization code, and application scenario data.

[0092] In one embodiment of the present application, when a third party issues a license to a specific application user, the third party authorizes and the application user of the digital person applies for authentication. The third-party user provides at least a portion of the multimodal information or the source of the multimodal information to the application user, and the application end generates an identity. For example, the generating end provides the electronic information of the digital person, the third-party end provides the authorization code, and the generating end or any of the third-party ends pre-generates the identity (in this application, the third-party end refers to the third-party user client or the third-party server end), and the identity is generated in real time when the application user applies for authentication. For example, the first client 111 sends an instruction to the generating server 110 to generate a video or picture of the digital person according to the set algorithm. The third client 131 sends an authorization code to the second client 121. The second client 121 or the third client 131 generates the identity of the digital person according to the method of steps 210 to 220 of the embodiment of the present application, and stores it in the digital person identity library as a pre-generated identity, which is managed by the digital person identity server 150. Among them, the identity of the digital person is generated according to the multimodal information. When the application user sends an instruction to the application server through the second client 121 to retrieve the digital human video or picture, the second client or application server needs to generate the identity of the digital human according to the multimodal information from the first client or the third client and the method of steps 210 to 220 of the embodiment of the present application. When the digital human identity server obtains the identity, it compares it with the pre-generated identity according to step 230 of the embodiment of the present application. After the authentication is passed, the application server can access the generation server and obtain the video or picture of the digital human. Here, it should be noted that the generation end, the third-party end and the application end should use the same algorithm to generate the identification, and the multimodal information based on it is also the same. As an authorization method, the generation end or the third-party end sends the multimodal information or multimodal information source used to generate the identity identification to the application end. For example, the generation end or the third-party end sends the multimodal information used to generate the identity identification to the application end, and the digital person identity information is generated based on the multimodal information; the generation end or the third-party end sends the multimodal information source used to generate the identity identification to the application end, including the digital person electronic information and the authorization code, and then detects the digital person electronic information and the authorization code to generate the multimodal information, and then generates the digital person identification. The third party here, such as Figure 1The audit user or authentication user in may be a regulatory agency. In this embodiment, the third server sends an instruction including an algorithm for generating an identity identifier to the second server to determine how to generate a digital human identifier. Alternatively, the third client sends an instruction including an algorithm for generating an identity identifier to the second client to determine how to generate a digital human identifier. In this embodiment, the third server sends multimodal information or an instruction including the source of multimodal information to the second server to determine how to generate a digital human identifier. Alternatively, the third client sends multimodal information or an instruction including the source of multimodal information to the second client to determine how to generate a digital human identifier.

[0093] In one embodiment of the present application, when a third party issues a license to a specific generating user, the third party authorizes and the generating user applies for authentication. The third-party user provides at least a portion of the multimodal information or the source of the multimodal information to the generating user, and the generating end generates an identity. For example, the generating end provides the electronic information of the digital person, the third-party end provides the authorization code, and any end pre-generates the identity, and the generating user generates the identity in real time. For example, the third-party end or the generating end provides the electronic information of the digital person, the third-party end provides the authorization code, and any end of the generating end or the third-party end pre-generates the identity, and the generating user generates the identity in real time when applying for authentication. For example, the first client 111 sends an instruction to the generating server 110 to generate a video or picture of the digital person according to the set algorithm. The third client 131 sends the authorization code to the first client 111. The first client 111 or the third client 131 generates the identity of the digital person according to the method of steps 210 to 220 of the embodiment of the present application, and stores it in the digital person identity library as a pre-generated identity, which is managed by the digital person identity server 150. Among them, the identity of the digital person is generated according to the multimodal information. When the generating user issues an instruction to the generating server through the first client 111 to request the generation of a digital human video or picture, the first client or the generating server needs to generate the identity of the digital human according to the method of steps 210 to 220 of the embodiment of the present application based on the multimodal information from the first client and / or the third client. When the digital human identity server obtains the identity, it compares it with the pre-generated identity according to step 230 of the embodiment of the present application. After the authentication is passed, the generating server can further generate the video or picture of the digital human. Here, it should be noted that the generating end and the third-party end should use the same algorithm to generate the identification, and the multimodal information based on it is also the same. As an authorization method, the third-party end sends the multimodal information or multimodal information source used to generate the identity to the generating end. For example, the third-party end sends the multimodal information used to generate the identity to the generating end, and the digital human identity information is generated according to the multimodal information; the third-party end sends the multimodal information source used to generate the identity to the generating end, including the digital human electronic information and the authorization code, and the digital human electronic information and the authorization code generate the multimodal information, and then generate the digital human identification. The third party here, such as Figure 1The audit user or authentication user in may be a regulatory agency. In this embodiment, the third server sends multimodal information or an indication containing the source of the multimodal information to the first server to determine how to generate a digital human identifier. Alternatively, the third client sends multimodal information or an indication containing the type of multimodal information to the first client to determine how to generate a digital human identifier. The third server sends an indication containing an algorithm for generating an identity identifier to the first server to determine how to generate a digital human identifier. Alternatively, the third client sends an indication containing an algorithm for generating an identity identifier to the first client to determine how to generate a digital human identifier.

[0094] It should be noted that in various embodiments of the present application, when authorization is performed using an authorization code, the multimodal information source indicated by the authorizer includes the authorization code and also includes at least a portion of the following information: digital human electronic information, digital human generation tool, identity data, and application scenario data. The information used for authorization may also be other multimodal information. In another embodiment, when authorization is performed using application scenario data, the indicated multimodal information source includes application scenario data and also includes at least a portion of the following information: digital human electronic information, digital human generation tool, and identity data. In another embodiment, when authorization is performed using identity data, the indicated multimodal information source includes identity data and also includes at least a portion of the following information: digital human electronic information, digital human generation tool, and application scenario data. In another embodiment, when authorization is performed using digital human generation tool information, the indicated multimodal information source includes a digital human generation tool and also includes at least a portion of the following information: digital human electronic information, application scenario data, and identity data.

[0095] When it is necessary to explain, the detection server for digital human can be used to detect the source of multimodal information and obtain the required multimodal information. It is also necessary to explain that, by using the detection server for digital human, the digital human generation tool information can be obtained by detecting the electronic information of the digital human, and by detecting the electronic information of the digital human, other multimodal information that can be used to generate the identity information of the digital human can be further generated. Those skilled in the art can understand this based on the above embodiments, and no further explanation is given here.

[0096] Figure 7 A digital human identity authentication device provided in an embodiment of the present application is used to implement the method described in the first aspect of the present application, including: an acquisition module 710, a generation module 720 and a determination module 730; The acquisition module 710 is used to obtain the multimodal information of the digital human; the functions implemented are as described in step 210 of the embodiment of the present application; The generation module 720 is used to generate fusion features and encode them through multi-modal tensor pooling fusion to generate the identity of the digital person; the functions implemented are as described in step 220 of the embodiment of this application. The determination module 730 is used to compare the identity with the pre-generated digital human identity to determine whether the authentication is passed; the function implemented is as described in step 230 of the embodiment of the present application.

[0097] In one embodiment, the acquisition module 710 includes a facial feature processing unit, which is used to: use a multi-task convolutional neural network to detect pictures in the digital human video and / or the pictures, and use VGGFace to determine the digital human facial feature vector.

[0098] In one embodiment, the acquisition module 710 includes a speech feature processing unit, and is further used to: detect the speech signal of the digital human using Mel-frequency cepstral coefficients to determine a speech feature vector.

[0099] In one embodiment, the acquisition module 710 includes a multimodal receiving unit for receiving multimodal information from other clients or servers, or receiving an indication of a source of multimodal information from other clients or servers.

[0100] In one embodiment, the acquisition module 710 includes an algorithm receiving unit, which is used to receive indication information about the algorithm for generating digital human identity information from other clients or servers.

[0101] In one embodiment, the generating module 720 is further used to: fuse the facial feature vector and the speech feature vector through multimodal tensor pooling to generate a fused feature and encode it.

[0102] The generation module 720 is also used for any one or more of the other sources of multimodal information, whose original information or video features, audio features and / or semantic features generated after artificial intelligence extraction can be combined with the selected facial feature vector and the audio feature vector The splicing is performed, and the fusion features are calculated and encoded to generate the identity of the digital person.

[0103] Among them, the sources of digital human multimodal information include one or more of the following: digital human electronic information, digital human generation tools, identity data, authorization code, and application scenario data.

[0104] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0105] Therefore, the present application also proposes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in any embodiment of the first aspect of the present application is implemented.

[0106] Furthermore, in order to generate the device described in the embodiment of the second aspect of this application, the present application also proposes an electronic device (or computing device), comprising a processor and a memory storing program instructions, wherein the processor is configured to implement the method described in the embodiment of the first aspect of this application when running the program instructions.

[0107] Figure 8 The electronic device 800 shown in the figure is only an example and should not bring any limitation to the functions and application scope of the embodiment of the present application.

[0108] It includes: one or more processors 820; a storage device 810, used to store one or more programs. When the one or more programs are executed by the one or more processors 820, the one or more processors 820 implement the digital human identity authentication method provided in the embodiments of the present application, and the method includes at least one step described in each embodiment of the present application.

[0109] The number of processors 820 in the electronic device may be one or more. Figure 8 A processor 820 is taken as an example; the processor 820, storage device 810, input device 830 and output device 840 in the electronic device can be connected via a bus or other methods, and the figure takes the connection via bus 850 as an example.

[0110] The storage device 810, as a computer-readable storage medium, can be used to store software programs, computer executable programs, and module units, such as program instructions corresponding to the digital human identity authentication method in the embodiment of the present application. The storage device 810 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the terminal, etc. In addition, the storage device 810 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the storage device 810 may further include a memory remotely arranged relative to the processor 820, and these remote memories may be connected via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0111] It can be understood that the processor 820 of this embodiment includes at least one module among the acquisition module 710, generation module 720 and determination module 730 of the embodiment of the present application, which is used to implement at least one part of the functions of the above-mentioned embodiment, and can be an entity composed of one or more components. The specific functions of each module will not be repeated here.

[0112] The input device 830 may be used to receive input numbers, character information or voice information, and to generate key signal inputs related to user settings and function controls of the electronic device.

[0113] The output device 840 may include electronic devices such as a display screen and a speaker.

[0114] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0115] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A digital human identity authentication method, It is characterized in that The following steps are involved: Obtain multimodal information of digital humans; By multi-modal tensor pooling fusion, fusion features are generated and encoded to generate the identity of the digital person; The identity is compared with a pre-generated digital human identity to determine whether the authentication is successful.

2. The digital human identity authentication method according to claim 1, It is characterized in that The multimodal information includes multiple types of the following information: image features, audio features, and semantic features; The sources of digital human’s multimodal information include one or more of the following: digital human electronic information, digital human generation tools, identity data, authorization code, and application scenario data.

3. The digital human identity authentication method according to claim 1, It is characterized in that At least a portion of the multimodal information is obtained from an authorization code, and the authorization code is set by at least one of the following clients: a digital human generation user client, a digital human application user client, and a third-party client.

4. The digital human identity authentication method according to claim 1, It is characterized in that At least a part of the multimodal information is obtained from application scenario data, and the application scenario data is set by at least one of the following clients: a digital human generation user client, a digital human application user client, and a third-party client.

5. The digital human identity authentication method according to claim 1, It is characterized in that At least a portion of the multimodal information is obtained from identity data, and the identity data includes at least one of the following: identity data of one or more digital human-generated users, identity data of one or more digital human-applied users, and identity data of one or more third parties.

6. The digital human identity authentication method according to claim 1, It is characterized in that include: The identity identifier is generated by sequentially expanding each dimension of the fusion feature and performing a hash operation.

7. A digital human identity authentication device, used to implement the digital human identity authentication method according to any one of claims 1 to 6, It is characterized in that include: Acquisition module, generation module and determination module; The acquisition module is used to obtain the multimodal information of the digital human; The generation module is used to generate and encode fusion features through multi-modal tensor pooling fusion to generate the identity of the digital person; The determination module is used to compare the identity with a pre-generated digital human identity to determine whether the authentication is passed.

8. A digital human identity authentication system, used to implement the method described in any one of claims 1 to 6, It is characterized in that Includes multiple clients and at least one server; Any one of the clients is used for at least one of the following functions: generating, identifying or editing the electronic data of the digital person; Any one of the servers is used to generate the identity of the digital person; Any of the servers is used to compare the identity with a pre-generated digital human identity to determine whether the authentication is successful.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Multi-modal input digital character generation method and device, equipment and storage medium

    CN117132864A

  • Virtual human identification and authentication method and system based on block chain

    CN118627048A

  • Video network asset identity credibility identification method and system

    CN118821029A

  • Systems and methods for facial attribute manipulation

    US20230252687A1