An identity recognition method, device, apparatus, and storage medium

Abstract

This application discloses an identity recognition method, apparatus, device, and storage medium, relating to the field of human-computer interaction technology. The method includes: acquiring audio and video stream data of a user appearing within a preset camera area; extracting facial features from the video stream data and querying a face vector library to determine if a face cluster corresponding to the facial feature vector exists; if so, extracting voiceprint features from the audio stream data and querying a voiceprint vector library to determine if a voiceprint cluster corresponding to the voiceprint feature vector exists; if so, determining the degree of mouth opening based on key points of different facial features in each frame image, and determining whether the current user is the current speaker based on the degree of mouth opening and the start and end times of the speech; if so, querying a relational graph database for user identifiers associated with the face cluster and voiceprint cluster to determine the current user's identity. This application can improve the accuracy of user identity recognition and meets the needs of real-time identity recognition.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and in particular to an identity recognition method, device, equipment and storage medium. Background Technology

[0002] Currently, in real-time voice dialogue applications in open scenarios, human-computer dialogue devices / voice dialogue devices have the technical characteristic of not having a fixed dialogue subject (i.e., speaker). This means that multiple users may interact with the artificial intelligence dialogue system (such as AI robot) separately or simultaneously. Specific application scenarios include intelligent interaction for different visitors in exhibition hall reception scenarios and personalized voice dialogue for different family members in family companionship scenarios.

[0003] To achieve high-quality and diverse voice dialogue effects and improve user experience, voice dialogue devices typically need to distinguish the identities of different users in real-time dialogue scenarios. Current mainstream user identification methods usually involve pre-collecting physiological features and using real-time retrieval and matching during the dialogue to identify the speaker, then calculating the voiceprint in real-time and comparing the real-time voiceprint with a voiceprint database to determine the speaker ID (i.e., speaker identity). However, in real-time dialogue scenarios, nearly half of the voice requests are short, lasting no more than 3 seconds. The shorter the voice, the lower the matching accuracy and the greater the difficulty.

[0004] For identity recognition methods based on facial recognition, it is impossible to identify who is talking when there are multiple people in the scene. To solve the above problem, a new audio-visual lip-syncing technology called "speaker recognition" (which refers to the joint matching of a piece of speech with the time sequence of different faces in the video to determine which face is making the speech) has been proposed. Although this method can select the face with the highest correlation to the voice from multiple people in the real-time scene, most of the models are offline inference models, and the inference speed is too slow to meet the needs of real-time identity recognition. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide an identity recognition method, apparatus, device, and storage medium that can improve the accuracy of user identity recognition and meet the needs of real-time identity recognition. The specific solution is as follows: In a first aspect, this application discloses an identity recognition method applied to a voice dialogue device, comprising: Under preset authorization conditions, the system acquires video stream data of the current user appearing within a preset camera area, as well as voice stream data generated during the dialogue between the current user and the voice dialogue device; the preset authorization conditions are that the voice dialogue device has pre-acquired authorization from each user within the preset camera area for the collection and recognition of facial and voice information. The video stream data is subjected to facial feature extraction using a facial vectorization model to obtain facial feature vectors, and a pre-created facial vector library is used to query whether a facial cluster corresponding to the facial feature vector exists. If a face cluster corresponding to the face feature vector exists in the face vector library, then the voiceprint feature is extracted from the speech stream data through the voiceprint vectorization model to obtain the voiceprint feature vector, and the voiceprint feature vector is obtained by querying the pre-created voiceprint vector library to see if a voiceprint cluster corresponding to the voiceprint feature vector exists. If a voiceprint cluster corresponding to the voiceprint feature vector exists in the voiceprint vector library, then key points of different facial parts in each frame of the video stream data are detected to obtain facial key points; the facial key points include mouth key points. Based on the facial key points, the degree of mouth opening and closing in each frame image is determined, and based on the degree of mouth opening and closing and the start and end time points of the speech stream data, it is determined whether the current user is the current speaker. If the current user is the current speaker, then the target user identifier that is associated with the face cluster and the voiceprint cluster is queried from the relationship graph database, and the identity of the current user is determined based on the target user identifier.

[0006] Optionally, the step of extracting voiceprint features from the speech stream data using a voiceprint vectorization model to obtain a voiceprint feature vector includes: The speech stream data is subjected to speech noise reduction processing to obtain noise-reduced speech stream data; The noise-reduced speech stream data is segmented using a speech activity detection model to obtain multi-segmented speech stream data. The voiceprint feature vector is obtained by sequentially extracting voiceprint features from the segmented speech stream data using a voiceprint vectorization model.

[0007] Optionally, the step of sequentially extracting voiceprint features from the segmented speech stream data using a voiceprint vectorization model to obtain voiceprint feature vectors includes: Determine whether the duration of the segmented speech stream data is less than the first duration; If the duration of the segmented speech stream data is not less than the first duration, then it is determined whether the amplitude of the segmented speech stream data is lower than a preset amplitude threshold. If the amplitude of the segmented speech stream data is not lower than a preset amplitude threshold, then the voiceprint feature is extracted from the segmented speech stream data through a voiceprint vectorization model to obtain a voiceprint feature vector.

[0008] Optionally, querying the pre-created face vector library to see if a face cluster corresponding to the face feature vector exists includes: The similarity between the face feature vector and the face feature vector in each face cluster in the face vector database is calculated sequentially to obtain the face similarity, and it is determined whether the face similarity exceeds the first similarity threshold. If the face similarity exceeds the first similarity threshold, it is determined that there is a face cluster in the face vector library that corresponds to the face feature vector; Accordingly, the step of querying the pre-created voiceprint vector library to see if a voiceprint cluster corresponding to the voiceprint feature vector exists includes: The similarity between the voiceprint feature vector and the voiceprint feature vector in each voiceprint cluster in the voiceprint vector library is calculated sequentially to obtain the voiceprint similarity, and it is determined whether the voiceprint similarity exceeds the second similarity threshold. If the voiceprint similarity exceeds the second similarity threshold, it is determined that there exists a voiceprint cluster in the voiceprint vector library that corresponds to the voiceprint feature vector.

[0009] Optionally, the identity recognition method further includes: If the face similarity corresponding to all face clusters in the face vector library does not exceed the first similarity threshold, it is determined that there is no face cluster in the face vector library corresponding to the face feature vector, and a first cluster identifier corresponding to the face feature vector is created in the face vector library, and then the face feature vector is bound to the first cluster identifier. If the similarity of the voiceprints corresponding to all voiceprint clusters in the voiceprint vector library does not exceed the second similarity threshold, it is determined that there is no voiceprint cluster in the voiceprint vector library corresponding to the voiceprint feature vector, and a second cluster identifier corresponding to the voiceprint feature vector is created in the voiceprint vector library, and then the voiceprint feature vector is bound to the second cluster identifier. In the relational graph database, a first node and a second node corresponding to the first cluster identifier and the second cluster identifier are created respectively. A target user identifier is assigned to the current user, and an association relationship between the first node and the second node is established based on the target user identifier.

[0010] Optionally, before extracting voiceprint features from the speech stream data using a voiceprint vectorization model to obtain the voiceprint feature vector, the method further includes: The number of face feature vectors in the face cluster corresponding to the face feature vector is counted to obtain the number of face vectors, and it is determined whether the number of face vectors exceeds a first number threshold. If the number of face vectors does not exceed the first number threshold, then the face feature vectors are added to the face cluster; Accordingly, before querying the target user identifier that is associated with the face cluster and the voiceprint cluster from the relationship graph database, the process further includes: The number of voiceprint feature vectors in the voiceprint cluster corresponding to the voiceprint feature vector is counted to obtain the number of voiceprint vectors, and it is determined whether the number of voiceprint vectors exceeds the second number threshold. If the number of voiceprint vectors exceeds the second quantity threshold, the voiceprint feature vectors in the voiceprint cluster are replaced by the voiceprint feature vectors according to the time priority rule. If the number of voiceprint vectors does not exceed the second number threshold, then determine whether the duration of the segmented speech stream data exceeds the second duration. If the duration of the segmented speech stream data exceeds the second duration, the voiceprint feature vector is added to the voiceprint cluster.

[0011] Optionally, the step of detecting key points of different facial regions within each frame of the video stream data to obtain facial key points includes: Face detection is performed on each frame of the video stream data using a preset face detection model to obtain face detection results; the face detection results include face detection boxes and detection box confidence scores. The face detection results are input into the target tracking model so that face detection boxes whose positions meet the preset proximity conditions between consecutive frames in the video stream data are associated across frames using track_id. The face detection results are input into a 3D key point detection model to detect key points of different face parts within each face detection box, thereby obtaining facial key points; the facial key points include mouth key points. Accordingly, determining the degree of mouth opening and closing in each frame image based on the facial key points, and determining whether the current user is the current speaker based on the degree of mouth opening and closing and the start and end times of the speech stream data, includes: Based on the facial key points, the degree of mouth opening and closing of each face in the corresponding frame image is calculated to obtain the mouth opening and closing value; The maximum and minimum values ​​are obtained from the mouth opening and closing value sequence by means of a sliding window voting method and based on the track_id; the mouth opening and closing value sequence is all the mouth opening and closing values ​​corresponding to the mouth region of the same face in different frame images in the video stream data; Calculate the difference between the maximum value and the minimum value to obtain a first difference, and determine whether the first difference is less than a preset threshold; If the first difference is not less than a preset threshold, it is determined that the current user is in a dialogue state, and the number of all frame images in the dialogue state is counted to obtain the statistical result. Determine whether the time period length corresponding to the statistical result matches the second difference; the second difference is the time difference between the start and end times of the speech stream data after segmentation. If the time period length corresponding to the statistical result matches the second difference, then the current user is determined to be the current speaker; If the time period length corresponding to the statistical result does not match the second difference, then it is determined that the current user is not the current speaker.

[0012] Secondly, this application discloses an identity recognition device applied to a voice dialogue device, comprising: The acquisition module is used to acquire video stream data of the current user appearing in a preset camera area, as well as voice stream data generated during the dialogue between the current user and the voice dialogue device, under the condition of meeting the preset authorization. The preset authorization condition is that the voice dialogue device has pre-acquired authorization from each user in the preset camera area for the collection and recognition of facial and voice information. The face feature extraction module is used to extract face features from the video stream data using a face vectorization model to obtain face feature vectors; The first query module is used to query from a pre-created face vector library whether a face cluster corresponding to the face feature vector exists; The voiceprint feature extraction module is used to extract voiceprint features from the speech stream data through a voiceprint vectorization model if a face cluster corresponding to the face feature vector exists in the face vector library. The second query module is used to query from a pre-created voiceprint vector library whether there is a voiceprint cluster corresponding to the voiceprint feature vector; The detection module is used to detect key points of different facial parts in each frame of the video stream data if a voiceprint cluster corresponding to the voiceprint feature vector exists in the voiceprint vector library, and to obtain facial key points; the facial key points include mouth key points. The judgment module is used to determine the degree of mouth opening and closing in each frame image based on the facial key points, and to determine whether the current user is the current speaker based on the degree of mouth opening and closing and the start and end time points of the speech stream data. The identity recognition module is used to query the target user identifier that is associated with the face cluster and the voiceprint cluster from the relationship graph database if the current user is the current speaker, and determine the identity of the current user based on the target user identifier.

[0013] Thirdly, this application discloses an electronic device, including a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the aforementioned identity recognition method.

[0014] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned identity recognition method.

[0015] As can be seen, this application is applied to voice dialogue devices. Under the condition of meeting preset authorization (that the voice dialogue device has pre-obtained authorization from each user within a preset camera area for the collection and recognition of facial and voice information), it acquires video stream data of the current user appearing within the preset camera area, as well as voice stream data generated during the current user's dialogue with the voice dialogue device. Then, it extracts facial features from the video stream data using a face vectorization model to obtain facial feature vectors, and queries a pre-created face vector library to see if a face cluster corresponding to the facial feature vector exists. If a face cluster corresponding to the facial feature vector exists in the face vector library, it then extracts voiceprint features from the voice stream data using a voiceprint vectorization model. The system obtains the voiceprint feature vector and queries the pre-created voiceprint vector library to see if a voiceprint cluster corresponding to the voiceprint feature vector exists. If a voiceprint cluster corresponding to the voiceprint feature vector exists in the voiceprint vector library, the system detects key points of different facial parts in each frame of the video stream data to obtain facial key points (including mouth key points). Based on the facial key points, the system determines the degree of mouth opening and closing in each frame of the image. Based on the degree of mouth opening and closing and the start and end time points of the speech in the speech stream data, the system determines whether the current user is the current speaker. If the current user is the current speaker, the system queries the relation graph database for target user identifiers that are associated with the face cluster and voiceprint cluster, and determines the identity of the current user based on the target user identifiers. Under the premise of meeting the preset authorization conditions, this application first extracts facial features from the video stream data within the preset camera area to obtain a facial feature vector. It then queries a pre-created facial vector library to see if a facial cluster corresponding to the current facial feature vector exists. If it exists, it extracts voiceprint features from the speech stream data to obtain a voiceprint feature vector. It then queries a pre-created voiceprint vector library to see if a voiceprint cluster corresponding to the current voiceprint feature vector exists. If it exists, it detects key points (including mouth key points) of different facial parts in each frame image and determines the degree of mouth opening based on these key points. Then, based on the degree of mouth opening and the start and end times of the speech in the speech stream data, it determines whether the current user is the current speaker. If so, it queries a relational graph database for a facial feature vector. The application identifies users whose clusters and voiceprint clusters are related, and identifies the current speaker's identity based on these user identifiers. Specifically, this application utilizes the collaboration between three vector libraries (face vector library, voiceprint vector library, and relationship graph database) and leverages data from both speech streams and video streams to query and match the current user's identity. This multimodal identity recognition method improves the accuracy of identity recognition. Furthermore, this application determines whether the current user is the current speaker by the degree of mouth opening and closing and the start and end times of speech, thus avoiding misjudgments and further improving the accuracy of user identity recognition. This judgment method is simple and does not involve complex model reasoning, therefore it can meet the needs of real-time identity recognition. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0017] Figure 1 This is a flowchart of an identity recognition method disclosed in this application; Figure 2 This is a schematic diagram of a specific video stream data processing flow disclosed in this application; Figure 3 This is a schematic diagram illustrating a specific face-voice correspondence disclosed in this application; Figure 4 This is a schematic diagram of a specific lip movement detection disclosed in this application; Figure 5 This is a schematic diagram illustrating a specific relationship between a voice node and a face node disclosed in this application; Figure 6 This is a schematic diagram illustrating a specific relationship between a voice node and a face node disclosed in this application; Figure 7 This is a schematic diagram of the structure of an identity recognition device disclosed in this application; Figure 8 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] This application discloses an identity recognition method applied to a voice dialogue device. See [link to relevant documentation]. Figure 1 As shown, the method includes: Step S11: Under the condition of meeting the preset authorization, acquire the video stream data of the current user appearing in the preset camera area, as well as the voice stream data generated during the dialogue between the current user and the voice dialogue device; the preset authorization condition is that the voice dialogue device has pre-acquired the authorization of each user in the preset camera area for the collection and recognition of facial and voice information.

[0020] It should be noted that the identity recognition scheme proposed in this application is applied to voice dialogue devices (such as AI robots, intelligent agents, etc.). This device can acquire video stream data of the current user appearing in a preset camera area under preset authorization conditions. The video stream data can be collected by a pre-installed camera. As long as a user is detected to enter the preset camera area, the video stream data of the user appearing in that area is automatically collected. In addition, the device can also acquire the voice information output by the current user in a real-time voice dialogue scenario to obtain voice stream data. This data can be collected by voice acquisition devices such as microphones.

[0021] It should be noted that the collection of video and audio stream data is conducted under pre-defined authorized conditions. Authorization has been obtained to collect and recognize audio and video information of each user within the pre-defined camera area, and no personal privacy has been violated. For example, in an exhibition hall reception scenario, after obtaining the visitor's consent in advance, the faces of users appearing within the pre-defined camera area (i.e., within the camera's field of view) are captured in real time (e.g., within 1 second), and the voice information output by the user during the conversation is automatically acquired.

[0022] Step S12: Extract facial features from the video stream data using a face vectorization model to obtain a face feature vector, and query the pre-created face vector library to see if there is a face cluster corresponding to the face feature vector.

[0023] In this embodiment, after obtaining the video stream data of the user appearing in the area, the face feature vector can be obtained by extracting face features from the video stream data through a face vectorization model, such as the ArcFace model (Additive Angular Margin Loss for Deep Face Recognition, a face recognition model based on deep learning). Then, the existence of a vector corresponding to the current face feature vector is retrieved / matched from each face cluster in the pre-created face vector library.

[0024] Each face cluster represents a cluster of highly similar vectors representing the facial physiological features (i.e., facial features) of the same user. Each face cluster corresponds to a unique identifier, such as face_id. Each face_id can correspond to multiple face images, such as frontal and side views.

[0025] Specifically, the step of querying the pre-created face vector library to determine whether a face cluster corresponding to the face feature vector exists may include: sequentially calculating the similarity between the face feature vector and the face feature vectors in each face cluster of the face vector library to obtain face similarity, and determining whether the face similarity exceeds a first similarity threshold; if the face similarity exceeds the first similarity threshold, then it is determined that a face cluster corresponding to the face feature vector exists in the face vector library. In this embodiment, the similarity (such as cosine similarity) between the face feature vector obtained after face feature extraction and the face feature vectors in each face cluster of the pre-created face vector library can be calculated first, and then it can be determined whether the current face similarity is greater than the first similarity threshold; if the current face similarity is greater than the first similarity threshold, then it is determined that a face cluster corresponding to the current face feature vector exists in the face vector library; if the current face similarity is not greater than the first similarity threshold, then it is determined that a face cluster corresponding to the current face feature vector does not exist in the face vector library.

[0026] Step S13: If a face cluster corresponding to the face feature vector exists in the face vector library, then the voiceprint feature is extracted from the speech stream data through the voiceprint vectorization model to obtain the voiceprint feature vector, and the voiceprint feature vector is obtained by querying the pre-created voiceprint vector library to see if a voiceprint cluster corresponding to the voiceprint feature vector exists.

[0027] In this embodiment, if a face cluster corresponding to the above face feature vector exists in the face vector library, then the voiceprint feature vector is obtained by extracting voiceprint features from the above speech stream data through a voiceprint vectorization model, such as the CAM++ model (Context-Aware Masking++). Then, the existence of a vector corresponding to the current voiceprint feature vector is searched / matched from each voiceprint cluster in the pre-created voiceprint vector library.

[0028] Each voiceprint cluster represents a cluster of highly similar vectors representing the voiceprint features of the same user, and each voiceprint cluster corresponds to a unique identifier, such as voice_id. It should be noted that voiceprint clusters and face clusters belong to different modalities of vector representation and can reside in different vector spaces (i.e., the two vector spaces are isolated). Furthermore, the mapping method to the user depends on the chosen vectorization model. Each user can be uniquely identified using a uid (User Identifier), which can be a string, and each uid can be bound to two clusters simultaneously (i.e., the voiceprint cluster and the face cluster).

[0029] In one specific implementation, the step of extracting voiceprint features from the speech stream data using a voiceprint vectorization model to obtain voiceprint feature vectors may specifically include: performing speech denoising on the speech stream data to obtain denoised speech stream data; segmenting the denoised speech stream data using a speech activity detection model to obtain multi-segmented speech stream data; and sequentially extracting voiceprint features from the segmented speech stream data using a voiceprint vectorization model to obtain voiceprint feature vectors. For example, first, a streaming speech denoising model, such as the Rnnoise model (a streaming speech denoising model that can process sequences with an RNN-like structure), is used to denoise the speech stream data collected by a microphone or other speech acquisition device to obtain denoised speech stream data. It should be noted that the Rnnoise model contains hidden states, is naturally adapted to long continuous speech sequences, can denoise input speech frames, and, see [link to relevant documentation], ... Figure 2As shown, the input shape and output shape of this model are the same. For example, when the input is a mono audio frame with a sampling rate of 48k and 480 sampling points (i.e., a frame width of 480), the output shape is equal to the input shape. Next, the denoised audio stream data is input to a rectifier to obtain mono, 48kHz sampling rate, and 512-frame-wide float32 audio frames. Then, a speech activity detection model, such as the silentVAD model (Voice Activity Detection), is used to detect the start and end points of the denoised audio stream data output by the rectifier. This identifies which audio stream is valid and needs to be processed in subsequent steps, i.e., the valid audio stream is segmented to obtain multiple continuous valid audio streams, i.e., segmented audio stream data (e.g., audio1 to audio31). In addition, during the output of continuous valid audio streams, a state machine can be used to monitor the output of the silentVAD model, and the silentVAD model can also output the probability value of the audio segment containing human voice (in the range of 0.0 to 1.0). When an end point is detected, the audio frames of the currently detected sentence can be concatenated to obtain the user's voice data for this dialogue (i.e., audio1 to audio31, which can be stored in an array). Furthermore, voiceprint features can be extracted from the segmented speech stream data (i.e., audio1 to audio31) sequentially using a voiceprint vectorization model (such as the CAM++ model) to obtain voiceprint feature vectors. At the same time, speech text (i.e., text) can be output through speech recognition.

[0030] Specifically, the step of sequentially extracting voiceprint features from the segmented speech stream data using a voiceprint vectorization model to obtain a voiceprint feature vector may include: determining whether the duration of the segmented speech stream data is less than a first duration; if the duration of the segmented speech stream data is not less than the first duration, determining whether the amplitude of the segmented speech stream data is lower than a preset amplitude threshold; if the amplitude of the segmented speech stream data is not lower than the preset amplitude threshold, then extracting voiceprint features from the segmented speech stream data using a voiceprint vectorization model to obtain a voiceprint feature vector. In this embodiment, after detecting the start and end points of the speech stream data after noise reduction using the sileroVAD model to obtain effective segmented speech stream data, duration and energy filtering can be applied to the data. Specifically, the duration of each segment of the segmented speech stream data is first calculated. Then, the duration of each segment is compared with a preset duration. If it is less than the preset duration (e.g., 600ms), it is considered noise. If it is not less than the preset duration (e.g., 600ms), the amplitude of the segmented speech stream data is further checked to see if it is lower than a preset amplitude threshold (which can be set according to the actual application scenario). For example, it is checked whether the RMS (Root Mean Square) of the segmented speech stream data is lower than a preset threshold. If it is lower than the preset threshold, it is considered noise; if it is not lower than the preset threshold, it is considered user-input dialogue. At this time, the voiceprint vectorization model (CAM++ model) can be used to extract voiceprint features (i.e., voiceprint recognition) from the segmented speech stream data (i.e., an audio array). It is understandable that voiceprint recognition has the problem of poor cross-device consistency, therefore, voiceprint recognition should be recorded on the same microphone hardware.

[0031] Specifically, the step of querying the pre-created voiceprint vector library to determine whether a voiceprint cluster corresponding to the voiceprint feature vector exists may include: sequentially calculating the similarity between the voiceprint feature vector and the voiceprint feature vectors in each voiceprint cluster of the voiceprint vector library to obtain the voiceprint similarity, and determining whether the voiceprint similarity exceeds a second similarity threshold; if the voiceprint similarity exceeds the second similarity threshold, it is determined that a voiceprint cluster corresponding to the voiceprint feature vector exists in the voiceprint vector library. In this embodiment, the similarity (such as cosine similarity) between the voiceprint feature vector after voiceprint recognition and each voiceprint feature vector in each voiceprint cluster of the pre-created voiceprint vector library is calculated sequentially to obtain the voiceprint similarity, and then it is determined whether the current voiceprint similarity exceeds a preset similarity threshold. If it does, it is determined that a voiceprint cluster corresponding to the current voiceprint feature vector exists in the voiceprint vector library.

[0032] In this embodiment, if the face similarity corresponding to all face clusters in the face vector library does not exceed a first similarity threshold, it is determined that there is no face cluster corresponding to the face feature vector in the face vector library, and a first cluster identifier corresponding to the face feature vector is created in the face vector library, and then the face feature vector is bound to the first cluster identifier; if the voiceprint similarity corresponding to all voiceprint clusters in the voiceprint vector library does not exceed a second similarity threshold, it is determined that there is no voiceprint cluster corresponding to the voiceprint feature vector in the voiceprint vector library, and a second cluster identifier corresponding to the voiceprint feature vector is created in the voiceprint vector library, and then the voiceprint feature vector is bound to the second cluster identifier; a first node and a second node corresponding to the first cluster identifier and the second cluster identifier are created in the relationship graph database respectively, a target user identifier is assigned to the current user, and an association relationship between the first node and the second node is established based on the target user identifier. In other words, if the vectors within all face clusters in the face vector database have low similarity to the current face feature vector, it indicates that no face cluster corresponding to the current feature vector can be retrieved / matched in the face vector database. In this case, a cluster identifier (luster_id, such as face_id) corresponding to the current face feature vector can be created in the face vector database, and then the current face feature vector is bound to the aforementioned cluster identifier (i.e., face_id), thereby saving the current face feature vector in the face vector database. Similarly, if no voiceprint cluster corresponding to the current voiceprint feature vector exists in the voiceprint vector database, a cluster identifier (luster_id, such as voice_id) corresponding to the voiceprint feature vector is created in the voiceprint vector database, thereby saving the current voiceprint feature vector in the voiceprint vector database. Therefore, if no vector information corresponding to the current user is found in the face vector database / voiceprint vector database, new user information will be created, and the corresponding vector will be saved / registered. Furthermore, in the graph database (used to store the association between voiceprints and faces), two nodes corresponding to the two cluster identifiers (face_id and voice_id) are created, designated as the first node and the second node. Then, a unique target user identifier (uid) is assigned to the current user, and the association between the first node and the second node is established based on this target user identifier. This process creates a new user identity and establishes the association between the new user's voiceprint feature vector and face feature vector. It is evident that in the graph database, voiceprint and face feature vectors exist as nodes represented by keys, and the edges connecting nodes represent an association pointing to a specific user uid.The established face vector database, voiceprint vector database, and relationship graph database are directly used in subsequent dialogues to match information with the user in this dialogue or other users in order to achieve user identification.

[0033] It's important to note that when a `face_id` and `voice_id` are created in the relational graph database, the `uid` is created along with the `face` node. That is, for each new user registering in the relational graph, a high-confidence voice-time associated face is created. If voiceprint recognition returns a unique, high-confidence `voice_id`, this `voice_id` can be directly used to query the relational graph for the `face` node connected to that voiceprint to obtain the `uid`. However, if multiple voiceprint recognition results are returned—for example, a single `voice_id` with low confidence, or two `voice_id`s with similar confidence—cross-modal recognition will be triggered.

[0034] For example, if the search results obtained after searching the voiceprint vector database and the face vector database are: Result = { "voice": [{"id":'voice_001', "score":0.72}], "face": [{"id":'face_007', "overlap":0.85}] }; This indicates that both the voiceprint and the face are unique, only one speaker meets the matching condition in terms of temporal overlap, and there is only one voiceprint recognition result. This is a high-confidence relationship between the face and the voiceprint. At this point, the edge weights of the two nodes can be strengthened in the relationship graph.

[0035] For example, if the search results obtained after searching the voiceprint vector database and the face vector database are: Result = { "voice": [{"id":'voice_001', "score":0.67}, {"id":'voice_003', "score":0.61}], "face": [{"id":'face_007', "overlap":0.76}, {"id":'face_016', "overlap":0.81}] }; This indicates that the retrieved voiceprint feature vector is not unique, but rather contains several similar results. Furthermore, the matched speaker's face within the current speech segment is not unique either, with multiple faces appearing simultaneously in the image. This constitutes an uncertain identity recognition attempt, which can be improved by using a relational graph database.

[0036] For example, if a face (face ID) is matched through speaker recognition (i.e., face recognition and voiceprint recognition), the system queries the relational graph database based on the voice ID and face ID to check if two corresponding nodes co-occur. The query method involves permuting and combining each voice ID with each face ID, then querying the edge weights of the corresponding two nodes in the relational graph database, returning the combination with the highest weight. If the weight value is higher than a preset threshold, this is considered an acceptable identity verification; otherwise, a fallback mechanism (a passive error handling strategy) is triggered, allowing the voice dialogue device to continue with the unidentified dialogue process.

[0037] For example, if the current conversation is too short or other reasons prevent the retrieval of a voiceprint feature vector that meets the minimum similarity requirement, the speaker's face_id matched within the current time period can be used. If the face_id is unique and the overlap is higher than a threshold (e.g., 0.8), the face node corresponding to that face_id can be retrieved from the relational graph database, and the user's identity can be determined based on the uid corresponding to that face node. If speaker matching fails or multiple speakers are matched, the identity recognition task is considered to have failed, and an alarm can be automatically triggered.

[0038] It should be noted that before extracting voiceprint features from the speech stream data using a voiceprint vectorization model to obtain voiceprint feature vectors, the method further includes: counting the number of face feature vectors in the face cluster corresponding to the face feature vector, obtaining the number of face vectors, and determining whether the number of face vectors exceeds a first quantity threshold; if the number of face vectors does not exceed the first quantity threshold, then the face feature vector is added to the face cluster. In this embodiment, to improve the overall identity recognition efficiency, the number of vectors in each voiceprint cluster and each face cluster can be limited, such as each vector cluster containing 3 to 5 vectors, that is, merging vectors with high similarity and ensuring that the number of vectors in a single cluster does not exceed 5. Specifically, the number of face feature vectors in the voiceprint cluster corresponding to the current face feature vector can be counted first, and then it can be determined whether the counted number of face vectors exceeds a preset quantity threshold (e.g., 5). If it does not exceed the preset quantity threshold (e.g., 5), then the current face feature vector is added to the corresponding face cluster.

[0039] Step S14: If a voiceprint cluster corresponding to the voiceprint feature vector exists in the voiceprint vector library, then key points of different facial parts in each frame of the video stream data are detected to obtain facial key points; the facial key points include mouth key points.

[0040] In this embodiment, if a voiceprint cluster corresponding to the above-mentioned voiceprint feature vector is found in the voiceprint vector library, then key points of different facial parts in each frame of the current video stream data are detected to obtain facial key points; facial key points include, but are not limited to, mouth key points, eye key points, nose key points, etc.

[0041] In one specific implementation, detecting key points of different facial features within each frame of the video stream data to obtain facial key points can specifically include: performing face detection on each frame of the video stream data using a preset face detection model to obtain face detection results; the face detection results include face detection boxes and detection box confidence scores; inputting the face detection results into a target tracking model to associate face detection boxes that meet preset proximity conditions between consecutive frames of the video stream data across frames using track_id; inputting the face detection results into a 3D key point detection model to detect key points of different facial features within each face detection box to obtain facial key points; the facial key points include mouth key points. In this embodiment, face detection and target tracking based on bounding boxes (i.e., face detection boxes) can be performed on each frame of the image at a preset sampling rate (e.g., 10 frames per second). Specifically, a lightweight face detection model, such as the SCRFD (Sample and Computation Redistribution for Face Detection) model, can be used. This model uses the ONNX (Open Neural Network Exchange) format to extract features from all faces appearing in each frame of the video stream, resulting in a list of faces, List[Face], containing multiple face feature vectors. Each feature vector in the list can contain the following information: {"bbox":bbox, "kps":kps, "score":score, "area":area}; where kps represents five simple key points (left eye center, right eye center, nose tip, left corner of mouth, and right corner of mouth) (which can be used for face correction), score is the detection box confidence score (which can be used for object tracking), area is the face area (which can be used for filtering and sorting), and bbox is the face detection box. Next, a Byte Track object tracking model is used to associate the face detection boxes (bboxes) across frames, specifically by linking bboxes that are close in position between consecutive frames using track_id. The Byte Track target tracking model takes a list of bounding boxes and scores as input and outputs a list of bounding boxes and track_ids. Finally, each face detected in each frame is bound to a track_id, which can be used to construct different face sequences.Furthermore, the face detection results (i.e., List[Face]) are input into a 3D keypoint detection model (such as the landmark3d model). Based on each face detection box (i.e., bbox) detected in a single frame image, 68 3D keypoints (x, y, z) for each face are predicted. The predicted 68 3D keypoints are then compared with 8 preset reference points to calculate the head yaw angle, such as pitch, yaw, and roll, which correspond to tilting the head up or down, turning the head left or right about the neck axis, and turning the head to the left or right, respectively. The right head tilt (shoulder-touching head tilt) calculates the head posture angle by comparing the deviation between real-time facial key points and standard key points. This is used to assess the current user's facial orientation and whether they are looking at the camera (which can be used for speaker detection). It also helps to assess whether the current user is facing forward or sideways (i.e., face recognition). In other words, it estimates the current user's head posture and uses the estimation results to help determine the current user's facial orientation. Next, the mouth key points extracted from 68 3D key points (x, y, z) can be used to analyze whether the mouth is moving and whether someone is speaking, i.e., speaker detection.

[0042] Step S15: Determine the degree of mouth opening and closing in each frame image based on the facial key points, and determine whether the current user is the current speaker based on the degree of mouth opening and closing and the start and end time points of the speech stream data.

[0043] In this embodiment, after obtaining facial key points including mouth key points, the degree of mouth opening and closing in each frame image can be determined based on the facial key points. Then, based on the degree of mouth opening and closing and the start and end time points of the speech stream data (such as the timestamps of the start and end speech frames), mouth movement detection / lip movement detection is performed to determine whether the current user is the current speaker. That is, mouth movement / lip movement is used as the judgment standard for visual speaker detection, thereby avoiding the situation where the current user A does not make a sound in the current scene, or the sound is made by another user B outside the scene.

[0044] It should be noted that the processing of each frame (such as face detection and object tracking, facial landmark acquisition and head pose estimation, and mouth movement detection) can be implemented on a graphics card (such as a GeForce RTX 4070) with a processing time of less than 30ms. Based on the actual processing time allocated to each frame (such as 100ms), it is possible to determine in real time whether each face (represented by track_id) is speaking in the current frame (i.e., the mouth is continuously moving) from a purely visual perspective. In addition, this application adopts dynamic face detection / detection, which is handled by a separate thread and runs asynchronously in the background. Therefore, it will not affect the latency of real-time dialogue. As long as a person enters the monitoring area of ​​the preset camera, face detection, object tracking, landmark recognition, and visual speaker detection will be performed at FPS (frames per second) = 10, and face recognition will be performed at FPS = 2 (returning the face_id of each face).

[0045] In one specific implementation, to minimize computational effort, the track_id generated by target tracking based on the face detection bounding box (bbox) can be used to query the corresponding face_id in the cache. If the track_id corresponding to the face detection bounding box (bbox) is found in the cache, the corresponding face_id is returned directly. If not found, face alignment (Face Alignment refers to the process of adjusting the face to a standard pose through geometric transformation after detecting a face) is performed first. Then, the ArcFace model is called to vectorize the images of each frame of the facial region and to retrieve the face feature vectors from the face vector library. If found, the corresponding face_id is returned; otherwise, a new face_id is registered.

[0046] Specifically, determining the degree of mouth opening and closing in each frame image based on the facial key points, and determining whether the current user is the current speaker based on the degree of mouth opening and closing and the start and end times of the speech in the speech stream data, may include: calculating the degree of mouth opening and closing for each face in the corresponding frame image based on the facial key points to obtain a mouth opening and closing value; obtaining the maximum and minimum values ​​from the opening and closing value sequence through a sliding window voting method based on the track_id; the opening and closing value sequence is all the mouth opening and closing values ​​corresponding to the mouth region of the same face in different frame images in the video stream data; and calculating the difference between the maximum and minimum values. The system calculates a first difference and determines whether the first difference is less than a preset threshold. If the first difference is not less than the preset threshold, the system determines that the current user is in a dialogue state. The system then counts the number of all frames in the dialogue state to obtain a statistical result. The system determines whether the time period corresponding to the statistical result matches a second difference. The second difference is the time difference between the start and end times of the segmented speech stream data. If the time period corresponding to the statistical result matches the second difference, the system determines that the current user is the current speaker. If the time period corresponding to the statistical result does not match the second difference, the system determines that the current user is not the current speaker. In this embodiment, the mouth opening value Mar of each face in the current frame can be calculated using 68 key points of each face in each frame image. The specific calculation formula is: Mar = d2 / d1; where d1 represents the distance between the two corners of the mouth, and d2 represents the distance between the center key points of the upper and lower lips. Then, using a sliding window voting method and based on track_id (i.e., the cross-frame association representation of the same face in multiple consecutive images), the maximum value and the maximum value are determined from all mouth opening values ​​Mar corresponding to the mouth region of the same face in different frame images (forming an opening value sequence, face.mar). The minimum value is determined, and the difference between the maximum and minimum values ​​is checked against a preset threshold. If the difference is less than the threshold, it indicates that the mouth is not moving within the current window, and the user is not engaged in a voice conversation. If the difference is not less than the threshold, it indicates that the user is engaged in a voice conversation. At this point, the number of all frames of the user in the conversation state can be counted to obtain the statistical results. The time period corresponding to the statistical results is checked against the time difference between the start and end times of the voice stream data (i.e., the effective voice segment) after the segmentation. If the results match, the user is determined to be the current speaker. If the results do not match, the user is determined to be the current speaker.

[0047] See Figure 3As shown, in the currently valid audio segment (audio_wav), two faces appear in the frame, namely face1 and face2, where face2 appears in the frame for a longer duration than face1; then, see... Figure 4 As shown, the start and end times of the current valid audio segment (audio_wav) are extracted, corresponding to frame1 (the first frame) to frame8 (the last frame). Lip movement detection is performed on each face, and the detection results are matched with the start and end times of the audio segment. Figure 4 In the audio segment (audio_wav), face2 has a higher matching degree. The mouth moves continuously across multiple consecutive frames, therefore face2 is identified as the current speaker. In other words, lip movement detection identifies the face with the higher temporal matching degree as the current speaker. Specifically, a preset matching threshold (e.g., 80%) can be used to determine whether it is the current speaker. For example, if multiple faces appear in the current frame, the face_id with the mouth movement temporal overlap exceeding the preset threshold (e.g., 80%) within the current valid audio segment is retained, and the corresponding face is identified as the current speaker.

[0048] In addition, for valid speech segments input in real time, they can be represented by the voiceprint vector cluster pointed to by voice_id and the start and end timestamps. If there are multiple faces in the current scene, the number of users with lip movements can be counted to obtain the lip movement statistics. For example, if there are k faces with lip movements, k can be 0, 1, ..., N, where N can be set as the maximum number of faces detected. k=0 means that no one's mouth is detected in the scene, and k>1 means that multiple people's mouths are moving in the current speech segment.

[0049] As can be seen, this application is based on cached multiple frames, such as the most recent 300 frames, corresponding to the most recent 30-second video segment, to determine which user's mouth is moving. Since it does not use a complex sound and mouth matching model, such as TalkNet, it can achieve fast recognition and millisecond-level real-time recognition, thus ensuring real-time performance.

[0050] Step S16: If the current user is the current speaker, query the target user identifier that is associated with the face cluster and the voiceprint cluster from the relationship graph database, and determine the identity of the current user based on the target user identifier.

[0051] In this embodiment, if the current user is determined to be the current speaker after lip movement detection, the system retrieves nodes that have a mapping relationship with the aforementioned voiceprint cluster and face cluster from the relation graph database, such as face nodes, and then determines the identity of the current user based on the target user identifier (uid) corresponding to the face node.

[0052] It should be noted that if a retrieval is successful in the graph database but a conflicting relationship is found, an identity update mechanism will be triggered, ultimately achieving the goal of retrieving existing users stably. Furthermore, the graph database has a fault-tolerance mechanism, allowing it to accommodate some errors introduced by rapid identity recognition.

[0053] For details, see Figure 5 As shown, if a single voice (voice_012) is simultaneously associated with two different faces (face_003 and face_005) in the relational graph database, and a single face (face_003) is also simultaneously associated with two different voices (voice_012 and voice_008), this disrupts the consistency between the user, face, and voiceprint. To address this conflict, the conflicting associations can be repaired accordingly. For details, see [link to documentation]. Figure 6 As shown, edges with high confidence can be continuously added to the relational graph database. If, during addition, it is found that the edge weight of the voice node corresponding to the current edge reaches 10 first, then all other edges connected to that voice node are disconnected (if there is already an old edge with a weight of 10, it is also cut off), and only the edge with the current weight of 10 is retained. When querying the relational graph database subsequently, there may still be cases where a single face_id corresponds to multiple voice_ids. However, all retrieved results can be sorted and filtered based on edge weights. For example, by inverting the order by edge weight, the face_id node corresponding to the edge with the highest weight is returned, and the user's identity is identified based on the uid bound to that face node. If there is only one face_id, the uid of the corresponding face node can be returned directly; if there is only one voice_id, and the confidence is greater than 0.6 (which can be set according to the actual application scenario), then the uid of the face node with the highest edge weight connected to that voice node is obtained, and user identity is identified based on the uid.

[0054] It should be noted that before querying the target user identifier associated with the face cluster and the voiceprint cluster from the relationship graph database, the process may further include: counting the number of voiceprint feature vectors in the voiceprint cluster corresponding to the voiceprint feature vector, obtaining the number of voiceprint vectors, and determining whether the number of voiceprint vectors exceeds a second quantity threshold; if the number of voiceprint vectors exceeds the second quantity threshold, then replacing the voiceprint feature vectors in the voiceprint cluster with the voiceprint feature vectors according to the time priority rule; if the number of voiceprint vectors does not exceed the second quantity threshold, then determining whether the duration of the segmented speech stream data exceeds a second duration; if the duration of the segmented speech stream data exceeds the second duration, then adding the voiceprint feature vector to the voiceprint cluster. In this embodiment, the number of voiceprint feature vectors in the voiceprint cluster corresponding to the current voiceprint feature vector is first counted. Then, it is determined whether the number of current voiceprint vectors exceeds a preset threshold (e.g., 5). If it does, the current voiceprint feature vector replaces the voiceprint feature vector in the corresponding voiceprint cluster in chronological order. Specifically, the cluster can be updated by overwriting the old vector with a new feature vector to save cluster space. In addition, if the number of voiceprint vectors does not exceed the preset threshold (e.g., 5), it is determined whether the duration of the current segmented speech stream data exceeds a preset duration (e.g., 3 seconds). If it does, it is determined that the current speech segment contains a lot of effective information. At this time, the current voiceprint feature vector can be added to the corresponding voiceprint cluster.

[0055] As can be seen, under the premise of meeting the preset authorization conditions, this application embodiment first extracts facial features from the video stream data within the preset camera area to obtain a facial feature vector, and then queries the pre-created facial vector library to see if there is a facial cluster corresponding to the current facial feature vector. If there is, it extracts voiceprint features from the speech stream data to obtain a voiceprint feature vector, and then queries the pre-created voiceprint vector library to see if there is a voiceprint cluster corresponding to the current voiceprint feature vector. If there is, it detects key points (including mouth key points) of different facial parts in each frame image, and determines the degree of mouth opening based on the key points. Then, it determines whether the current user is the current speaker based on the degree of mouth opening and the speech start and end time points corresponding to the speech stream data. If so, it queries the relation graph database. The application identifies users who are associated with face clusters and voiceprint clusters, and identifies the current speaker based on these user identifiers. Specifically, this application utilizes the collaboration of three vector libraries (face vector library, voiceprint vector library, and relationship graph database) and leverages data from both speech streams and video streams to query and match the current user's identity. This multimodal identity recognition method improves the accuracy of identity recognition. Furthermore, this application determines whether the current user is the current speaker by the degree of mouth opening and the start and end times of speech, thus avoiding misjudgments and further improving the accuracy of user identity recognition. This judgment method is simple and does not involve complex model reasoning, therefore it can meet the needs of real-time identity recognition.

[0056] Accordingly, this application also discloses an identity recognition device applied to a voice dialogue device, see [link to relevant documentation]. Figure 7 As shown, the device includes: The acquisition module 11 is used to acquire video stream data of the current user appearing in a preset camera area, and voice stream data generated during the dialogue between the current user and the voice dialogue device, under the condition of meeting the preset authorization; the preset authorization condition is that the voice dialogue device has pre-acquired authorization from each user in the preset camera area for the collection and recognition of facial and voice information. The face feature extraction module 12 is used to extract face features from the video stream data through a face vectorization model to obtain face feature vectors; The first query module 13 is used to query from a pre-created face vector library whether there is a face cluster corresponding to the face feature vector; The voiceprint feature extraction module 14 is used to extract voiceprint features from the speech stream data through a voiceprint vectorization model if a face cluster corresponding to the face feature vector exists in the face vector library, thereby obtaining a voiceprint feature vector. The second query module 15 is used to query from a pre-created voiceprint vector library whether there is a voiceprint cluster corresponding to the voiceprint feature vector; The detection module 16 is used to detect key points of different facial parts in each frame of the video stream data if a voiceprint cluster corresponding to the voiceprint feature vector exists in the voiceprint vector library, and obtain facial key points; the facial key points include mouth key points. The judgment module 17 is used to determine the degree of mouth opening and closing in each frame image based on the facial key points, and to determine whether the current user is the current speaker based on the degree of mouth opening and closing and the start and end time points of the speech stream data. The identity recognition module 18 is used to query the target user identifier that is associated with the face cluster and the voiceprint cluster from the relationship graph database if the current user is the current speaker, and determine the identity of the current user based on the target user identifier.

[0057] The specific workflow of each of the above modules can be found in the relevant content disclosed in the foregoing embodiments, and will not be repeated here.

[0058] Furthermore, embodiments of this application also disclose an electronic device, Figure 8 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0059] Figure 8 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the identity recognition method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0060] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0061] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0062] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the identity verification method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0063] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed identity recognition method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0064] Furthermore, embodiments of this application also disclose a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the aforementioned identity recognition method.

[0065] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0066] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0067] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0068] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0069] The present application provides a detailed description of an identity recognition method, apparatus, device, and storage medium. Specific examples have been used to illustrate the principles and implementation methods of the present application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present application. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present application. Therefore, the content of this specification should not be construed as a limitation of the present application.

Claims

1. An identity recognition method, characterized by, Applications in voice interaction devices, including: Under preset authorization conditions, the system acquires video stream data of the current user appearing within a preset camera area, as well as voice stream data generated during the dialogue between the current user and the voice dialogue device; the preset authorization conditions are that the voice dialogue device has pre-acquired authorization from each user within the preset camera area for the collection and recognition of facial and voice information. The video stream data is subjected to facial feature extraction using a facial vectorization model to obtain facial feature vectors, and a pre-created facial vector library is used to query whether a facial cluster corresponding to the facial feature vector exists. If a face cluster corresponding to the face feature vector exists in the face vector library, then the voiceprint feature is extracted from the speech stream data through the voiceprint vectorization model to obtain the voiceprint feature vector, and the voiceprint feature vector is obtained by querying the pre-created voiceprint vector library to see if a voiceprint cluster corresponding to the voiceprint feature vector exists. If a voiceprint cluster corresponding to the voiceprint feature vector exists in the voiceprint vector library, then key points of different facial parts in each frame of the video stream data are detected to obtain facial key points; the facial key points include mouth key points. Based on the facial key points, the degree of mouth opening and closing in each frame image is determined, and based on the degree of mouth opening and closing and the start and end time points of the speech stream data, it is determined whether the current user is the current speaker. If the current user is the current speaker, then the target user identifier that is associated with the face cluster and the voiceprint cluster is queried from the relationship graph database, and the identity of the current user is determined based on the target user identifier.

2. The identity recognition method according to claim 1, characterized in that, The step of extracting voiceprint features from the speech stream data using a voiceprint vectorization model to obtain a voiceprint feature vector includes: The speech stream data is subjected to speech noise reduction processing to obtain noise-reduced speech stream data; The noise-reduced speech stream data is segmented using a speech activity detection model to obtain multi-segmented speech stream data. The voiceprint feature vector is obtained by sequentially extracting voiceprint features from the segmented speech stream data using a voiceprint vectorization model.

3. The identity recognition method according to claim 2, characterized in that, The process of extracting voiceprint features from the segmented speech stream data sequentially using a voiceprint vectorization model to obtain voiceprint feature vectors includes: Determine whether the duration of the segmented speech stream data is less than the first duration; If the duration of the segmented speech stream data is not less than the first duration, then it is determined whether the amplitude of the segmented speech stream data is lower than a preset amplitude threshold. If the amplitude of the segmented speech stream data is not lower than a preset amplitude threshold, then the speech stream data is processed by a voiceprint vectorization model to extract voiceprint features and obtain a voiceprint feature vector.

4. The identity recognition method according to claim 3, characterized in that, The step of querying the pre-created face vector library to see if a face cluster corresponding to the face feature vector exists includes: The similarity between the face feature vector and the face feature vector in each face cluster in the face vector database is calculated sequentially to obtain the face similarity, and it is determined whether the face similarity exceeds the first similarity threshold. If the face similarity exceeds the first similarity threshold, it is determined that there is a face cluster in the face vector library that corresponds to the face feature vector; Accordingly, the step of querying the pre-created voiceprint vector library to see if a voiceprint cluster corresponding to the voiceprint feature vector exists includes: The similarity between the voiceprint feature vector and the voiceprint feature vector in each voiceprint cluster in the voiceprint vector library is calculated sequentially to obtain the voiceprint similarity, and it is determined whether the voiceprint similarity exceeds the second similarity threshold. If the voiceprint similarity exceeds the second similarity threshold, it is determined that there exists a voiceprint cluster in the voiceprint vector library that corresponds to the voiceprint feature vector.

5. The identity recognition method according to claim 4, characterized in that, Also includes: If the face similarity corresponding to all face clusters in the face vector library does not exceed the first similarity threshold, it is determined that there is no face cluster in the face vector library corresponding to the face feature vector, and a first cluster identifier corresponding to the face feature vector is created in the face vector library, and then the face feature vector is bound to the first cluster identifier. If the similarity of the voiceprints corresponding to all voiceprint clusters in the voiceprint vector library does not exceed the second similarity threshold, it is determined that there is no voiceprint cluster in the voiceprint vector library corresponding to the voiceprint feature vector, and a second cluster identifier corresponding to the voiceprint feature vector is created in the voiceprint vector library, and then the voiceprint feature vector is bound to the second cluster identifier. In the relational graph database, a first node and a second node corresponding to the first cluster identifier and the second cluster identifier are created respectively. A target user identifier is assigned to the current user, and an association relationship between the first node and the second node is established based on the target user identifier.

6. The identity recognition method according to claim 3, characterized in that, Before extracting voiceprint features from the speech stream data using a voiceprint vectorization model to obtain the voiceprint feature vector, the method further includes: The number of face feature vectors in the face cluster corresponding to the face feature vector is counted to obtain the number of face vectors, and it is determined whether the number of face vectors exceeds a first number threshold. If the number of face vectors does not exceed the first number threshold, then the face feature vectors are added to the face cluster; Accordingly, before querying the target user identifier that is associated with the face cluster and the voiceprint cluster from the relationship graph database, the process further includes: The number of voiceprint feature vectors in the voiceprint cluster corresponding to the voiceprint feature vector is counted to obtain the number of voiceprint vectors, and it is determined whether the number of voiceprint vectors exceeds the second quantity threshold. If the number of voiceprint vectors exceeds the second quantity threshold, the voiceprint feature vectors in the voiceprint cluster are replaced by the voiceprint feature vectors according to the time priority rule. If the number of voiceprint vectors does not exceed the second number threshold, then determine whether the duration of the segmented speech stream data exceeds the second duration. If the duration of the segmented speech stream data exceeds the second duration, the voiceprint feature vector is added to the voiceprint cluster.

7. The identity recognition method according to any one of claims 2 to 6, characterized in that, The step of detecting key points of different facial features in each frame of the video stream data to obtain facial key points includes: Face detection is performed on each frame of the video stream data using a preset face detection model to obtain face detection results; the face detection results include face detection boxes and detection box confidence scores. The face detection results are input into the target tracking model so that face detection boxes whose positions meet the preset proximity conditions between consecutive frames in the video stream data are associated across frames using track_id. The face detection results are input into a 3D key point detection model to detect key points of different face parts within each face detection box, thereby obtaining facial key points; the facial key points include mouth key points. Accordingly, determining the degree of mouth opening and closing in each frame image based on the facial key points, and determining whether the current user is the current speaker based on the degree of mouth opening and closing and the start and end times of the speech stream data, includes: Based on the facial key points, the degree of mouth opening and closing of each face in the corresponding frame image is calculated to obtain the mouth opening and closing value; The maximum and minimum values ​​are obtained from the mouth opening and closing value sequence by means of a sliding window voting method and based on the track_id; the mouth opening and closing value sequence is all the mouth opening and closing values ​​corresponding to the mouth region of the same face in different frame images in the video stream data; Calculate the difference between the maximum value and the minimum value to obtain a first difference, and determine whether the first difference is less than a preset threshold; If the first difference is not less than a preset threshold, it is determined that the current user is in a dialogue state, and the number of all frame images in the dialogue state is counted to obtain the statistical result. Determine whether the time period length corresponding to the statistical result matches the second difference; the second difference is the time difference between the start and end times of the speech stream data after segmentation. If the time period length corresponding to the statistical result matches the second difference, then the current user is determined to be the current speaker; If the time period length corresponding to the statistical result does not match the second difference, then it is determined that the current user is not the current speaker.

8. An identity recognition device, characterized in that, Applications in voice interaction devices, including: The acquisition module is used to acquire video stream data of the current user appearing in a preset camera area, as well as voice stream data generated during the dialogue between the current user and the voice dialogue device, under the condition of meeting the preset authorization. The preset authorization condition is that the voice dialogue device has pre-acquired authorization from each user in the preset camera area for the collection and recognition of facial and voice information. The face feature extraction module is used to extract face features from the video stream data using a face vectorization model to obtain face feature vectors; The first query module is used to query from a pre-created face vector library whether a face cluster corresponding to the face feature vector exists; The voiceprint feature extraction module is used to extract voiceprint features from the speech stream data through a voiceprint vectorization model if a face cluster corresponding to the face feature vector exists in the face vector library. The second query module is used to query from a pre-created voiceprint vector library whether there is a voiceprint cluster corresponding to the voiceprint feature vector; The detection module is used to detect key points of different facial parts in each frame of the video stream data if a voiceprint cluster corresponding to the voiceprint feature vector exists in the voiceprint vector library, thereby obtaining facial key points; the facial key points include mouth key points. The judgment module is used to determine the degree of mouth opening and closing in each frame image based on the facial key points, and to determine whether the current user is the current speaker based on the degree of mouth opening and closing and the start and end time points of the speech stream data. The identity recognition module is used to query the target user identifier that is associated with the face cluster and the voiceprint cluster from the relationship graph database if the current user is the current speaker, and determine the identity of the current user based on the target user identifier.

9. An electronic device, characterized in that, It includes a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the identity recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the identity recognition method as described in any one of claims 1 to 7.

Images