System and method for target speaker identification using voiceprint and visual recognition

Abstract

A system and method for targeting a speaker using voiceprint and visual recognition include capturing live audio through a recording device and acquiring regional images using a camera device. A voiceprint recognition module separates multiple voices from the audio and synchronizes them with the images. Lip movement recognition technology is used to analyze and identify the voice that matches the lip movements in the image, marking it as the target speaker. The camera device is then notified to lock onto the target speaker, while the recording device filters out irrelevant audio. A dialogue module initiates a conversation with the target speaker by playing dialogue model-generated speech through a speaker.

Description

Technical Field

[0001] A dialogue system that utilizes voice, particularly a system and method for identifying a target speaker using voiceprint and visual recognition. Prior Technology

[0002] In recent years, with the development of deep learning technology, significant progress has been made in Automatic Speech Recognition (ASR) and Speaker Identification / Verification. However, in situations where multiple people are speaking simultaneously, the problem of not being able to "lock in" the voice of the same user still exists.

[0003] In terms of automated order generation technology, fixed models are typically used to generate orders for both individual and repeat customers, and natural language processing is mostly employed to handle speech. However, this method is prone to errors when multiple people are speaking simultaneously and fails to be customized based on the ordering habits of repeat customers.

[0004] The purpose of this invention is to propose a technology specifically designed to "lock onto a single voiceprint and ignore other sounds," combining voiceprint recognition, audio separation, and speech recognition mechanisms to improve recognition accuracy in multi-source environments. Simultaneously, based on each customer's order habits, deep learning technology is used to generate customized dialogue models that meet the needs of different customers. Summary of the Invention

[0005] In a multi-person environment, this invention not only identifies the source of the sound but also recognizes whether the speaker is engaging in conversation with the "target interlocutor" based on their lip movements. The system enhances recognition accuracy through two channels: sound (speech recognition and voiceprint recognition) and image (visual recognition and lip-syncing).

[0006] This system includes: voiceprint modeling: extracting the user's voice features into comparable vectors (such as x-vectors and d-vectors); dynamic sound source separation or filtering: after identifying the most similar target voiceprint, the system automatically amplifies or separates the target sound source while suppressing other sound sources; and real-time speech recognition: the separated target speech is input into an ASR (Automatic Speech Recognition) module for transcription, and the results are applied to human-computer dialogue or subsequent applications.

[0007] A system and method for identifying target interlocutors using voiceprint and visual recognition. This system is primarily an artificial intelligence system used to identify specific individuals for dialogue through voiceprint and visual recognition technologies. It includes: a camera: capturing images of the entire area, including multiple seats, each with a corresponding number; a microphone: collecting all sounds from the scene, including voices from all seats; a speaker: playing the dialogue content to the target based on a dialogue model; a server including voiceprint recognition: separating different people's voices from the recorded sound and comparing them with voiceprints in a database. If the similarity reaches a certain standard, the system further uses lip-reading recognition to confirm whether the voice comes from the corresponding lip movements in the image, thus identifying the target interlocutor; seat recognition: using image recognition technology to determine the interlocutor's seat number based on the seat position in the image; a dialogue module: allowing the AI ​​to interact with the interlocutor through the speaker, while the recording device transmits the dialogue back to the server; and a database: storing a seating area map, voiceprint data, and dialogue models for system operation. Order generation: By analyzing the conversation through natural language processing, the system automatically generates orders and records the seat numbers. In short, this system can accurately locate specific targets through voice and image recognition, engage in conversations with them, and even automatically process orders. It's an interactive system that combines AI and intelligent recognition technology.

[0008] This system utilizes deep learning techniques, such as convolutional neural networks (CNNs) or temporal convolutional networks (TCNs), to identify changes in lip shape and map them to speech signals. These models are able to extract lip movements from images and convert them into corresponding speech content.

[0009] This system pre-registers members' voiceprints and performs visual registration for each member. This allows the system to simultaneously verify the member's identity when audio and video data are combined.

[0010] When the target speaker begins to speak, the voiceprint recognition module detects that the voice belongs to the member. At this point, if the image detects that the speaker's lip shape matches the member's facial features, the accuracy of the target speaker's recognition is further improved.

[0011] This system employs deep learning-based audio source separation technology: guided by voiceprints, it separates audio into multiple channels, then uses voiceprint comparison to determine which channel's sound best matches the target voiceprint, while other channels are ignored or suppressed. The system continuously and dynamically updates separation parameters; if it detects a decrease in voiceprint similarity across different time segments, it automatically adjusts or initiates a re-comparison mechanism.

[0012] One embodiment of this system uses voiceprint and visual recognition technology to identify a speaking customer in an automated ordering scenario. It works as follows: First, a camera captures footage of the restaurant, including multiple seats, each with a corresponding number. Simultaneously, a microphone records the ambient sound, which may come from different customers. Next, the system uses voiceprint recognition technology to separate the different voices and compares which voices match the criteria for "ordering voiceprints" (i.e., the sound of someone possibly ordering). If a match is found, the system further uses lip-reading recognition technology to identify the person whose lip movements match the voice in the image, thus accurately identifying the actual conversation partner. Then, the system compares the conversation partner's seat location to confirm the corresponding seat number. Once the target conversation partner is identified, the system initiates a conversation with the customer through a speaker to take the order. The audio of the entire conversation is simultaneously transmitted to the server for real-time processing. Finally, the system uses natural language processing technology to analyze the order, automatically generate an order, and add the corresponding seat number to the order, ensuring the food is delivered to the correct seat. This allows restaurants to achieve a more efficient and accurate automated ordering process, significantly improving the customer experience. Simple Explanation of the Diagram

[0013] [Figure 1] Schematic diagram of a system that uses voiceprint and visual recognition to locate the target speaker; [Figure 2] Schematic diagram of an automated dialogue device that uses voiceprint and visual recognition to locate the target speaker; [Figure 3] Schematic diagram of the system environment area for locating the target speaker using voiceprint and visual recognition; [Figure 4] Flowchart of an embodiment of a method for identifying a target speaker using voiceprint and visual recognition; [Figure 5] Flowchart of another embodiment of the method for identifying a target speaker using voiceprint and visual recognition; [Figure 6] Schematic diagram of another embodiment of a system environment that uses voiceprint and visual recognition to locate the target speaker; [Figure 7] Flowchart of the method for using voiceprint and visual recognition to locate the target interlocutor in an automated ordering scenario. Implementation

[0014] The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numerals will be used in the drawings and the following description to refer to the same or similar parts. Although several illustrative embodiments are described herein, modifications, adjustments, and other implementations of these embodiments are possible. For example, the apparatus and steps shown in the drawings can be substituted, added, or modified, and the illustrative methods described herein can be modified by substituting, reordering, removing, or adding steps. Therefore, the following detailed description is not limited to the disclosed embodiments, and the appropriate scope of the invention should be defined by the claims of the appended patent application.

[0015] In this specification, the terms "device," "device," "module," "model," "database," "server," or "system," etc., refer to a combination of hardware and software executed by the corresponding hardware. For example, hardware can be a data processing device, such as a mobile device and personal computer with a built-in central processing unit or other processor. Furthermore, software executed by hardware can refer to executable programs, objects, executable files, threads, and programs, etc.

[0016] Those skilled in the art should possess basic knowledge of computer architecture and organization, and therefore understand that this invention is not limited to a specific form of computer or network connection architecture. The server in this invention can be composed of multiple computers, as long as they can provide the corresponding functions. A computer includes a processor, a memory module connected to the processor, and a server network interface. The processor can be used to execute operating systems and applications stored in memory, including database management systems (DBMS), web servers, and / or web application servers, to implement the multiple steps in the embodiments described in this invention. This invention as a whole can be implemented through a website platform, an application (APP), or other means.

[0017] A system and method for identifying a target speaker using voiceprint and visual recognition, as shown in Figure 1, includes: a camera device 22 capturing an area image 4; a sound recording device 21 capturing ambient sound; and a server 1 connected to the camera device 22 and the sound recording device 21. The server 1 uses voiceprint recognition technology to separate multiple individual voices from the ambient sound and compares these voices with a voiceprint stored in a database 3 to see if they exceed a threshold. If so, it uses lip-reading recognition technology to analyze whether the lip-reading of these voices matches that in the area image 4, and marks the lip-reading matching in the area image 4 as a target speaker. The server 1 sends a notification to the camera device 22 and the sound recording device 21 to identify the target speaker, and the camera device 22 then identifies the target speaker. The sound recording device 21 filters out other sounds besides the voice corresponding to the target speaker.

[0018] A system and method for identifying a target speaker using voiceprint and visual recognition, as shown in Figure 2, includes: an automated dialogue device 2, comprising: a camera 22 capturing an area image 4, the area image 4 including a plurality of seats, each seat corresponding to a seat number 5; a microphone 21 capturing ambient sound at a listening distance covering the seats; a speaker 23 emitting dialogue voice to the seats according to a dialogue model; a database 3 connected to the automated dialogue device 2, the database 3 storing a seat area map, a recognized voiceprint, and the dialogue model; and a server 1 connected to the automated dialogue device 2 and the database 3, the server 1 comprising: A voiceprint recognition module 11 uses voiceprint recognition technology to separate multiple individual voices from the ambient sound and calculates whether the difference between these voices and the recognized voiceprint is greater than a threshold. If so, it further uses lip-shape recognition technology to analyze whether the voices match the lip shapes in the area image 4 and marks the voices that match the lip shapes in the area image 4 as a target conversation partner. A table recognition module 12 compares the area image 4 and the target conversation partner with the seating area map and uses image recognition technology to calculate a seat number. A dialogue module 13 transmits the voice of a dialogue model to the automatic dialogue device 2. The speaker 23 of the automatic dialogue device 2 initiates a dialogue with the target conversation partner based on the voice of the dialogue model. The sound receiving device 21 transmits the dialogue process to the server 1. An order generation module 14 uses natural language processing technology to analyze the dialogue process and generate an order, and further adds the seat number to the order.

[0019] A system and method for identifying a target interlocutor using voiceprint and visual recognition, as shown in Figure 3, includes: a camera device 22 capturing a region image 4, the region image 4 including a plurality of seats, each seat corresponding to a seat number 5; a sound receiving device 21 capturing ambient sound at a listening distance covering the seats; a speaker 23 emitting dialogue sound to the seats according to a dialogue model; and a server 1 connecting the camera device 22, the sound receiving device 21, and the speaker 23, the server 1 including: A voiceprint recognition module 11 uses voiceprint recognition technology to separate multiple individual voices from the ambient sound and calculates whether the difference between these voices and a recognized voiceprint is greater than a threshold. If so, it further uses lip-shape recognition technology to analyze whether these voices match the lip shapes in the area image 4, and marks those voices that match the lip shapes in the area image 4 as a target conversation partner. A table recognition module 12 compares the area image 4 and the target conversation partner with a seating area map and uses image recognition technology to calculate a seat number. A dialogue module 13 transmits the voice of a dialogue model to the speaker 23, and initiates a dialogue with the target interlocutor through the speaker 23. The sound receiving device 21 transmits the dialogue process to an order generation module 14 of the server 1. The order generation module 14 uses a natural language processing technology to analyze the dialogue process and generate an order, and further adds the seat number to the order. A database 3 is connected to the server 1, and the database 3 stores the seat area map, the identified voiceprint, and the dialogue model.

[0020] The system also includes a member identification module, which compares the voice of the target conversationalist with the voiceprint of a member stored in the database 3. If not, it guides the user into a registration model, which includes a registration process for the member's voiceprint and facial features, and stores the member's voiceprint and facial features in the member's data.

[0021] If so, the member identification module sends the member information corresponding to the voice to the dialogue module 13, and the dialogue module 13 transmits the voice of the dialogue model corresponding to the member information to the speaker 23.

[0022] In one embodiment, if the voice and facial image of the target interlocutor are in the member information, the member recognition module sends the member information corresponding to the voice and facial image to the dialogue module 13, and the dialogue module 13 transmits the voice of the dialogue model corresponding to the member information to the speaker 23.

[0023] The system also includes an AI training module. The dialogue module 13 stores a dialogue record for each member in the database 3. The AI ​​training module uses deep learning technology to generate a customized dialogue model for each member based on the dialogue record.

[0024] The image recognition technology includes identifying the target interlocutor in the area image 4, then overlaying the feature points of the area image 4 and the seating area map, and calculating the seat number of the target interlocutor in the seating area map.

[0025] One embodiment discloses a system for identifying a target speaker using voiceprint and visual recognition, comprising a voiceprint / image recognition module that combines image recognition and voiceprint recognition. The system includes: A camera device 22 captures an area image 4; a sound recording device 21 captures ambient sound; a speaker 23 emits dialogue sounds to the seats according to a dialogue model; a voiceprint / image recognition module is connected to the camera device 22, the sound recording device 21, and the speaker 23. The voiceprint / image recognition module uses voiceprint recognition technology to separate multiple individual voices from the ambient sound and compares these voices with a recognized voiceprint to see if the difference exceeds a threshold. If so, it uses lip-reading recognition technology to analyze whether the lip-reading of these voices matches that in the area image 4 and marks the lip-reading of the person matching the lip-reading in the area image 4 as a target conversationalist; a dialogue module 13 transmits the voice of a dialogue model to the speaker 23 and initiates the dialogue model to the target conversationalist through the speaker 23; the sound recording device 21 transmits the dialogue process to the voiceprint / image recognition module; and an order generation module 14 uses natural language processing technology to analyze the dialogue process and generate an order.

[0026] The server 1 can connect to the automated dialogue device 2 or the speaker 23 via wired or wireless connection.

[0027] The automated dialogue device 2 also includes a display 24 on which the dialogue model is displayed in text form.

[0028] The photographic device 22 includes a monitor, a camera, and electronic equipment capable of capturing images and transmitting them.

[0029] The recording device 21 includes an omnidirectional microphone, a directional microphone, and a device for transmitting sound using electronic signals.

[0030] The system also includes a printing device, and the order generation module transmits the order to the printing device for output.

[0031] A method for identifying a target speaker using voiceprint and visual recognition, the method comprising: a server 1 receiving a live sound captured by a recording device 21 and a region image 4 captured by a camera device 22; the server 1 using voiceprint recognition technology to separate multiple individual voices from the live sound, further calculating whether the difference between these voices and an identified voiceprint is greater than a threshold; if it is greater than the threshold, the server 1 synchronizing the voices and the region image 4 in a time sequence, using lip-reading recognition technology to analyze the voices and the region image 4 and marking those whose lip movements match as a target speaker; and the server 1 transmitting a target speaker identification notification to the camera device 22 and the recording device 21, the camera device 22 identifying the target speaker, and the recording device 21 filtering out other sounds besides the voice corresponding to the target speaker.

[0032] A system and method for identifying a target interlocutor using voiceprint and visual recognition, as shown in Figure 4, includes the following steps: S10 A server 1 receives real-time ambient sound from a sound receiving device 21 and real-time area image 4 from a photographic device 22 in an automatic dialogue device 2; S20 A voiceprint recognition module 11 of the server 1 uses voiceprint recognition technology to separate multiple individual voices from the ambient sound; S30 The voiceprint recognition module 11 calculates whether the difference between each of the voices and a recognized voiceprint stored in a database 3 is greater than a threshold; S40 If the difference is less than or equal to the threshold, the voice is ignored; if the difference is greater than the threshold, the following steps are continued; S50 The voiceprint recognition module 11 synchronizes the voices and the area image 4 sequentially, and further uses lip-syncing technology to analyze the voices and the area image 4 for those with matching lip shapes, marking those with matching lip shapes as a target interlocutor of the voiceprint; S60 The voiceprint recognition module 11 sends a notification to the automated dialogue device 2 to lock onto the target conversationalist; S70 The automated dialogue device 2 locks onto the target conversationalist using the camera device 22 and filters out other sounds besides the voice corresponding to the target conversationalist using the microphone device 21 according to the notification; S80 A table recognition module 12 of the server 1 compares the area image 4 and the notification to lock onto the target conversationalist with a seating area map stored in the database 3 and calculates a seat number corresponding to the target conversationalist; S90 A dialogue module 13 of the server 1 sends a dialogue model to the automated dialogue device 2, and a speaker 23 of the automated dialogue device 2 initiates a dialogue with the target conversationalist according to the dialogue model; S100 The automated dialogue device 2 transmits the real-time dialogue process to an order generation module 14 of the server 1; S110 The order generation module 14 analyzes the dialogue process using natural language processing (NLP) and generates an order, further adding the seat number to the order.

[0033] In one embodiment, the above process further includes S41: If there are multiple voices greater than the threshold, the voiceprint recognition module 11 selects the single voice with the highest weight according to a weight.

[0034] In one embodiment, the weight calculation method includes the weight being directly proportional to the volume of the sound, the weight being inversely proportional to the difference between the human voice and the identified voiceprint, and the weight of a member being higher than that of a non-member.

[0035] In one embodiment, the above process further includes S51, a member identification module comparing whether the voice of the target interlocutor matches a member voiceprint stored in the database 3; S52, if the voice does not match the member voiceprint, the member identification module notifies the dialogue module 13 that the transmitted dialogue model is a registration model; S65, if the voice matches the member voiceprint, the member identification module notifies the dialogue module 13 that the transmitted dialogue model is the dialogue model corresponding to the member voiceprint.

[0036] In one embodiment, the above process further includes S53: the member recognition module captures a facial image of the target conversationalist in the image 4 of the region and compares it with a member's facial features stored in the database 3; S54: if the facial image does not match the member's facial features, the member recognition module notifies the automatic dialogue device 2 to enter the registration model.

[0037] The registration model includes the server 1's dialogue module 13 transmitting one of the dialogue models to the automatic dialogue device 2, and the automatic dialogue device 2 initiating a registration dialogue with the target dialogue user according to the registration dialogue model.

[0038] In one embodiment of the registration dialogue model, for example, the speaker of the automatic dialogue device 2 emits a voice message such as "Please tell me the guest's name, birthday, taste preferences, etc..." related to the member's information. After the guest answers, the voice recorder 21 records the message and transmits it to the database 3 to store the member's information.

[0039] The automated dialogue device 2 initiates the dialogue with the target interlocutor and also includes a display 24 showing the dialogue model.

[0040] In another embodiment of the registration dialogue model, for example, the display 24 of the automatic dialogue device 2 displays the relevant information of the member, such as "Please tell me the guest's name, birthday, taste preferences, etc..." After the guest answers, the recording device 21 records the information and transmits it to the database 3 to store the member information.

[0041] The registered conversation model contains a member's facial features of a registered guest's facial features retrieved by the photographic device 22 , and a member's voice striation characteristic of a registered guest's voice retrieved from the radio device 21 , together with the storage to the member data.

[0042] The conversation process for each member is stored into a corresponding conversation record, according to which the AI ​​training module uses deep learning technology to generate a customized conversation model that conforms to each member.

[0043] A system and method of locking a target interlocutor using voiceprint and visual recognition is shown in Figure 4 , which comprises: A10 a server 1 receives a live tone retrieved by a radio-receiving device 21 and an image of an area 4 retrieved by a photographic device 22; The voice pattern recognition module 11 computes whether the difference between those human voices and one recognition voice pattern stored in a database 3 is greater than a threshold;A40 if less than or equal to the threshold, the human voice is ignored; The voice pattern recognition module 11 synchronizes those human voices with the image of the area 4 in chronological order, further analyzes those voices consistent with the mouth shape in the imagery 4 of the area using one-mouth recognition technique, and labels them as a target interlocutor; The photography device 22 locks the target interlocutor and the radio device 21 filters out other sounds other than the human voice corresponding to the target interlocutor according to the notification of the locked target interlocutor; A dialog module 13 of the server 1 transmits the speech of a dialog model to a loudspeaker 23 , which initiates a dialog to the target interlocutor according to the dialog model;

[0044] The recognition voice pattern contains a keyword and a plural of item names, which can be but not only: I order, order, boss, waiter, etc., and some commonly used order generation antecedents.

[0045] This dialogue model includes inquiries about order items, specifications, quantity, price, discounts, delivery time, special requirements, and greetings.

[0046] An example of restaurant ordering includes a dialogue model that includes inquiries about main courses, side dishes, appetizers, soups, desserts, seasonings (sweet, sour, bitter, spicy, salty, etc.), special requests, and greetings.

[0047] The member's voiceprint is extracted in a fixed frame length (e.g., 500ms~1s) to capture short speech segments, and the same "voiceprint extraction model" is input to generate an instant voiceprint vector.

[0048] The sound transmitted by the recording device 21 and the image of the area transmitted by the photography device 22 contain a timestamp.

[0049] The member information includes name, birthday, ID, member number, preferences, order generation habits, member voiceprint, member facial features, etc.

[0050] As shown in Figures 2 and 5, the automatic dialogue device 2 includes a fixed device, a portable device, and an automatically mobile device.

[0051] One embodiment discloses a system and method for identifying a target speaker using voiceprint and visual recognition. The method includes a server 1 receiving ambient sound captured by a recording device 21 and a region image 4 captured by a photographic device 22; a voiceprint recognition module 11 of the server 1 using voiceprint recognition technology to separate multiple individual voices from the ambient sound; the voiceprint recognition module 11 calculating whether the difference between the individual voices and a recognized voiceprint is greater than a threshold; if the difference is less than or equal to the threshold, the individual voice is ignored; the voiceprint recognition module 11 synchronously synchronizes the individual voices with the region image 4 in a time sequence, and uses lip-reading recognition technology to analyze the lip shapes in the individual voices and the region image 4. The voiceprint recognition module 11 sends a notification to the camera device 22 to lock onto the target speaker, and the camera device 22 locks onto the target speaker. The microphone device 21 filters out other sounds besides the voice corresponding to the target speaker. A dialogue module 13 of the server 1 transmits the voice of a dialogue model to a speaker 23, and the speaker 23 initiates a dialogue with the target speaker according to the dialogue model. The microphone device 21 transmits the dialogue process to an order generation module 14 of the server 1. The order generation module 14 analyzes the dialogue process using natural language processing technology and generates an order.

[0052] One embodiment discloses a method for identifying a target speaker using voiceprint and visual recognition, comprising a voiceprint / image recognition module that combines image recognition and voiceprint recognition. The method includes: the voiceprint / image recognition module receiving ambient sound captured by a recording device 21 and a region image 4 captured by a photographic device 22; the voiceprint / image recognition module using voiceprint recognition technology to separate multiple individual voices from the ambient sound; the voiceprint / image recognition module calculating whether the difference between these voices and a recognized voiceprint is greater than a threshold; if it is greater than the threshold, the voiceprint / image recognition module sequentially synchronizes the voices with the region image 4, and analyzes the voices using lip-syncing technology. A voice matching the lip movements in the image 4 of the area is identified as a target conversationalist; the voiceprint / image recognition module sends a notification to the camera device 22 to lock onto the target conversationalist, and the camera device 22 locks onto the target conversationalist and the sound receiving device 21 filters out other sounds besides the voice corresponding to the target conversationalist; a dialogue module 13 transmits the voice of a dialogue model to a speaker 23, and the speaker 23 initiates a dialogue with the target conversationalist according to the dialogue model; the sound receiving device 21 transmits the dialogue process to an order generation module 14; and the order generation module 14 analyzes the dialogue process using a natural language processing technology and generates an order.

[0053] One embodiment discloses a system for identifying a target interlocutor using voiceprint and visual recognition, applied in an automated ordering scenario. The system includes: an automated dialogue device 2, comprising: a camera 22 capturing an area image 4 containing a plurality of dining seats, each seat corresponding to a seat number 5; a microphone 21 capturing ambient sound over a listening distance covering the dining seats; a speaker 23 emitting dialogue sounds to the dining seats according to a dialogue model; a database 3 connected to the automated dialogue device 2, storing a restaurant seating area map, an ordering voiceprint, and the dialogue model; and a server 1 connected to the automated dialogue device 2 and the database 3, the server 1 comprising: A voiceprint recognition module 11 uses voiceprint recognition technology to separate multiple individual voices from the ambient sound and calculates whether the difference between these voices and the voiceprint used for ordering food is greater than a threshold. If so, it further uses lip-syncing technology to analyze whether the voices match the lip-syncing in the area image 4 and marks the voices that match the lip-syncing in the area image 4 as a target conversation partner. A table recognition module 12 compares the area image 4 and the target conversation partner with the restaurant seating area map and uses image recognition technology to calculate a seat number. An ordering dialogue module 13 transmits the voice of a dialogue model to the automatic dialogue device 2. The speaker 23 of the automatic dialogue device 2 initiates an ordering dialogue with the target conversation partner based on the voice of the dialogue model. The sound receiving device 21 transmits the ordering dialogue process to the server 1. An order generation module 14 uses natural language processing technology to analyze the ordering dialogue process and generate an order, and further adds the seat number to the order.

[0054] One embodiment discloses a method for identifying a target conversationalist using voiceprint and visual recognition, applied in an automated ordering scenario, as shown in Figure 7. The method includes: B10 A server 1 receives real-time ambient sound from a recording device 21 and real-time area image 4 from a camera device 22 in an automated dialogue device 2; B20 A voiceprint recognition module 11 of the server 1 uses voiceprint recognition technology to separate multiple individual voices from the ambient sound; B30 The voiceprint recognition module 11 calculates whether the difference between each of these voices and a customer ordering voiceprint stored in a database 3 is greater than a threshold; B40 If the difference is less than or equal to the threshold, the voice is ignored; if it is greater than the threshold, the following steps continue; B50 The voiceprint recognition module 11 synchronizes the voices and the area image 4 sequentially, and further uses lip-syncing technology to analyze the voices and the area image 4 for those with matching lip shapes, marking those with matching lip shapes as a target conversationalist for the voiceprint; B60 The voiceprint recognition module 11 sends a target conversation notification to the automated dialogue device 2; B70 The automated dialogue device 2, based on the target conversation notification, locks the camera device 22 onto the target conversation participant and the microphone device 21 filters out other sounds besides the voice corresponding to the target conversation participant; B80 A table recognition module 12 of the server 1 compares the area image 4 and the target conversation participant notification with a restaurant seating area map stored in the database 3, and calculates a seat number corresponding to the target conversation participant; B90 A dialogue module 13 of the server 1 sends a food ordering dialogue model to the automated dialogue device 2, and a speaker 23 of the automated dialogue device 2 initiates a food ordering dialogue with the target conversation participant based on the food ordering dialogue model; B100 The automated dialogue device 2 transmits the real-time food ordering dialogue process to an order generation module 14 of the server 1; B110 The order generation module 14 uses natural language processing (NLP) to analyze the food ordering dialogue process and generate an order, and further adds the seat number to the order.

[0055] The target speaker identification method involves continuously extracting short speech segments from the input audio during system operation and obtaining real-time voiceprint vectors using the same voiceprint extraction model. Next, the similarity (e.g., cosine similarity or Euclidean distance) between this real-time voiceprint vector and the target voiceprint vector is calculated. If the similarity is higher than a threshold, the voice is determined to belong to the target speaker; otherwise, the voice is ignored. If the system has only a single microphone, a deep learning audio source separation model (such as Conv-TasNet or DPRNN) can be used to separate the audio into multiple channels, and the audio source track closest to the target voiceprint is selected based on the voiceprint comparison results. If a multi-microphone array is used, beamforming technology can be combined to enhance the volume of the target sound source based on spatial information while suppressing other sound sources.

[0056] During the conversation, if the voiceprint recognition module 11 continuously detects the voiceprint of the target conversation participant, the automatic conversation device 2 remains in a "locked" state. If the detected voiceprint similarity is consistently too low, it indicates that the target conversation participant has paused speaking or left, and the device enters a waiting state.

[0057] The voiceprint recognition module 11 separates human voices from the ambient sound, a process commonly referred to as source separation or voice separation. Techniques employed include Blind Source Separation (BSS), Independent Component Analysis (ICA), Non-negative Matrix Factorization (NMF), FastICA (Scikit-Learn), or nussl (Python). It also utilizes deep learning models such as Demucs v4 (for speech, music, and noise separation), SepFormer (for multiple speech sources), and Conv-TasNet (commonly used for speech separation).

[0058] This lip reading technology infers speech content by analyzing lip movements. It includes OpenCV: face detection and lip tracking, Dlib: 68 facial landmark annotations, DeepFace: a multi-functional face recognition suite, and S3FD: a high-accuracy face detection model.

[0059] Depending on the application requirements, this system can choose to process the audio in real time on a local edge device or upload the audio to the cloud for deep learning inference and then return the results.

[0060] This system offers the following advantages: 1. Improved accuracy: Through a specialized voiceprint comparison mechanism, the system significantly reduces the identification error rate in multi-person conversations or noisy environments. 2. Real-time and dynamic: The system automatically updates separation parameters and continuously tracks the user's voiceprint status when the target user joins or leaves the conversation. 3. Applicable to various order generation application scenarios, such as smart speakers, meeting recording systems, in-vehicle voice assistants, and remote customer service, enhancing ease of use and privacy security.

[0061] While certain exemplary embodiments and implementations have been described herein, other embodiments and modifications will be apparent from that description. Therefore, the concept of the invention is not limited to these exemplary embodiments, but is covered with a broader scope according to the proposed claims and various obvious modifications and equivalent arrangements.

[0062] 1: Server 2: Automated Dialogue Devices 3: Database 4: Regional Imagery 5: Seat Number 11: Voiceprint Recognition Module 12: Table position recognition module 13: Dialogue Module 14: Order Generation Module 21: Radio device 22: Photographic Device 23: Speaker 24: Monitor A1~A120: Methods and procedures for identifying target speakers using voiceprint and visual recognition B1~B120: Methods and procedures for using voiceprint and visual recognition to locate the target speaker in automated ordering scenarios. S1~S120: Method and process for using voiceprint and visual recognition to locate the target speaker in automated dialogue devices

Claims

1. A system for identifying a target interlocutor using voiceprint and visual recognition, the system comprising: a speaker emitting dialogue sounds according to a dialogue model; a camera capturing an area image; a microphone capturing ambient sound; and a server connected to the camera and the microphone, the server using voiceprint recognition technology to separate multiple individual voices from the ambient sound, and comparing these voices with a stored voiceprint in a database to see if the difference exceeds a threshold; if so, lip-reading recognition technology is used to analyze the voices and identify those whose lip shapes match in the area image, and marking those whose lip shapes match in the area image as a target interlocutor; wherein, The server sends a notification to the camera device and the audio device to lock onto the target conversation partner. The camera device then locks onto the target conversation partner, and the audio device filters out other sounds besides the voice corresponding to the target conversation partner. The server also includes image recognition technology to calculate the target conversation partner's seat number in a restaurant seating area map. The server initiates an ordering conversation with the target conversation partner through the speaker according to the conversation model. The audio device transmits the conversation back to the server. The server also uses natural language processing technology to analyze the conversation and generate an order, further adding the seat number to the order.

2. The system as described in claim 1, wherein if the server compares the voice of the target conversation partner with the threshold of a member's voiceprint stored in the database, and / or compares the facial image of the target conversation partner with the member's facial features stored in the database, the server is directed to a registration model, which includes recording a member's voiceprint and a member's facial features.

3. The system as described in claim 1, wherein the server stores the dialogue process as a dialogue record corresponding to a member's information, and an AI training module uses deep learning technology to generate a customized dialogue model for each member based on the dialogue record.

4. A method for identifying a target speaker using voiceprint and visual recognition, the method comprising: a server receiving ambient sound captured by a recording device and an area image captured by a camera device; the server using voiceprint recognition technology to separate multiple individual voices from the ambient sound, and further calculating whether the difference between the individual voices and an identified voiceprint is greater than a threshold; if it is greater than the threshold, the server sequentially synchronizing the individual voices and the area image, and using lip-reading recognition technology to analyze the individual voices and the lip-reading in the area image and marking them as a target speaker; the server transmitting a target speaker identification notification to the camera device and the recording device, the camera device identifying the target speaker, and the recording device filtering out other sounds besides the voice corresponding to the target speaker; The server initiates an ordering conversation with the target user through a speaker based on a dialogue model. The microphone transmits the conversation back to the server. The server also uses natural language processing technology to analyze the conversation and generate an order. The server also includes image recognition technology to calculate the target user's seat number in a restaurant seating area map and adds the seat number to the order.

5. The method as described in claim 4, wherein the server uses the voiceprint recognition technology to compare whether the voice of the target conversationalist is greater than the threshold of a member's voiceprint stored in the database, and uses an image recognition technology to compare whether a facial feature of the target conversationalist matches a member's facial feature stored in the database. If both are not true, the server is guided to a registration model, which includes recording a member's voiceprint and a member's facial feature. If one of them is true, the conversation model of a member's data corresponding to the member's voiceprint is extracted from the database.

6. As described in claim 5, the server stores the dialogue process as a dialogue record corresponding to the member's information, and an AI training module uses deep learning technology to generate a customized dialogue model for each member based on the dialogue record.

Images