An AI animation digital human device and method for face-to-face communication with people

Abstract

The application discloses an AI animation digital human device and method for face-to-face communication with people, which successfully solves the problem of lack of naturalness and immersion in virtual digital human interaction with users by combining sound positioning, infrared laser ranging and face recognition technology. Through the microphone unit and the infrared laser ranging sensor, the device can accurately locate the sound source position and measure the distance between the object and the display screen, avoiding the limitations of traditional speech recognition technology affected by environmental noise. Combined with YOLO algorithm and FaceNet deep learning model, the device can identify the user's facial features in real time and accurately locate the eye position, realizing the synchronous gaze of the digital human eyes and the user's eyes. When the user moves, the device calculates the accurate physical position of the user and adjusts the digital human eye direction to ensure that the digital human always faces the user.

Description

Technical Field

[0001] This application relates to the field of AI animated digital human technology, and in particular to an AI animated digital human device and method for enabling face-to-face communication with people. Background Technology

[0002] With the rapid development of artificial intelligence technology, the application of virtual characters and human-computer interaction devices is becoming increasingly widespread. In many fields, how to enable virtual digital humans to interact with users in a more natural and realistic way has become a pressing technical challenge.

[0003] While some virtual human interaction devices can locate users through voice recognition and image processing technologies, most devices can only provide a limited interactive experience.

[0004] Therefore, how to enable virtual digital humans to communicate with users more naturally and smoothly, and enhance the sense of immersion and realism, has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides an AI-animated digital human device and method for enabling face-to-face communication with people, aiming to further enhance the immersiveness and realism of users interacting with virtual digital humans.

[0006] In a first aspect, an AI-animated digital human device enabling face-to-face communication with people is provided, the device comprising:

[0007] Display screen used to present AI-animated digital humans;

[0008] Multiple microphone units distributed on the display screen are used to independently collect sound information of users near the display screen;

[0009] Multiple ranging sensors distributed on the display screen are used to measure the distance between the display screen and the user, respectively.

[0010] Multiple cameras distributed on the display screen are used to collect the user's facial information;

[0011] The sound source localization module is used to determine the location of the sound source based on the sound information collected by the multiple microphone units and the distance between the multiple ranging sensors and the user;

[0012] The visual tracking module is used to determine the camera closest to the current sound source location among the multiple cameras based on the sound source location, control the camera to capture the user's face image, and detect facial feature information based on the user's face image; perform subsequent face matching and tracking based on the facial feature information, and control the AI ​​animated digital human to move within the display screen following the user's actual position changes, so that the AI ​​animated digital human and the user maintain a face-to-face effect.

[0013] Optionally, in the above scheme, the display screen is an arc-shaped cylindrical surface.

[0014] Optionally, in the above scheme, the cross-section of the display screen is semi-circular.

[0015] In the above scheme, optionally, the plurality of microphone units correspond one-to-one with the ranging sensor, and the microphone unit and its corresponding ranging sensor are integrated into one unit.

[0016] Optionally, in the above scheme, the plurality of microphone units are respectively installed on the top bezel, bottom bezel, left bezel, and right bezel of the display screen.

[0017] In the above scheme, optionally, there are multiple microphone units located on the top bezel of the display screen and multiple microphone units located on the bottom bezel of the display screen, and the number of microphone units is equal; there are multiple microphone units located on the left bezel of the display screen and multiple microphone units located on the right bezel of the display screen, and the number of microphone units is equal.

[0018] Optionally, in the above scheme, the plurality of cameras are mounted on the upper bezel of the display screen.

[0019] In the above scheme, optionally, all microphone units located on the upper bezel of the display screen are divided into several groups of equal number, and the multiple cameras are distributed at intervals between adjacent groups of microphone units.

[0020] Optionally, in the above scheme, the subsequent face matching and tracking based on facial feature information includes determining the coordinates of the user's eyes based on the characteristics of the facial features, and using a ray selection algorithm that simulates a mouse on a three-dimensional virtual simulation model object to control the AI ​​animated digital human to maintain a face-to-face relationship with the user while keeping the eye-to-eye connection aligned.

[0021] Secondly, a method for controlling the orientation of a virtual digital human face on a display screen is characterized by comprising the following steps:

[0022] Step 1: Acquire the sound source signals received by multiple microphones in the microphone array, and measure the time difference of the signals received by each microphone using the TDOA algorithm;

[0023] Step 2: Calculate the distance difference between the sound source and each receiver in the microphone array based on the time difference;

[0024] Step 3: Based on the measured distance difference, the three-dimensional coordinates of the sound source are solved using the least squares method or the Chan algorithm by measuring the time difference between multiple receivers.

[0025] Step 4: Use an infrared laser rangefinder to measure the physical distance between the object and the display screen, and calculate the user's relative position;

[0026] Step 5: Based on the user's relative position, obtain the face image captured by the camera closest to the user and use the YOLO algorithm to perform face detection on the image and extract the feature points of the user's face;

[0027] Step 6: Based on the extracted facial feature points, use a deep learning model to generate the feature embedding vector of the user's face and calculate the coordinates of the user's eye position;

[0028] Step 7: Merge the sound source coordinates from Step 3 with the distance data from Step 4 and the facial coordinate data from Step 6, and map these data to the physical location of the display screen using a coordinate transformation algorithm;

[0029] Step 8: Based on the mapped physical location, the virtual digital human's eye orientation is adjusted using a simulated mouse control algorithm via the control module to ensure it is facing the user's eyes.

[0030] Step 9: If the user's location changes, repeat the above steps to update in real time, ensuring that the virtual digital human's eyes are always aligned with the user's eyes.

[0031] Compared with the prior art, this application has at least the following beneficial effects:

[0032] Based on further analysis and research into the problems of existing technologies, this application recognizes the lack of a device capable of simultaneously processing sound and visual information and adjusting the digital human's gaze in real time, resulting in a relatively stiff and monotonous user interaction experience. By combining sound localization, infrared laser ranging, and facial recognition technologies, this application successfully solves the problem of the lack of naturalness and immersion in the interaction between virtual digital humans and users mentioned in the background technology. Through an array of microphones and an infrared laser ranging sensor, the device can accurately locate the sound source and measure the distance between the object and the display screen, avoiding the limitations of traditional speech recognition technology caused by environmental noise interference. Combining the YOLO algorithm and the FaceNet deep learning model, the device can recognize the user's facial features in real time and accurately locate the eye position, achieving synchronized eye contact between the digital human and the user. When the user moves, the device calculates the user's precise physical position and adjusts the digital human's eye orientation to ensure that the digital human always maintains a face-to-face relationship with the user. In this way, the virtual digital human not only provides higher interaction accuracy but also works stably in various environments, avoiding the problems of misaligned gaze and stiff interaction in the background technology, greatly enhancing the user's immersion and interactive experience. Overall, the technical solution of this invention enables more natural and realistic interaction between virtual humans and users, solves many shortcomings in the prior art, and improves the effect and practicality of intelligent human-computer interaction. Attached Figure Description

[0033] Figure 1 This is a diagram of an AI-animated digital human device for face-to-face communication with people, provided as an embodiment of this application; the figure shown is a digital human image generated by AI. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0035] In one embodiment, such as Figure 1 As shown, an AI-animated digital human device for face-to-face communication with people is provided, the device comprising:

[0036] Display screen used to present AI-animated digital humans;

[0037] The multiple microphone units distributed on the display screen are used to independently collect the sound information of users near the display screen. The multiple microphone units distributed on the display screen should be interpreted broadly. For example, they can be evenly distributed around the display screen, or they can be set around the display screen in a preset manner, or they can be set on the sides, top and bottom of the display screen in a preset manner. This embodiment does not make specific limitations on this.

[0038] Multiple distance sensors distributed on the display screen are used to measure the distance between the display screen and the user. The multiple distance sensors distributed on the display screen should be interpreted broadly. For example, they can be evenly distributed around the display screen, or they can be set around the display screen in a preset manner, or they can be set on the sides, top and bottom of the display screen in a preset manner. This embodiment does not make specific limitations on this.

[0039] Multiple cameras distributed on the display screen are used to collect the user's facial information;

[0040] The multiple cameras based on the display screen distribution should be interpreted broadly. For example, they can be evenly distributed around the display screen, or they can be set around the display screen in a preset manner, or they can be set on the sides, top and bottom of the display screen in a preset manner. This embodiment does not make specific limitations on this.

[0041] The sound source localization module is used to determine the location of the sound source based on the sound information collected by the multiple microphone units and the distance between the multiple ranging sensors and the user; here, the user refers to the current speaker communicating with the AI ​​animated digital human.

[0042] The visual tracking module is used to determine the camera closest to the current sound source location among the multiple cameras based on the sound source location, control the camera to capture the user's face image, and detect facial feature information based on the user's face image; perform subsequent face matching and tracking based on the facial feature information, and control the AI ​​animated digital human to move within the display screen following the user's actual position changes, so that the AI ​​animated digital human and the user maintain a face-to-face effect.

[0043] In this embodiment, the display screen is an arc-shaped cylindrical surface.

[0044] In this embodiment, the cross-section of the display screen is semi-circular.

[0045] In this embodiment, each of the plurality of microphone units corresponds to one of the ranging sensors, and the microphone unit and its corresponding ranging sensor are integrated into one unit.

[0046] In this embodiment, the plurality of microphone units are respectively installed on the top bezel, bottom bezel, left bezel, and right bezel of the display screen.

[0047] In this embodiment, there are multiple microphone units located on the top bezel and the bottom bezel of the display screen, and the number of microphone units is equal; there are also multiple microphone units located on the left bezel and the right bezel of the display screen, and the number of microphone units is equal.

[0048] In this embodiment, the plurality of cameras are mounted on the upper bezel of the display screen.

[0049] In this embodiment, all microphone units located on the upper bezel of the display screen are divided into several groups of equal number, and the multiple cameras are distributed at intervals between adjacent groups of microphone units.

[0050] In this embodiment, the subsequent face matching and tracking based on facial feature information includes determining the coordinates of the user's eyes based on the characteristics of the facial features, and using a ray selection algorithm that simulates a mouse on a three-dimensional virtual simulation model object to control the AI ​​animated digital human to maintain a face-to-face relationship with the user while keeping the eye-to-eye lines aligned.

[0051] The microphone array is one of the core components of this embodiment, using the TDOA (Time Difference of Arrival) algorithm to accurately locate the sound source. The microphone array consists of multiple microphones mounted around the bezel of the display screen, which cooperate and work synchronously. At each microphone location, an infrared laser rangefinder is installed to measure the physical distance between the object and the screen in real time. The installation of the infrared sensors ensures accurate distance information between the sound source and the screen, supporting subsequent coordinate calculations.

[0052] Using the TDOA algorithm, the microphone array can measure the time difference between the sound emitted by the user and the arrival time at each microphone. Based on these time differences, the system calculates the distance difference between the sound source and each microphone using the formula \(\Delta d = c \cdot \Delta t\), and then converts the time differences into positional differences through a mathematical model, ultimately locating the sound source. This process involves solving for the time differences between multiple receivers using the least squares method or the Chan algorithm to determine the two-dimensional or three-dimensional coordinates of the sound source.

[0053] The system acquires a user's facial image using a camera mounted on the display screen. Face detection is performed using the YOLO (You Only LookOnce) algorithm to extract facial feature points from the image. An embedding vector containing the user's unique facial features is then generated using the deep learning model FaceNet. In this way, the system can accurately identify and locate the user's facial features, particularly the position of the eyes, providing a basis for adjusting the orientation of the digital human's eyes.

[0054] The control module fuses the sound source location obtained through the TDOA algorithm, the physical distance data measured by the infrared laser rangefinder, and the user's facial features extracted by the face recognition module to calculate the user's precise physical position on the screen. Then, based on this data, the control module maps the user's position to the physical coordinates of the display screen using a coordinate transformation algorithm. Next, according to the mapped physical position, the control module uses a simulated mouse control algorithm to adjust the virtual digital human's eye orientation, ensuring it is facing the user's eyes and providing a face-to-face communication effect.

[0055] As the user moves, the microphone array and camera capture changes in the user's position in real time and dynamically adjust using the aforementioned algorithm. The virtual digital human's eyes adjust synchronously with the user's position, thus always maintaining a face-to-face orientation. This feature enables the virtual digital human to interact naturally and smoothly with the user from different angles and distances.

[0056] This embodiment successfully solves the problems of limited and unrealistic virtual digital human interaction experiences in existing technologies by comprehensively utilizing sound localization, infrared laser ranging, and facial recognition technologies, especially in achieving natural interaction effects in face-to-face communication scenarios.

[0057] Utilizing an array of microphones and an infrared laser rangefinder, the system can accurately measure the distance between the sound source and the display screen, ensuring that the digital human can correctly locate the user's position. This technology effectively overcomes the interference of environmental noise on positioning accuracy in traditional voice recognition technology and can work stably under different environmental conditions, providing higher interactive accuracy.

[0058] Using the YOLO algorithm and the FaceNet deep learning model, the system can detect and recognize users' facial features in real time, especially the position of their eyes. This technology improves the accuracy and real-time performance of face recognition, avoiding the recognition failures of traditional face recognition technologies in low-light environments or at incorrect angles. Furthermore, by combining face recognition with sound localization, the system can accurately track users from different positions and angles.

[0059] Through the aforementioned technologies, the interaction between virtual digital humans and users becomes more vivid and natural, effectively eliminating the stiffness and unnaturalness of traditional virtual human interactions and enhancing user immersion. During face-to-face communication, users can feel the digital human's gaze and responses, greatly improving the realism of virtual interaction, especially in scenarios requiring high interactivity, providing an experience more in line with human communication habits. The combination of multiple technologies allows for real-time acquisition and adjustment of the user's position, exhibiting strong adaptability. Regardless of the user's position or angle, the virtual digital human can adjust its eye orientation at any time, ensuring that the digital human always maintains a direct facing relationship with the user. Furthermore, its strong real-time performance enables the system to operate stably in dynamic environments, making it suitable for various application scenarios with high real-time requirements, such as intelligent customer service and virtual assistants.

[0060] This embodiment combines sound localization, infrared ranging, and facial recognition technologies to provide a high-precision, high-real-time virtual digital human interaction solution. It not only solves the problems mentioned in the background technology, but also greatly enhances the naturalness and immersion of virtual digital human interaction, providing a new technical path for the further development of the field of intelligent human-computer interaction.

[0061] In one embodiment, a method for controlling the orientation of a virtual digital human face on a display screen is provided, characterized by comprising the following steps:

[0062] Step 1: Acquire the sound source signals received by multiple microphones in the microphone array, and measure the time difference of the signals received by each microphone using the TDOA algorithm;

[0063] Step 2: Calculate the distance difference between the sound source and each receiver in the microphone array based on the time difference;

[0064] Step 3: Based on the measured distance difference, the three-dimensional coordinates of the sound source are solved using the least squares method or the Chan algorithm by measuring the time difference between multiple receivers.

[0065] Step 4: Use an infrared laser rangefinder to measure the physical distance between the object and the display screen, and calculate the user's relative position;

[0066] Step 5: Based on the user's relative position, obtain the face image captured by the camera closest to the user and use the YOLO algorithm to perform face detection on the image and extract the feature points of the user's face;

[0067] Step 6: Based on the extracted facial feature points, use a deep learning model to generate the feature embedding vector of the user's face and calculate the coordinates of the user's eye position;

[0068] Step 7: Merge the sound source coordinates from Step 3 with the distance data from Step 4 and the facial coordinate data from Step 6, and map these data to the physical location of the display screen using a coordinate transformation algorithm;

[0069] Step 8: Based on the mapped physical location, the virtual digital human's eye orientation is adjusted using a simulated mouse control algorithm via the control module to ensure it is facing the user's eyes.

[0070] Step 9: If the user's location changes, repeat the above steps to update in real time, ensuring that the virtual digital human's eyes are always aligned with the user's eyes.

[0071] This embodiment controls the virtual digital human's eye orientation through multiple steps. First, the time difference of sound signals arriving at each microphone is measured using a microphone array, and the location of the sound source is calculated using the TDOA algorithm. Next, the system obtains the physical distance between the user and the screen using an infrared laser rangefinder, while simultaneously using a camera for face detection and extracting feature points from the user's face using the FaceNet model. The control module combines this information, maps the user's position to the physical coordinates of the display screen through coordinate transformation, and finally adjusts the virtual digital human's eye orientation using a simulated mouse control algorithm.

[0072] This embodiment enables a complete process from voice localization and facial recognition to eye orientation adjustment, ensuring that the virtual digital human can respond to changes in the user's location in real time and interact naturally and smoothly. Through precise facial feature extraction and multi-sensor data fusion, the system can significantly improve the accuracy of the virtual digital human and the quality of the interactive experience.

[0073] In one embodiment, an AI-animated digital human device is provided that enables face-to-face communication with people by utilizing sound localization and face tracking, such as... Figure 1 As shown:

[0074] The hollow circle is a microphone device with infrared ranging function;

[0075] The solid circle represents a miniature camera mounted vertically on the outside of the screen;

[0076] The semi-circular section is a flexible LED screen device, and the figures inside are virtual simulation three-dimensional digital humans;

[0077] An array of microphones is installed on the top, bottom, left, and right bezels of the screen. The microphones are equally spaced. An infrared laser rangefinder is also installed at the location of each microphone.

[0078] When someone approaches, the laser ranging array detects the nearby object. For example, if five sets of infrared laser sensors simultaneously sense the distance to the object, they will form an array of dots at different distances, a1, a2, a3, a4, and a5.

[0079] Noise reduction design: Beamforming algorithm combined with RNNoise noise reduction module.

[0080] When a person speaks close to the screen, the microphone array locates the sound source using a microphone array algorithm based on TDOA (Time Difference of Arrival). Simultaneously, it measures the distance between the questioner and various distance measurement points on the curved screen to pinpoint the physical location of the person's mouth.

[0081] Visual tracking: Based on the determined physical location, locate the nearest camera in that direction and detect faces using the camera. Utilize a face detection algorithm (based on deep learning's YOLO algorithm) to detect faces within the camera's image. Obtain the position, size, and feature points of each face. Based on facial features, provide the eye position coordinates (x, y) of the current speaker.

[0082] The detected faces are feature-extracted and stored in a database, and a deep learning model (FaceNet) is used to generate face embeddings. These embeddings can be used for subsequent face matching and tracking.

[0083] Based on the current camera installation location, coordinate transformation is performed to convert it to a physical location with the screen as the reference reference.

[0084] Based on the latest calculated physical position, a ray selection algorithm simulating mouse movement over a 3D virtual simulation model is used to control the orientation of the digital human on the screen, ensuring that the digital human's eyes are aligned with the eyes of the person asking the question in the physical world.

[0085] The person asking the question can have a face-to-face conversation with the digital human on the screen, and as the person moves, the digital human moves in sync, always maintaining a face-to-face effect.

[0086] If the questioner does not make a sound, the digital human does not respond to that person.

[0087] This embodiment first uses sound localization to capture the location of the questioner, then uses facial recognition to find the person's face, and then uses the location of the person's eyes to enable the digital human to look into the questioner's eyes and speak, truly achieving the purpose of face-to-face communication.

[0088] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. An AI-animated digital human device that enables face-to-face communication with people, characterized in that, include: Display screen used to present AI-animated digital humans; Multiple microphone units distributed on the display screen are used to independently collect sound information of users near the display screen; Multiple ranging sensors distributed on the display screen are used to measure the distance between the display screen and the user, respectively. Multiple cameras distributed on the display screen are used to collect the user's facial information; The sound source localization module is used to determine the location of the sound source based on the sound information collected by the multiple microphone units and the distance between the multiple ranging sensors and the user; The visual tracking module is used to determine the camera closest to the current sound source location among the multiple cameras based on the sound source location, control the camera to capture the user's face image, and detect facial feature information based on the user's face image; Subsequent face matching and tracking are performed based on facial feature information, and the AI ​​animated digital human is controlled to move within the display screen following the user's actual position changes, so that the AI ​​animated digital human and the user maintain a face-to-face effect; Each of the multiple microphone units corresponds to one of the ranging sensors, and the microphone unit and its corresponding ranging sensor are integrated into one unit; The multiple microphone units are respectively installed on the top bezel, bottom bezel, left bezel, and right bezel of the display screen; There are multiple microphone units located on the top bezel of the display and multiple microphone units located on the bottom bezel of the display, and the number of microphone units is equal. There are multiple microphone units located on the left bezel of the display and multiple microphone units located on the right bezel of the display, and the number of microphone units is equal. The multiple cameras are mounted on the upper bezel of the display screen; All microphone units located on the upper bezel of the display screen are divided into several groups of equal number, and the multiple cameras are distributed at intervals between adjacent groups of microphone units; The subsequent face matching and tracking based on facial feature information includes determining the coordinates of the user's eyes based on the characteristics of the facial features, and using a ray selection algorithm that simulates a mouse to select objects in a three-dimensional virtual simulation model, thereby controlling the AI ​​animated digital human to keep the user facing each other while also keeping the eye-to-eye line aligned.

2. The apparatus according to claim 1, characterized in that, The display screen is a curved cylindrical surface.

3. The apparatus according to claim 1, characterized in that, The display screen has a semi-circular cross-section.

4. A method for controlling the orientation of a virtual digital human face on a display screen, characterized in that, Includes the following steps: Step 1: Acquire the sound source signals received by multiple microphones in the microphone array, and measure the time difference of the signals received by each microphone using the TDOA algorithm; Step 2: Calculate the distance difference between the sound source and each receiver in the microphone array based on the time difference; Step 3: Based on the measured distance difference, the three-dimensional coordinates of the sound source are solved using the least squares method or the Chan algorithm by measuring the time difference between multiple receivers. Step 4: Use an infrared laser rangefinder to measure the physical distance between the object and the display screen, and calculate the user's relative position; Step 5: Based on the user's relative position, obtain the face image captured by the camera closest to the user and use the YOLO algorithm to perform face detection on the image and extract the feature points of the user's face; Step 6: Based on the extracted facial feature points, use a deep learning model to generate the feature embedding vector of the user's face and calculate the coordinates of the user's eye position; Step 7: Merge the sound source coordinates from Step 3 with the distance data from Step 4 and the facial coordinate data from Step 6, and map these data to the physical location of the display screen using a coordinate transformation algorithm; Step 8: Based on the mapped physical location, the virtual digital human's eye orientation is adjusted using a simulated mouse control algorithm via the control module to ensure it is facing the user's eyes. Step 9: If the user's location changes, repeat the above steps to update in real time, ensuring that the virtual digital human's eyes are always aligned with the user's eyes.

Images