A multimodal large model system for driving real-time interaction of two-dimensional digital humans

Abstract

The present invention discloses a multimodal large model system for driving real-time interaction of a two-dimensional digital human, and relates to the technical field of digital human large models, comprising a data acquisition module, a multimodal processing module, and a real-time interaction module. The data acquisition module is used to use computer vision technology to perform area division and feature extraction on a user's face, and to detect the amplitude and frequency of head shaking, as well as to detect the user's pupil gaze point during video communication between users. The multimodal processing module is used to analyze the collected facial feature data and pupil gaze point data, evaluate the attractiveness of each facial area, and determine whether each area adopts a real-time capture or expression template method. The real-time interaction module is used to control the digital human expression corresponding to the user according to the expression presentation method, and to interact between each user. The present invention has the characteristics of combining the advantages of both.

Description

Technical Field

[0001] The present invention relates to the technical field of digital human large models, and in particular to a multimodal large model system for driving real-time interaction of a two-dimensional digital human. Background Art

[0002] In modern office and remote collaboration environments, multi-person meetings that use two-dimensional digital human technology for facial scanning have become popular. Participants scan their faces through specific devices or applications, and the system generates a two-dimensional digital image of each participant, which can provide users with a more immersive and interactive experience while protecting user privacy.

[0003] The existing two-dimensional real-time interactive large model has two ways of presenting facial expressions. One is to capture the details of each facial feature in real time and display the face on the digital human in real time. The other is to detect the general facial expression of the person at that time, then search for the expression template of the digital human and select the appropriate expression to present.

[0004] However, for people who frequently shake their heads, the first method makes it difficult for the facial recognition system to stably lock and track these features, increasing the complexity of image processing. Because the amount of data transmitted is large, it is easy to cause expression display delays. The second method cannot accurately capture the user's unique expressions or subtle emotional changes, resulting in the digital human's emotions being not rich and natural enough. Therefore, it is necessary to design a multimodal large model system that combines the advantages of both to drive the real-time interaction of two-dimensional digital humans. Summary of the invention

[0005] The purpose of the present invention is to provide a multimodal large model system for driving real-time interaction of two-dimensional digital humans, so as to solve the problems raised in the above-mentioned background technology.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: a multimodal large model system for driving real-time interaction of two-dimensional digital humans, comprising a data acquisition module, a multimodal processing module, and a real-time interaction module. The data acquisition module is used to use computer vision technology to divide the user's face into regions and extract features, and detect the amplitude and frequency of head shaking, as well as detect the user's pupil gaze point during video communication between users. The multimodal processing module is used to analyze the collected facial feature data and pupil gaze point data, evaluate the attractiveness of each facial area, and decide whether to use real-time capture or expression templates for each area. The real-time interaction module is used to control the digital human expression corresponding to the user according to the way of presenting the expression, so as to interact between users.

[0007] According to the above technical solution, the data acquisition module includes a camera unit, a facial region recognition module, an expression feature extraction module, a database, a head shaking detection module, a timing module, and a pupil gaze point capture module. The camera unit is electrically connected to the facial region recognition module, the facial region recognition module is electrically connected to the expression feature extraction module, the expression feature extraction module is electrically connected to the database, the head shaking detection module is electrically connected to the timing module and the facial region recognition module, the camera unit is used to shoot the user's face, the facial region recognition module is used to recognize the face and divide the face into multiple regions, the expression feature extraction module is used to extract the expression features of each facial region, the database is used to store the expression feature data of each user, the head shaking detection module is used to detect the shaking frequency and amplitude of the user's head, the timing module is used to count the time, and the pupil gaze point capture module is used to detect the user's pupil gaze point during video communication between users;

[0008] The multimodal processing module includes an attractiveness evaluation module, a pre-setting module, and an automatic setting module. The attractiveness evaluation module is electrically connected to the pupil gaze point capture module, the pre-setting module is electrically connected to the facial region recognition module, and the automatic setting module is electrically connected to the attractiveness evaluation module, the head shaking detection module, and the facial region recognition module. The attractiveness evaluation module is used to evaluate the attractiveness of each facial region of the current user based on the common pupil gaze point data of other users. The pre-setting module is used to set which facial regions are to be captured in real time and which facial regions are to be captured in expression templates based on the attractiveness data. The automatic setting module is used to determine the proportion of facial regions to be captured in real time in combination with the user's head shaking frequency and the attractiveness of each facial region.

[0009] The real-time interactive module includes an expression qualitative module, an expression template library, a templated display module, and a real-time capture display module. The expression qualitative module is electrically connected to the expression feature extraction module. The expression template library is used to store the expression templates of the digital human. The templated display module is used to use the expression templates to display the corresponding facial area on the facial features of the digital human. The real-time capture display module is used to display the corresponding facial area on the facial features of the digital human using the real-time capture method.

[0010] According to the above technical solution, the working method of the large model is:

[0011] S0. The user scans the face through the camera, divides the face into different areas, and collects feature data of these areas;

[0012] S1. When the current user conducts a communication test with other users by video communication, after obtaining the consent of other users, the pupil gaze points of other users are detected to obtain the facial areas that are generally more attractive to other users;

[0013] S2, using real-time capture to obtain dynamic expressions for the facial area that the current user is more attracted to, and using expression templates to match and pre-set the facial area that is less attractive;

[0014] S3, when the current user interacts with other users in the form of a digital human, the head shaking frequency of the current user is captured in real time, and the proportion of the facial area captured in real time is adjusted in combination with a pre-set matching method;

[0015] S4, qualitatively characterize the facial expression of the current user's facial area that needs template matching, and find the expression that best matches the current facial area in the expression template library, and display it on the face of the digital human. The expressions of other facial areas are displayed on the face of the digital human through real-time capture and facial fitting methods. The system integrates the real-time captured facial area and the template matched facial area to realize the expression output of the digital human;

[0016] S5. Count the time difference between the digital human expressions viewed by other users and the real-time expressions detected by the current user. If the time difference exceeds a threshold value t0, optimize the proportion of the facial area captured in real time until the time difference is lower than the threshold value.

[0017] According to the above technical solution, in S0, the specific method of dividing the face into different areas is: the facial features, forehead, cheeks, and chin of the current user are identified through the facial area recognition module, and the facial area covered by these parts is divided. When the expression of the current user changes, the identified area outline also changes with the change of the facial area covered by these parts.

[0018] According to the above technical solution, in S1, the method for detecting the pupil gaze point is:

[0019] S1-1. Use the pupil gaze point capture module to identify the pupils of other users, calculate the three-dimensional vector of the line of sight based on the pupil center point combined with the head angle, and calculate the actual focus position through the intersection of the line of sight vector and the screen plane;

[0020] S1-2, using the timing module to count the time that the focus position of other users' sights stays in the facial area of ​​the current user within a specified time period, which are {T1, T2, ..., T n}, where n is the number of facial regions;

[0021] S1-3. Test multiple other users and average the average of the stay time to get the attractiveness value of each facial area of ​​the current user. Here, i is the serial number of the facial region, and the larger the attractiveness value is, the more attractive the facial region is.

[0022] According to the above technical solution, the method for presetting in S2 is: first determine the initial proportion of the facial area captured in real time, denoted as a%, and calculate the attractiveness value F of each facial area. i Sort from large to small, and sum up in order starting from the largest item until the sum exceeds a%, then remove the last item, and the facial areas corresponding to the other summed items are captured in real time, and the facial areas corresponding to the items not included in the sum are matched by expression templates.

[0023] According to the above technical solution, in S3, the specific method for adjusting the facial area ratio captured in real time is:

[0024] S3-1, mark the feature points in the current user's facial area, and calculate the distance between each feature point and the screen, which are {h1, h2, ..., h n}, when the user's head shakes, the distance between each feature point and the screen also changes in real time, and the distance change values ​​{Δh1, Δh2, ..., Δh n}, and sum them up to get the total distance change value ∑Δh n ,∑Δh n The larger it is, the more frequently the current user's head shakes;

[0025] S3-2. The more frequently the head shakes, the smaller the proportion of the facial area captured in real time is, that is, the correction ratio of the facial area captured in real time is A=(a-μ∑Δh n )%, where μ is the influence coefficient of head shaking on the facial area captured in real time, which is determined according to actual conditions, and A% replaces a% in S2 to adjust the real-time capture ratio.

[0026] According to the above technical solution, in S5, the specific method for optimizing the proportion of the facial area captured in real time is: counting the time difference {t1, t2, ..., t k}, where k is the number of other users who interact with the current user in real time using digital humans, and the average time difference is obtained by averaging when When, according to The correction ratio A% of the face area captured in real time is reduced in direct proportion to the size of No changes are made when .

[0027] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: the present invention first divides the scanned user's face into regions, uses a method of capturing expression details in real time for the facial region that needs to be more delicately expressed, and uses an expression template for the facial region that does not need to be delicately expressed, so that the digital human can capture the user's unique expression with the least amount of data transmission;

[0028] By collecting the amplitude and frequency of the user's head shaking in daily interactions, the proportion of the area that needs to capture facial expression details in real time is adjusted to ensure attractiveness as much as possible while reducing facial expression delays. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0030] In the attached picture:

[0031] Figure 1 It is a schematic diagram of the overall module structure of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] See also Figure 1 The present invention provides a technical solution: a multimodal large model system for driving real-time interaction of two-dimensional digital human, including a data acquisition module, a multimodal processing module, and a real-time interaction module. The data acquisition module is used to use computer vision technology to divide the user's face into regions and extract features, and detect the amplitude and frequency of head shaking, as well as the user's pupil gaze point during video communication between users. The multimodal processing module is used to analyze the collected facial feature data and pupil gaze point data, evaluate the attractiveness of each facial region, and decide whether to use real-time capture or expression template for each region. The real-time interaction module is used to control the expression of the digital human corresponding to the user according to the expression presentation method, and interact between users.

[0034] The data acquisition module includes a camera unit, a facial region recognition module, an expression feature extraction module, a database, a head shaking detection module, a timing module, and a pupil gaze point capture module. The camera unit is electrically connected to the facial region recognition module, the facial region recognition module is electrically connected to the expression feature extraction module, the expression feature extraction module is electrically connected to the database, the head shaking detection module is electrically connected to the timing module and the facial region recognition module, the camera unit is used to shoot the user's face, the facial region recognition module is used to recognize the face and divide the face into multiple regions, the expression feature extraction module is used to extract expression features of each facial region, the database is used to store the expression feature data of each user, the head shaking detection module is used to detect the shaking frequency and amplitude of the user's head, the timing module is used to count the time, and the pupil gaze point capture module is used to detect the user's pupil gaze point when the users communicate via video;

[0035] The multimodal processing module includes an attractiveness evaluation module, a pre-setting module, and an automatic setting module. The attractiveness evaluation module is electrically connected to the pupil gaze point capture module, the pre-setting module is electrically connected to the facial region recognition module, and the automatic setting module is electrically connected to the attractiveness evaluation module, the head shaking detection module, and the facial region recognition module. The attractiveness evaluation module is used to evaluate the attractiveness of each facial region of the current user based on the common pupil gaze point data of other users. The pre-setting module is used to set which facial regions are to be captured in real time and which facial regions are to be captured using expression templates based on the attractiveness data. The automatic setting module is used to determine the proportion of facial regions to be captured in real time based on the user's head shaking frequency and the attractiveness of each facial region.

[0036] The real-time interactive module includes an expression qualitative module, an expression template library, a template display module, and a real-time capture display module. The expression qualitative module is electrically connected to the expression feature extraction module. The expression template library is used to store expression templates of digital humans. The template display module is used to use expression templates on the facial features of digital humans to display corresponding facial areas. The real-time capture display module is used to use real-time capture on the facial features of digital humans to display corresponding facial areas.

[0037] The working method of this large model is:

[0038] S0. The user scans the face through the camera, divides the face into different areas, and collects feature data of these areas;

[0039] S1. When the current user conducts a communication test with other users by video communication, after obtaining the consent of other users, the pupil gaze points of other users are detected to obtain the facial areas that are generally more attractive to other users;

[0040] S2, using real-time capture to obtain dynamic expressions for the facial area that the current user is more attracted to, and using expression templates to match and pre-set the facial area that is less attractive;

[0041] S3, when the current user interacts with other users in the form of a digital human, the head shaking frequency of the current user is captured in real time, and the proportion of the facial area captured in real time is adjusted in combination with a pre-set matching method;

[0042] S4, qualitatively characterize the facial expression of the current user's facial area that needs template matching, and find the expression that best matches the current facial area in the expression template library, and display it on the face of the digital human. The expressions of other facial areas are displayed on the face of the digital human through real-time capture and facial fitting methods. The system integrates the real-time captured facial area and the template matched facial area to realize the expression output of the digital human;

[0043] S5, counting the time difference between the digital human expressions viewed by other users and the real-time expressions detected by the current user, if the time difference exceeds the threshold t0, optimizing the proportion of the facial area captured in real time until the time difference is lower than the threshold;

[0044] In S0, the specific method of dividing the face into different regions is: the facial features, forehead, cheeks, and chin of the current user are identified by the facial region recognition module, and the facial region covered by these parts is divided, and when the expression of the current user changes, the identified region contour also changes with the change of the facial region covered by these parts;

[0045] In order to maintain the symmetry of the facial features, the two eyes and two ears are included in the same facial area.

[0046] In S1, the method for detecting pupil gaze point is:

[0047] S1-1. Use the pupil gaze point capture module to identify the pupils of other users, calculate the three-dimensional vector of the line of sight based on the pupil center point combined with the head angle, and calculate the actual focus position through the intersection of the line of sight vector and the screen plane;

[0048] S1-2, using the timing module to count the time that the focus position of other users' sights stays in the facial area of ​​the current user within a specified time period, which are {T1, T2, ..., T n}, where n is the number of facial regions;

[0049] S1-3. Test multiple other users and average the average of the stay time to get the attractiveness value of each facial area of ​​the current user. Where i is the serial number of the facial region, and the larger the attractiveness value is, the more attractive this facial region is;

[0050] In S2, the presetting method is as follows: firstly, the initial ratio of the facial area captured in real time is determined, which is denoted as a%, and the attractiveness value F of each facial area is i Sort from largest to smallest, and sum up in order starting from the largest item until the sum exceeds a%, remove the last item, and use real-time capture for the facial areas corresponding to the other summed items, and use expression template matching for the facial areas not included in the summation;

[0051] In S3, the specific method for adjusting the proportion of the facial area captured in real time is:

[0052] S3-1, mark the feature points in the current user's facial area, and calculate the distance between each feature point and the screen, which are {h1, h2, ..., h n}, when the user's head shakes, the distance between each feature point and the screen also changes in real time, and the distance change values ​​{Δh1, Δh2, ..., Δh n}, and sum them up to get the total distance change value ΣΔh n ,ΣΔh n The larger it is, the more frequently the current user's head shakes;

[0053] S3-2. The more frequently the head shakes, the smaller the proportion of the facial area captured in real time is, that is, the correction ratio of the facial area captured in real time is A=(a-μ∑Δh n )%, where μ is the influence coefficient of head shaking on the facial area captured in real time, which is determined according to the actual situation, and A% replaces a% in S2 to adjust the real-time capture ratio;

[0054] In S5, the specific method for optimizing the proportion of the facial area captured in real time is: counting the time difference {t1, t2, ..., t k}, where k is the number of other users who interact with the current user in real time using digital humans, and the average time difference is obtained by averaging when When, according to The correction ratio A% of the face area captured in real time is reduced in direct proportion to the size of No changes are made when .

[0055] This adjustment method takes into account the real-time shaking degree of the user's head in front of the screen. If the user's head shakes greatly at a certain moment, the use of template matching will greatly reduce the computational burden of the facial recognition algorithm and make the display clearer, because the effect of real-time capture and display will be worse when the shaking degree is large.

[0056] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0057] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-modal large model system for driving real-time interaction of two-dimensional digital humans, characterized by: It includes a data acquisition module, a multimodal processing module, and a real-time interaction module. The data acquisition module is used to use computer vision technology to divide the user's face into regions and extract features, and detect the amplitude and frequency of head shaking, as well as the user's pupil gaze point during video communication between users. The multimodal processing module is used to analyze the collected facial feature data and pupil gaze point data, evaluate the attractiveness of each facial region, and decide whether to use real-time capture or expression template for each region. The real-time interaction module is used to control the digital human expression corresponding to the user according to the expression presentation method, and interact between users. The data acquisition module includes a camera unit, a facial region recognition module, an expression feature extraction module, a database, a head shaking detection module, a timing module, and a pupil gaze point capture module. The camera unit is electrically connected to the facial region recognition module, the facial region recognition module is electrically connected to the expression feature extraction module, the expression feature extraction module is electrically connected to the database, the head shaking detection module is electrically connected to the timing module and the facial region recognition module, the camera unit is used to shoot the user's face, the facial region recognition module is used to recognize the face and divide the face into multiple regions, the expression feature extraction module is used to extract expression features of each facial region, the database is used to store the expression feature data of each user, the head shaking detection module is used to detect the shaking frequency and amplitude of the user's head, the timing module is used to count the time, and the pupil gaze point capture module is used to detect the user's pupil gaze point when the users communicate via video; The multimodal processing module includes an attractiveness evaluation module, a pre-setting module, and an automatic setting module. The attractiveness evaluation module is electrically connected to the pupil gaze point capture module, the pre-setting module is electrically connected to the facial region recognition module, and the automatic setting module is electrically connected to the attractiveness evaluation module, the head shaking detection module, and the facial region recognition module. The attractiveness evaluation module is used to evaluate the attractiveness of each facial region of the current user based on the common pupil gaze point data of other users. The pre-setting module is used to set which facial regions are to be captured in real time and which facial regions are to be captured in expression templates based on the attractiveness data. The automatic setting module is used to determine the proportion of facial regions to be captured in real time in combination with the user's head shaking frequency and the attractiveness of each facial region. The real-time interactive module includes an expression qualitative module, an expression template library, a templated display module, and a real-time capture display module. The expression qualitative module is electrically connected to the expression feature extraction module. The expression template library is used to store the expression templates of the digital human. The templated display module is used to use the expression templates to display the corresponding facial area on the facial features of the digital human. The real-time capture display module is used to display the corresponding facial area on the facial features of the digital human using the real-time capture method.

2. A multi-modal large model system for driving real-time interaction of a two-dimensional digital human according to claim 1, characterized in that: The working method of this large model is: S0. The user scans the face through the camera, divides the face into different areas, and collects feature data of these areas; S1. When the current user conducts a communication test with other users by video communication, after obtaining the consent of other users, the pupil gaze points of other users are detected to obtain the facial areas that are generally more attractive to other users; S2, using real-time capture to obtain dynamic expressions for the facial area that the current user is more attracted to, and using expression templates to match and pre-set the facial area that is less attractive; S3, when the current user interacts with other users in the form of a digital human, the head shaking frequency of the current user is captured in real time, and the proportion of the facial area captured in real time is adjusted in combination with a pre-set matching method; S4, qualitatively characterize the facial expression of the current user's facial area that needs template matching, and find the expression that best matches the current facial area in the expression template library, and display it on the face of the digital human. The expressions of other facial areas are displayed on the face of the digital human through real-time capture and facial fitting methods. The system integrates the real-time captured facial area and the template matched facial area to realize the expression output of the digital human; S5. Count the time difference between the digital human expressions viewed by other users and the real-time expressions detected by the current user. If the time difference exceeds a threshold value t0, optimize the proportion of the facial area captured in real time until the time difference is lower than the threshold value.

3. A multi-modal large model system for driving real-time interaction of a two-dimensional digital human according to claim 2, characterized in that: In the S0, the specific method of dividing the face into different areas is: the facial features, forehead, cheeks, and chin of the current user are identified through the facial area recognition module, and the facial area covered by these parts is divided. When the expression of the current user changes, the identified area outline also changes with the change of the facial area covered by these parts.

4. A multi-modal large model system for driving real-time interaction of a two-dimensional digital human according to claim 3, characterized in that: In S1, the method for detecting the pupil gaze point is: S1-1. Use the pupil gaze point capture module to identify the pupils of other users, calculate the three-dimensional vector of the line of sight based on the pupil center point combined with the head angle, and calculate the actual focus position through the intersection of the line of sight vector and the screen plane; S1-2, using the timing module to count the time that the focus position of other users' sights stays in the facial area of ​​the current user within a specified time period, which are {T1, T2, ..., T n }, where n is the number of facial regions; S1-3. Test multiple other users and average the average of the stay time to get the attractiveness value of each facial area of ​​the current user. Here, i is the serial number of the facial region, and the larger the attractiveness value is, the more attractive the facial region is.

5. A multi-modal large model system for driving real-time interaction of a two-dimensional digital human according to claim 4, characterized in that: The method for presetting in S2 is: first determine the initial ratio of the facial area captured in real time, denoted as a%, and calculate the attractiveness value F of each facial area. i Sort from large to small, and sum up in order starting from the largest item until the sum exceeds a%, then remove the last item, and the facial areas corresponding to the other summed items are captured in real time, and the facial areas corresponding to the items not included in the sum are matched by expression templates.

6. A multi-modal large model system for driving real-time interaction of a two-dimensional digital human according to claim 5, characterized in that: In S3, the specific method for adjusting the facial area ratio captured in real time is: S3-1, mark the feature points in the current user's facial area, and calculate the distance between each feature point and the screen, which are {h1, h2, ..., h n }, when the user's head shakes, the distance between each feature point and the screen also changes in real time, and the distance change values ​​{Δh1, Δh2, ..., Δh n }, and sum them up to get the total distance change value ΣΔh n ,ΣΔh n The larger it is, the more frequently the current user's head shakes; S3-2. The more frequently the head shakes, the smaller the proportion of the facial area captured in real time is, that is, the correction ratio of the facial area captured in real time is A=(a-μ∑Δh n )%, where μ is the influence coefficient of head shaking on the facial area captured in real time, which is determined according to actual conditions, and A% replaces a% in S2 to adjust the real-time capture ratio.

7. A multi-modal large model system for driving real-time interaction of a two-dimensional digital human according to claim 6, characterized in that: In S5, the specific method for optimizing the proportion of the facial area captured in real time is: counting the time difference {t1, t2, ..., t k }, where k is the number of other users who interact with the current user in real time using digital humans, and the average time difference is obtained by averaging when When, according to The correction ratio A% of the face area captured in real time is reduced in direct proportion to the size of No changes are made when .

Images