Facial information display method, device, equipment and storage medium

Abstract

The present application discloses a facial information display method, apparatus, device, and storage medium, comprising acquiring each frame of a user's video stream, determining a specific facial region of each frame based on each frame of the user's video stream, determining multiple expression parameters of the video stream based on the specific facial region of each frame, obtaining the user's facial information based on the multiple expression parameters, combining the user's facial information with a pre-set cartoon character facial model to obtain target facial information, and finally displaying the target facial information. By acquiring each frame of the user's video stream, then gradually determining the final facial information based on each frame of the video stream, and then combining the facial information with the cartoon character facial model, this method can be applied to common communication devices, improving applicability and facilitating user use.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a facial information display method, apparatus, device and storage medium. Background Art

[0002] With the development of the internet and the widespread adoption of smartphones, the way people communicate has changed significantly. Video calls have become a favorite and most common method of communication. As the use of virtual digital scenes becomes more widespread, a new method is emerging that captures user expressions and combines them with virtual characters to display the faces of both parties, enhancing the fun of video chats.

[0003] The traditional method of obtaining virtual digital facial information relies on large-scale motion capture equipment, which requires multiple cameras or higher-quality cameras to extract features from the user's face. However, this method is not suitable for ordinary communication equipment and is inconvenient for users to use. Summary of the Invention

[0004] In view of this, the present application provides a facial information display method, apparatus, device and storage medium to address the defect that the traditional method of obtaining virtual digital facial information is not suitable for ordinary communication equipment and is inconvenient for users.

[0005] To achieve the above objectives, the following solutions are proposed:

[0006] In a first aspect, a facial information display method includes:

[0007] Get each frame of the user's video stream;

[0008] Determining a specific face region of each frame of image according to each frame of image in the video stream;

[0009] Determining a plurality of expression base parameters of the video stream according to a specific face area of ​​each frame image;

[0010] Obtaining facial information of the user according to the multiple expression base parameters, and combining the facial information of the user with a preset facial model of a cartoon character to obtain target facial information;

[0011] The target facial information is displayed.

[0012] Preferably, determining the specific face area of ​​each frame image according to each frame image of the video stream includes:

[0013] Using a preset face region model, processing each frame of the video stream to obtain a specific face region of each frame;

[0014] The face region model is obtained by using each frame image in the acquired face data set sample as a training sample and using the real specific face region of each frame image in the face data set sample as a sample label for training.

[0015] Preferably, the using a preset face region model to process each frame of the video stream to obtain a specific face region of each frame includes:

[0016] Utilizing the coefficient acquisition module of the face region model, coefficients are extracted for each frame of image to obtain coordinate parameters of each frame of image in a rectangular coordinate system and the confidence level of each frame of image;

[0017] The specific face region obtaining module of the face region model is used to integrate the coordinate parameters of each frame image in a rectangular coordinate system and the confidence level of each frame image to obtain the specific face region of each frame image.

[0018] Preferably, the process of acquiring samples of the face dataset includes:

[0019] Generate fake face datasets using a pre-set adversarial generative network model;

[0020] Performing data enhancement processing on the fake face dataset to obtain samples of the face dataset.

[0021] Preferably, the coordinate parameters of each frame of image in the rectangular coordinate system include:

[0022] The horizontal coordinate of the upper left corner of the image, the vertical coordinate of the upper left corner of the image, the horizontal coordinate of the lower right corner of the image, and the vertical coordinate of the lower right corner of the image.

[0023] Preferably, determining the multiple expression base parameters of the video stream according to the specific face area of ​​each frame image includes:

[0024] Using a preset expression basic parameter model, processing the specific face area of ​​each frame image to obtain a plurality of expression basic parameters of the video stream;

[0025] The expression basic parameter model is obtained by using a specific face area of ​​each frame image in the acquired face data set sample as a training sample and using the real expression basic parameters of the face data set sample as sample labels for training.

[0026] Preferably, the process of setting the facial model of the cartoon character includes:

[0027] Obtaining the facial features, skin color, hair texture, and hairstyle of the user based on the specific facial region of each frame of image;

[0028] The facial model of the cartoon character is obtained according to the facial features, skin color, hair texture and hairstyle of the user.

[0029] In a second aspect, a facial information display device includes:

[0030] The video stream acquisition module is used to obtain each frame of the user's video stream;

[0031] A face detection module, configured to determine a specific face region of each frame of the video stream;

[0032] An expression coefficient regression module, configured to determine a plurality of expression base parameters of the video stream according to a specific facial region of each frame image;

[0033] A driving module, which obtains the user's facial information according to the multiple expression base parameters, and combines the user's facial information with a preset cartoon character's facial model to obtain target facial information;

[0034] A display module is used to display the target facial information.

[0035] In a third aspect, a facial information display device includes a memory and a processor;

[0036] The memory is used to store programs;

[0037] The processor is used to execute the program to implement the various steps of the facial information display method as described in the first aspect.

[0038] In a fourth aspect, a storage medium stores a computer program thereon, wherein when the computer program is executed by a processor, the steps of the facial information display method as described in the first aspect are implemented.

[0039] As can be seen from the above technical solution, the present application provides a facial information display method, apparatus, device, and storage medium, including obtaining each frame of the user's video stream, determining a specific facial area of ​​each frame based on each frame of the user's video stream, determining multiple expression parameters of the video stream based on the specific facial area of ​​each frame, obtaining the user's facial information based on the multiple expression parameters, and then combining the user's facial information with a pre-set cartoon character facial model to obtain target facial information, and finally displaying the target facial information. This solution obtains each frame of the user's video stream, then determines the final facial information step by step based on each frame of the video stream, and then combines the facial information with the cartoon character facial model, so that the method can be applied to ordinary communication equipment, improves applicability, and facilitates user use. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 An optional flow chart of a facial information display method provided in an embodiment of the present application;

[0041] Figure 2 and Figure 3 An expression-based schematic diagram provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of a facial information display device provided in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of the structure of a facial information display device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0045] With the popularity of smart phones and the rapid development of Internet technology, the application of virtual digital scenes has become more and more extensive. Currently, there is a calling method that uses virtual cartoon character models with the same expressions and postures as the user to replace the real faces of the two communicating parties. That is, a video method that can obtain the user's expressions or postures and combine them with virtual character images to display the facial information of the two parties on the call, thereby enhancing the fun of video chat.

[0046] However, the traditional method of obtaining virtual digital facial information relies on large-scale motion capture equipment, which requires multiple cameras or higher-quality cameras to extract features from the user's face. However, this method is not suitable for ordinary communication equipment and is inconvenient for users to use.

[0047] Based on the above defects, the embodiment of the present application provides a facial information display solution. Figure 1 The facial information display method of this application is described as follows: Figure 1 As shown, the method includes:

[0048] S1: Get each frame of the user's video stream.

[0049] The present application can be applied to terminals including display interfaces, such as mobile phones and computers. When a user uses such a terminal device, the user's video stream is obtained using the terminal device's built-in camera or acquisition module, wherein the video stream can be regarded as a collection of countless frame images.

[0050] S2: Determine a specific face region of each frame image according to each frame image of the video stream.

[0051] Specifically, each frame of the video stream is a plurality of pictures constituting the video stream, and a specific face region in each picture can be obtained from each picture.

[0052] S3: Determine multiple expression base parameters of the video stream according to the specific face area of ​​each frame image.

[0053] For each frame of image, multiple expression base parameters of the image can be obtained based on the specific facial region of the image, which can be 52 expression base parameters. Therefore, multiple groups of expression base parameters of the video stream can be obtained.

[0054] S4: Obtaining the user's facial information according to the multiple expression base parameters, and combining the user's facial information with a pre-set cartoon character's facial model to obtain target facial information.

[0055] The user can set a facial model of a cartoon character according to his or her preferences, and the user's facial information is combined with the facial model of the cartoon character to obtain the target facial information.

[0056] S5: Displaying the target facial information.

[0057] The target facial information can be displayed on a display screen connected to the terminal device and presented to the user.

[0058] As can be seen from the above technical solution, the present application provides a facial information display method, apparatus, device, and storage medium, including obtaining each frame of the user's video stream, determining a specific facial area of ​​each frame based on each frame of the user's video stream, determining multiple expression parameters of the video stream based on the specific facial area of ​​each frame, obtaining the user's facial information based on the multiple expression parameters, and then combining the user's facial information with a pre-set cartoon character facial model to obtain target facial information, and finally displaying the target facial information. This solution obtains each frame of the user's video stream, then determines the final facial information step by step based on each frame of the video stream, and then combines the facial information with the cartoon character facial model, so that the method can be applied to ordinary communication equipment, improves applicability, and facilitates user use.

[0059] Specifically, in step S2, determining a specific face region of each frame of the video stream includes:

[0060] Using a preset face region model, processing each frame of the video stream to obtain a specific face region of each frame;

[0061] The face region model is obtained by using each frame image in the acquired face data set sample as a training sample and using the real specific face region of each frame image in the face data set sample as a sample label for training.

[0062] Optionally, the process of processing each frame of the video stream using a preset face region model to obtain a specific face region in each frame includes:

[0063] The coefficient acquisition module of the facial region model is used to extract coefficients from each image frame to obtain coordinate parameters of each image frame in a rectangular coordinate system and a confidence level for each image frame. The specific facial region acquisition module of the facial region model is then used to integrate the coordinate parameters of each image frame in a rectangular coordinate system and the confidence level for each image frame to obtain a specific facial region for each image frame. The coordinate parameters of each image frame in a rectangular coordinate system may include: the horizontal coordinate of the upper left corner of the image; the vertical coordinate of the upper left corner of the image; the horizontal coordinate of the lower right corner of the image; and the vertical coordinate of the lower right corner of the image. Furthermore, a confidence threshold may be set to determine whether the specific facial region is accurate facial data. Specifically, the confidence threshold may be set to 0.99. When the confidence level of the image is greater than 0.99, the image is deemed to contain the user's complete facial information. When the confidence level of the image is less than 0.99, the image is deemed not to contain the user's complete facial information. The lower the confidence level, the smaller the range of faces contained in the image. It can be considered that the specific face area of ​​the image is not the exact face data. Therefore, when determining the multiple expression base parameters of the video stream based on the specific face area of ​​each frame image, the frame image that is not the exact face data can be deleted, that is, the multiple expression base parameters of the obtained video stream all belong to the user's own expression base parameters.

[0064] Preferably, the face dataset samples are obtained in the following two ways:

[0065] 1) Generate a fake face dataset using a pre-set adversarial generative network model; then perform data augmentation on the fake face dataset to obtain a face dataset sample. 2) Use an existing open source face detection dataset as the face dataset sample for this application.

[0066] In the embodiment provided in the present application, a lightweight deep learning network model (Mobilenetv3) can be used to construct an initial model corresponding to the face area model, wherein the number of channels (channel), module layers (block), and linear layers (linear) of the lightweight deep learning network model are reduced, and the number of model output parameters is controlled.

[0067] Preferably, the face region model can be trained 100+ times using the obtained face dataset samples to obtain a better face region model. The face dataset samples can contain more than 100,000 image samples, and each frame of image sample is trained 100+ times. After the face region model training is completed, the model can be converted using an open source cross-platform deep learning inference framework (TNN). The resulting face region model can be less than 1m in size and have an inference speed of less than 10ms.

[0068] Specifically, in step S3, the process of determining multiple expression base parameters of the video stream according to the specific face area of ​​each frame image may include:

[0069] Using a preset expression basic parameter model, processing the specific face area of ​​each frame image to obtain a plurality of expression basic parameters of the video stream;

[0070] The expression basic parameter model is obtained by using a specific face area of ​​each frame image in the acquired face data set sample as a training sample and using the real expression basic parameters of the face data set sample as sample labels for training.

[0071] Among them, the method of obtaining specific facial area samples includes: using Live Link Face software to record videos of a large number of real people, collecting 52 basic expression parameters for each frame of the recorded video, and cropping each frame of the recorded video to crop out the accurate facial area, and then forming these facial areas and the corresponding 52 basic expression parameters of each frame into specific facial area samples.

[0072] During the training process of the expression-based parameter model, 20w+ specific face area samples can be used to train the expression-based parameter model, and finally the regression loss of the model is reduced to about 0.0187 to obtain the optimal expression-based parameter model.

[0073] Optionally, after training the expression-based parameter model, it can be converted using the open-source, cross-platform Deep Learning Inference Framework (TNN). The resulting expression-based parameter model can be approximately 2.3 megabytes in size, with an inference speed of less than 15 milliseconds. This solution significantly improves model computation speed and reduces device performance overhead.

[0074] In one embodiment of the present application, the expression base parameter can be understood as the degree of exaggeration of the user's facial expression. An expression base represents a type of expression. For example, the expression base jawopen represents opening the mouth, and the expression base parameter is the degree of opening the user's mouth. When the expression base parameter is 0, it can be understood that the user's expression state is not opening the mouth (closing the mouth). Figure 2 As shown; when the expression base parameter is 1, it can be understood that the expression state is that the user opens his mouth to the maximum, such as Figure 3 shown.

[0075] Specifically, in step S4, the process of setting the facial model of the cartoon character may include:

[0076] Based on the specific facial area of ​​each frame image, the user's facial features, skin color, hair material and hairstyle are obtained, and then the facial model of the cartoon character is obtained according to the user's facial features, skin color, hair material and hairstyle.

[0077] The facial information display device provided in an embodiment of the present application is described below. The facial information display device described below and the facial information display method described above can be referenced to each other.

[0078] Combine Figure 4 , the facial information display device is introduced, such as Figure 4 As shown, the device may include:

[0079] The video stream acquisition module 10 is used to acquire each frame of the user's video stream;

[0080] A face detection module 20 is used to determine a specific face area of ​​each frame image according to each frame image of the video stream;

[0081] An expression coefficient regression module 30 is used to determine a plurality of expression base parameters of a video stream according to a specific face region of each frame image;

[0082] The driving module 40 obtains the user's facial information according to the multiple expression base parameters, and combines the user's facial information with a preset cartoon character's facial model to obtain target facial information;

[0083] The display module 50 is used to display the target facial information.

[0084] Optionally, in the driving module 40, a driving instruction can be set first, and the driving instruction can be stored in the terminal device in the form of shared memory. The module reads the latest driving instruction from the terminal device in real time, and applies the driving instruction to the pre-set facial model of the cartoon character, thereby achieving the purpose of real-time driving.

[0085] It should be noted that the present application can be applied to terminals that include a display interface, such as mobile phones and computers. With user authorization, the facial information display method provided in this application can be used to obtain the facial information or human biometrics of the authorized user. For example, at a judicial agency, the facial information display method of this application is used to obtain the facial information or human biometrics of the authorized user for user registration; for another example, with user authorization, the facial information display device uses the facial information display method of this application to obtain the facial information or human biometrics of the authorized user for authorized user identity identification and attendance registration.

[0086] The facial information display device provided by this application can be applied to the field of public security maintenance. For example, it can be applied to public safety places such as enterprises and communities to provide certain protection for the safety of enterprise personnel and community residents. When using this application, an indicator sign can be set up in a specific application scenario to remind the user that when using the facial information display device of this application, facial information or human biometrics will be collected and obtained. Then, with the user's authorization, facial information or human biometrics can be legally collected and obtained to implement the facial information display device of this application.

[0087] Furthermore, the embodiment of the present application provides a facial information display device. Optionally, Figure 5 The hardware structure diagram of the facial information display device is shown. Figure 5 The hardware structure of the facial information display device may include: at least one processor 01, at least one communication interface 02, at least one memory 03 and at least one communication bus 04.

[0088] In the embodiment of the present application, the number of the processor 01 , the communication interface 02 , the memory 03 , and the communication bus 04 is at least one, and the processor 01 , the communication interface 02 , and the memory 03 communicate with each other through the communication bus 04 .

[0089] The processor 01 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0090] The memory 03 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0091] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to execute the facial information display method described in the method embodiment.

[0092] Optionally, the detailed functions and extended functions of the program may refer to the description of the facial information display method in the method embodiment.

[0093] An embodiment of the present application also provides a storage medium, which can store a program suitable for execution by a processor, and the program is used to execute the facial information display method described in the method embodiment.

[0094] Specifically, the storage medium may be a computer-readable storage medium, and the computer-readable storage medium may be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk or a ROM.

[0095] Optionally, the detailed functions and extended functions of the program may refer to the description of the facial information display method in the method embodiment.

[0096] In addition, the functional modules in the various embodiments of the present disclosure can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. If the function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a live broadcast device, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present disclosure.

[0097] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0098] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0099] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A facial information display method, characterized in that: include: Get each frame of the user's video stream; Determining a specific face region for each frame of the video stream; comprising: utilizing a coefficient acquisition module of a preset face region model to extract coefficients from each frame of the image to obtain coordinate parameters of each frame of the image in a rectangular coordinate system and a confidence level of each frame of the image; utilizing a specific face region acquisition module of the face region model to integrate the coordinate parameters of each frame of the image in a rectangular coordinate system and the confidence level of each frame of the image to obtain a specific face region for each frame of the image; wherein the face region model is obtained by using each frame of the acquired face dataset sample as a training sample and using the actual specific face region of each frame of the face dataset sample as a sample label for training; Based on the confidence level, images that do not contain exact facial data in the specific facial area are deleted, and multiple expression base parameters of the video stream are determined according to the specific facial area of ​​each remaining frame image; including: using a preset expression base parameter model to process the specific facial area of ​​each frame image to obtain multiple expression base parameters of the video stream; wherein the expression base parameter model is obtained by using the specific facial area of ​​each frame image in the acquired facial data set sample as a training sample and the real expression base parameters of the facial data set sample as sample labels for training; using LiveLink Face software to record a real-person video, collecting 52 expression base parameters of each frame image from the recorded video and cropping the accurate facial area of ​​each frame image, and forming the accurate facial area of ​​each frame image and the corresponding 52 expression base parameters into a specific facial area sample; converting the expression base parameter model after training; Obtaining facial information of the user according to the multiple expression base parameters, and combining the facial information of the user with a preset facial model of a cartoon character to obtain target facial information; The target facial information is displayed.

2. The method according to claim 1, characterized in that The process of obtaining samples of the face dataset includes: Generate fake face datasets using a pre-set adversarial generative network model; Performing data enhancement processing on the fake face dataset to obtain samples of the face dataset.

3. The method according to claim 1, characterized in that The coordinate parameters of each frame image in the rectangular coordinate system include: The horizontal coordinate of the upper left corner of the image, the vertical coordinate of the upper left corner of the image, the horizontal coordinate of the lower right corner of the image, and the vertical coordinate of the lower right corner of the image.

4. The method according to claim 1, wherein The process of setting the facial model of the cartoon character includes: Obtaining the facial features, skin color, hair texture, and hairstyle of the user based on the specific facial region of each frame of image; The facial model of the cartoon character is obtained according to the facial features, skin color, hair texture and hairstyle of the user.

5. A facial information display device, characterized in that: include: The video stream acquisition module is used to obtain each frame of the user's video stream; A face detection module is configured to determine, based on each frame of the video stream, a specific face region of each frame; the module comprises: utilizing a coefficient acquisition module of a preset face region model to extract coefficients from each frame to obtain coordinate parameters of each frame in a rectangular coordinate system and a confidence level of each frame; utilizing a specific face region acquisition module of the face region model to integrate the coordinate parameters of each frame in a rectangular coordinate system and the confidence level of each frame to obtain a specific face region of each frame; wherein the face region model is obtained by using each frame of an acquired face dataset sample as a training sample and using the actual specific face region of each frame of the face dataset sample as a sample label for training; An expression coefficient regression module is used to delete images that do not contain exact facial data in a specific facial area based on the confidence level, and determine multiple expression base parameters of the video stream according to the specific facial area of ​​each remaining frame image; comprising: using a preset expression base parameter model to process the specific facial area of ​​each frame image to obtain multiple expression base parameters of the video stream; wherein the expression base parameter model is obtained by using the specific facial area of ​​each frame image in the acquired facial dataset sample as a training sample and the real expression base parameters of the facial dataset sample as sample labels for training; using Live Link Face software to record a real-person video, collecting 52 expression base parameters of each frame image from the recorded video and cropping the accurate facial area of ​​each frame image, and forming the accurate facial area of ​​each frame image and the corresponding 52 expression base parameters into a specific facial area sample; and converting the expression base parameter model after training; A driving module, which obtains the user's facial information according to the multiple expression base parameters, and combines the user's facial information with a preset cartoon character's facial model to obtain target facial information; A display module is used to display the target facial information.

6. A facial information display device, characterized in that: including memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the various steps of the facial information display method according to any one of claims 1 to 4.

7. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the facial information display method according to any one of claims 1 to 4 is implemented.

Images