A method, medium and system for simulating lip shapes with tone for a three-dimensional avatar

Abstract

The application provides a three-dimensional image with emotional mouth shape simulation method, medium and system, and belongs to the three-dimensional image simulation technical field. The three-dimensional image with emotional mouth shape simulation method comprises the following steps: a tester reads a text with emotional change marks, and simultaneously collects a reading video of the tester; a three-dimensional coordinate system is established, and a tester face image at an emotional change moment is obtained; changes of the tester face image are determined, and a mixed Gaussian background model is used to distinguish changed foreground points and background points; a three-dimensional virtual image mouth shape model is established, all Gaussian categories are used as foreground key points of emotional mouth shape, and the three-dimensional virtual image mouth shape model is adjusted according to the emotional mouth shape key points; a three-dimensional image reads a text with emotional change marks according to needs, a phoneme mouth shape driving method is used to generate and output a three-dimensional image with emotional mouth shape sequence according to the three-dimensional virtual image mouth shape model after emotional adjustment.

Description

Technical Field

[0001] This invention belongs to the field of three-dimensional image simulation technology, specifically, it relates to a method, medium and system for simulating the lip movements of a three-dimensional image with intonation. Background Technology

[0002] In daily life, people often speak with intonation. However, most current 3D avatars do not consider intonation and are directly driven by lip movements, resulting in poor realism in the generated 3D avatar speaking process.

[0003] Chinese invention patent CN111081270B (application number CN201911314031.3) discloses a real-time audio-driven method for lip-sync control of virtual characters. The method includes the following steps: identifying pixel probabilities from a real-time audio stream; filtering the pixel probabilities; converting the sampling rate of the pixel probabilities to the same sampling rate as the virtual character's rendering frame rate; and converting the pixel probabilities into a standard lip-sync configuration and rendering the lip-sync. This method avoids the requirement to synchronously transmit phoneme sequences or lip-sync sequences when transmitting the audio stream, significantly reducing system complexity, coupling, and implementation difficulty, and is suitable for various application scenarios involving rendering virtual characters on display devices.

[0004] The invention described above can synchronize the lip movements of virtual characters with real-time audio, but it cannot solve the technical problem of controlling the lip movements of virtual characters with audio containing intonation. Summary of the Invention

[0005] In view of this, the present invention provides a method, medium and system for simulating the lip movements of a three-dimensional image with intonation, which can solve the technical problem of controlling the lip movements of virtual characters with intonation audio.

[0006] This invention is implemented as follows:

[0007] A first aspect of the present invention provides a method for simulating the lip movements of a three-dimensional image with intonation, comprising the following steps:

[0008] S10: Testers read aloud text marked with tone changes, and video recordings of the testers reading aloud are captured simultaneously.

[0009] S20: Establish a three-dimensional coordinate system and acquire a first image of the test subject's face taken at the moment of tone change and a second image of the test subject's face taken at the moment of tone stability;

[0010] S30: Determine a first change detection region of the first image and a second change detection region of the second image; wherein, the first change detection region is the lip region in the first image, and the second change detection region is the lip region in the second image;

[0011] S40: Obtain all first patches of the first change detection region and all second patches of the second change detection region;

[0012] S50: Perform deformation matching between the first all facets and the second all facets and preset representative facets respectively to obtain the first facet matching result of the first change detection area and the second facet matching result of the second change detection area;

[0013] S60: Perform local feature matching on the first patch matching result and the second patch matching result to obtain the change value of the first patch matching result compared with the second patch matching result;

[0014] S70: Convert the first image into a mask image, and input the mask image into a Gaussian mixture background model to obtain the Gaussian category of the lip-shape key points in the region to be detected output by the Gaussian mixture background model. The Gaussian category includes foreground and background.

[0015] S80: Establish a three-dimensional virtual image lip-sync model, use all Gaussian categories as foreground key points as tone lip-sync key points, and adjust the tone of the three-dimensional virtual image lip-sync model using tone lip-sync key points;

[0016] S90: As needed, the three-dimensional image reads aloud text with tone change markers. The phoneme-driven method generates a lip-shape model sequence based on the lip-shape model of the three-dimensional virtual image after tone adjustment, which serves as the lip-shape sequence of the three-dimensional image with tone.

[0017] Based on the above technical solution, the method for simulating lip movements with intonation in a three-dimensional image according to the present invention can be further improved as follows:

[0018] Specifically, step S20 includes:

[0019] Establish a three-dimensional coordinate system based on the MPEG-4 standard;

[0020] The acquisition of a first image of the test subject's face taken at the moment of tone change and a second image of the test subject's face taken at the moment of tone stability includes:

[0021] Acquire the original images of the test subject's face at the moment of tone change and the original images of the test subject's face at the moment of tone stability;

[0022] Features are extracted from the original images of the test subject's face at moments of change in tone of voice and from the original images of the test subject's face at moments of stable tone of voice; wherein, the features include facial key points and texture features;

[0023] Based on the aforementioned features, determine the tone of the test subject's face at the moment of tone change and the tone of the test subject's face at the moment of tone stability;

[0024] Based on the tone of the test subject's face at the moment of tone change, the original image of the test subject's face at the moment of tone change is subjected to corresponding dehazing or noise reduction image enhancement processing to obtain the first image of the test subject's face at the moment of tone change.

[0025] Based on the tone of the test subject's face at the moment of stable tone, the original image of the test subject's face at the moment of stable tone is subjected to corresponding dehazing or noise reduction image enhancement processing to obtain a second image of the test subject's face taken at the moment of stable tone.

[0026] Specifically, step S30 includes:

[0027] Image enhancement is performed on the first change detection region and the second change detection region respectively to obtain the enhanced image of the first change detection region and the enhanced image of the second change detection region.

[0028] The enhanced images of the first change detection region and the second change detection region are respectively filtered by a mean filter to obtain the filtered images of the first change detection region and the second change detection region.

[0029] Edge detection and patch finding are performed on the filtered images of the first change detection region and the second change detection region, respectively, to obtain the first all patches of the first change detection region and the second all patches of the second change detection region.

[0030] Specifically, step S50 includes:

[0031] Calculate the first normalized central moments of all the first facets and the second normalized central moments of all the second facets, respectively;

[0032] The first patch feature is calculated based on the first normalized central moment; and the second patch feature is calculated based on the second normalized central moment; wherein, both the first patch feature and the second patch feature include a center, an arc length, and an area;

[0033] The first similarity between the first set of all faces and the representative face is calculated based on the first normalized central moments and the first face features; and the second similarity between the second set of all faces and the representative face is calculated based on the second normalized central moments and the second face features.

[0034] The first patch matching result of the first change detection region is obtained based on the first similarity, and the second patch matching result of the second change detection region is obtained based on the second similarity.

[0035] Specifically, step S60 includes:

[0036] Local feature matching is performed on the first patch matching result and the second patch matching result to obtain the change value of the first patch matching result compared with the second patch matching result, specifically:

[0037] The first set of all feature points of the first patch matching result and the second set of all feature points of the second patch matching result are generated using the Hessian matrix.

[0038] By convolving the first patch matching result and the second patch matching result with box filters of different sizes, the first scale space of the first patch matching result and the second scale space of the second patch matching result are obtained.

[0039] Based on all the first feature points of the first patch matching result and the first scale space of the first patch matching result, feature points are located to obtain a first stable feature point set; and based on all the second feature points of the second patch matching result and the second scale space of the second patch matching result, feature points are located to obtain a second stable feature point set.

[0040] Using the first stable feature point set and the second stable feature point set as the center, statistically analyze the Harr wavelet features within a preset radius to obtain the first principal direction set of the first stable feature point set and the second principal direction set of the second stable feature point set;

[0041] A first feature point description subset is generated based on the first principal direction set, and a second feature point description subset is generated based on the second principal direction set;

[0042] Based on the first feature point description subset and the second feature point description subset, and using the least squares method to calculate the matching degree between any point in the first stable feature point set and all points in the second stable feature point set, the corresponding point in the second stable feature point set for any point in the first stable feature point set is obtained based on the matching degree.

[0043] A matching point pair is formed by any point in the first stable feature point set and its corresponding point in the second stable feature point set;

[0044] The change in the first patch matching result compared to the second patch matching result is calculated based on the matching point pairs.

[0045] Specifically, step S80 includes:

[0046] Step 1: Based on the MPEG-4 standard, use blendshape to create a 3D virtual lip-sync model;

[0047] Step 2: Select the basic key points in the 3D virtual image lip shape model corresponding to the key points of tone and mouth shape, and update the coordinates of the basic key points using the coordinates of the key points of tone and mouth shape.

[0048] Step 3: Replace the coordinates of the corresponding key points in the 3D virtual image lip-sync model with the updated coordinates of the basic key points.

[0049] Furthermore, the specific steps for updating the coordinates of the basic key points using the coordinates of the lip-sync key points include:

[0050] Step 1: Calculate the similarity between the coordinates of each tone and mouth shape key point and the coordinates of its corresponding basic key point;

[0051] Step 2: Select tone and mouth shape key points with a similarity of less than 0.618, and use the center point between the coordinates of the tone and mouth shape key points and the coordinates of their corresponding basic key points as the coordinates of the basic key points to update the coordinates of the basic key points.

[0052] Step 3: Calculate the similarity of the updated base keypoint coordinates. If the similarity result is less than 0.618, iterate through the following steps:

[0053] The coordinates of the basic key points are updated by using the center point between the coordinates of the lip-sync key points and the coordinates of their corresponding basic key points; until the similarity is greater than or equal to 0.618.

[0054] S90: As needed, the three-dimensional image reads aloud text with tone change markers. The phoneme-driven method generates an output lip shape model sequence based on the lip shape model of the three-dimensional virtual image after tone adjustment, which serves as the lip shape of the three-dimensional image with tone.

[0055] Step S90 specifically includes:

[0056] Step 1: Obtain the text that needs to be read aloud with tone change markers in 3D;

[0057] Step 2: Establish a phoneme set based on the phonemes corresponding to the characters in the text;

[0058] Step 3: Based on the phoneme set, search for and preload the corresponding three-dimensional image mouth shape in the preset three-dimensional mouth shape library as the basic mouth shape;

[0059] Step 4: Replace the lip movements corresponding to the tone change markers in the text with the 3D virtual avatar lip movement model after tone adjustment;

[0060] Step 5: Obtain the lip-sync sequence of the three-dimensional image with intonation.

[0061] A second aspect of the present invention provides a computer-readable storage medium, wherein computer program instructions are stored on the computer-readable storage medium; when the computer program instructions are executed by a processor, they implement the above-described method for lip-syncing with intonation in a three-dimensional image.

[0062] A third aspect of the present invention provides a three-dimensional image lip-syncing system with intonation, comprising: a computer-readable storage medium as described above.

[0063] Compared with existing technologies, the beneficial effects of the three-dimensional image lip-syncing simulation method, medium, and system provided by this invention are as follows: When a three-dimensional image is needed to read text aloud, tone marking is performed, and a first image of the test subject's face taken at the moment of tone change and a second image of the test subject's face taken at the moment of tone stabilization are acquired; it can collect facial changes when the test subject's tone changes, use a Gaussian mixture background model to filter key points related to facial changes, and record them using key points of tone and lip-syncing. Compared with current point cloud models, it can represent tone changes with a small number of key points, which can greatly improve simulation efficiency. Attached Figure Description

[0064] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a flowchart of a three-dimensional image lip-syncing method with intonation disclosed in this invention;

[0066] Figure 2 A schematic diagram of the 84 Face Definition Parameters (FDP) in the MPEG-4 standard; Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0069] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0070] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0071] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0072] like Figure 1 The diagram shown is a flowchart of a method for simulating the lip movements of a three-dimensional image with intonation, provided by the first aspect of the present invention. This method includes the following steps:

[0073] S10: Testers read aloud text marked with tone changes, and video recordings of the testers reading aloud are captured simultaneously.

[0074] S20: Establish a three-dimensional coordinate system and acquire the first image of the test subject's face taken at the moment of tone change and the second image of the test subject's face taken at the moment of tone stability.

[0075] S30: Determine a first change detection region in the first image and a second change detection region in the second image; wherein, the first change detection region is the lip region in the first image, and the second change detection region is the lip region in the second image;

[0076] S40: Obtain all first faces of the first change detection region and all second faces of the second change detection region;

[0077] S50: Perform deformation matching between the first all-face patches and the second all-face patches and the preset representative face patches respectively to obtain the first face patch matching result of the first change detection area and the second face patch matching result of the second change detection area;

[0078] S60: Perform local feature matching on the first patch matching result and the second patch matching result to obtain the change value of the first patch matching result compared with the second patch matching result;

[0079] S70: Convert the first image into a mask image, and input the mask image into a Gaussian mixture background model to obtain the Gaussian categories of the lip key points in the detection area output by the Gaussian mixture background model. The Gaussian categories include foreground and background.

[0080] S80: Establish a three-dimensional virtual image lip-sync model, use all Gaussian categories as foreground key points as tone lip-sync key points, and adjust the tone of the three-dimensional virtual image lip-sync model using tone lip-sync key points;

[0081] S90: As needed, the three-dimensional image reads aloud text with tone change markers. The phoneme-driven method generates a lip-shape model sequence based on the lip-shape model of the three-dimensional virtual image after tone adjustment, which serves as the lip-shape sequence of the three-dimensional image with tone.

[0082] Among them, the text with tone change markings refers to the text in which the tone of the tester is marked when pronouncing the text, including tone and duration. The tester needs to read the text with the tone, and the moment of tone change is marked in the reading video according to the text with tone change markings.

[0083] The first and second image regions are the same, both being images of the tester's face. Due to changes in tone, the same pronunciation will produce different facial images. Therefore, the tester's facial regions are recorded as the first and second images respectively according to the different tones.

[0084] In this scheme, the facial description adopts a face animation system based on the MPEG-4 standard, such as... Figure 2As shown, MPEG-4 is an international multimedia compression standard with universal applicability. It includes a series of sub-standards, such as audio / video conversion, encoding, and facial animation standards. Due to the crucial role of human emotional expression and human-computer interaction in multimedia systems, MPEG-4 has a dedicated section for describing the definition and synthesis of 3D facial animation. In this section, MPEG-4 defines Facial Definition Parameters (FDP), Facial Animation Parameters (FAP), and Facial Animation Parameter Units (FAPU, FAP). The role of FDP is to characterize the positional information and texture features of a specific 3D facial model; the role of FAP is to realize the movement of the facial features; and the role of FAPU is to describe the differences between different faces in certain typical features, making FAP universal. By defining the usage of these parameters, MPEG-4 establishes a complete scheme for describing the synthesis of facial animation.

[0085] MPEG-4 provides the location of 84 feature points on the human face. This scheme uses the feature points of the mouth as the key points of the mouth shape. The key points are connected to form a three-dimensional mesh structure with multiple facets.

[0086] A mask image refers to an image, graphic, or object used to occlude (fully or partially) the image to be processed, thereby controlling the area or process of image processing. The specific image or object used for occlusion is called a mask or template. In optical image processing, masks can be films, filters, etc. In digital image processing, masks are two-dimensional matrix arrays, and sometimes multi-valued images are also used.

[0087] In digital image processing, image masks are mainly used for:

[0088] ① Extract the region of interest (ROI). Multiply the image to be processed with a pre-made ROI mask to obtain the ROI image. The image values ​​within the ROI remain unchanged, while the image values ​​outside the ROI are all 0.

[0089] ② Masking effect. Using a mask to shield certain areas of an image, preventing them from participating in processing or the calculation of processing parameters, or processing or statistics only on the shielded area.

[0090] ③ Structural feature extraction. Similarity variables or image matching methods are used to detect and extract structural features in the image that are similar to the mask.

[0091] ④ Creation of specially shaped images. Using selected images, graphics, or objects to occlude (fully or partially) the image to be processed, thereby controlling the area or process of image processing. The specific image or object used for occlusion is called a mask or template.

[0092] Gaussian mixture background modeling is suitable for separating the background and foreground from an image sequence when the camera is fixed. Pixel-based Gaussian mixture models are effective for modeling multi-peak backgrounds and are robust to objects with repetitive motion, such as leaves rustling in a breeze.

[0093] The Hessian matrix, also known as the Hessian matrix, is a square matrix composed of the second-order partial derivatives of a multivariable function, describing the local curvature of the function. It was first proposed in the 19th century by the German mathematician Ludwig Otto Hesse and named after him. The Hessian matrix is ​​frequently used in Newton's method for solving optimization problems and can be used to determine the extrema of multivariable functions. In the optimization design of practical engineering problems, the objective function is often very complex. To simplify the problem, the objective function is often expanded into a Taylor polynomial in the neighborhood of a point to approximate the original function. In this case, the matrix form of the Taylor expansion of the function at that point involves the Hessian matrix.

[0094] In the above technical solution, step S20 specifically includes:

[0095] Establish a three-dimensional coordinate system based on the MPEG-4 standard;

[0096] Acquire a first image of the test subject's face taken at moments of tone change and a second image of the test subject's face taken at moments of stable tone, including:

[0097] Acquire the original images of the test subject's face at the moment of tone change and the original images of the test subject's face at the moment of stable tone;

[0098] Features are extracted from the original images of the test subject's face at moments of change in tone of voice and from the original images of the test subject's face at moments of stable tone of voice; the features include facial key points and texture features.

[0099] Based on the characteristics, determine the tone of the tester's face when the tone changes and the tone of the tester's face when the tone is stable;

[0100] Based on the tone of voice of the tester's face at the moment of tone change, the original image of the tester's face at the moment of tone change is subjected to corresponding dehazing or noise reduction image enhancement processing to obtain the first image of the tester's face at the moment of tone change.

[0101] Based on the tone of the test subject's face at the moment of stable speech, the original image of the test subject's face at the moment of stable speech is subjected to corresponding dehazing or noise reduction image enhancement processing to obtain a second image of the test subject's face taken at the moment of stable speech.

[0102] In the above technical solution, step S30 specifically includes:

[0103] Image enhancement is performed on the first change detection region and the second change detection region respectively to obtain the enhanced image of the first change detection region and the enhanced image of the second change detection region.

[0104] The enhanced images of the first and second change detection regions are filtered by a mean filter to obtain the filtered images of the first and second change detection regions, respectively.

[0105] Edge detection and patch finding are performed on the filtered images of the first and second change detection regions, respectively, to obtain the first all patches of the first change detection region and the second all patches of the second change detection region.

[0106] In the above technical solution, step S50 specifically includes:

[0107] Calculate the first normalized central moments of all first faces and the second normalized central moments of all second faces respectively;

[0108] The first patch feature is calculated based on the first normalized central moment; and the second patch feature is calculated based on the second normalized central moment; wherein both the first patch feature and the second patch feature include the center, arc length and area;

[0109] The first similarity between all first facets and the representative facet is calculated based on the first normalized central moments and the first facet features; and the second similarity between all second facets and the representative facet is calculated based on the second normalized central moments and the second facet features.

[0110] The first patch matching result of the first change detection region is obtained based on the first similarity, and the second patch matching result of the second change detection region is obtained based on the second similarity.

[0111] In the above technical solution, step S60 specifically includes:

[0112] Local feature matching is performed on the first and second patch matching results to obtain the change value of the first patch matching result compared to the second patch matching result, specifically:

[0113] The first set of all feature points of the first patch matching result and the second set of all feature points of the second patch matching result are generated using the Hessian matrix.

[0114] By convolving the first patch matching result and the second patch matching result with box filters of different sizes, the first scale space of the first patch matching result and the second scale space of the second patch matching result are obtained.

[0115] Based on the first set of all feature points of the first patch matching result and the first scale space of the first patch matching result, feature points are located to obtain a first stable set of feature points; and based on the second set of all feature points of the second patch matching result and the second scale space of the second patch matching result, feature points are located to obtain a second stable set of feature points.

[0116] Using the first stable feature point set and the second stable feature point set as the center, statistically analyze the Harr wavelet features within a preset radius to obtain the first principal direction set of the first stable feature point set and the second principal direction set of the second stable feature point set;

[0117] A first feature point description subset is generated based on the first principal direction set, and a second feature point description subset is generated based on the second principal direction set;

[0118] Based on the first and second feature point descriptor subsets, and using the least squares method to calculate the matching degree between any point in the first stable feature point set and all points in the second stable feature point set, the corresponding point in the second stable feature point set is obtained based on the matching degree.

[0119] A matching point pair is formed by any point in the first stable feature point set and its corresponding point in the second stable feature point set.

[0120] The change in the matching result of the first facet is calculated based on the matching point pairs compared to the matching result of the second facet.

[0121] In the above technical solution, step S80 specifically includes:

[0122] Step 1: Based on the MPEG-4 standard, use blendshape to create a 3D virtual lip-sync model;

[0123] Step 2: Select the basic key points in the 3D virtual image lip shape model corresponding to the key points of tone and mouth shape, and update the coordinates of the basic key points using the coordinates of the key points of tone and mouth shape.

[0124] Step 3: Replace the coordinates of the corresponding key points in the 3D virtual image lip-sync model with the updated coordinates of the basic key points.

[0125] Furthermore, in the above technical solution, the specific steps for updating the coordinates of the basic key points using the coordinates of the lip-sync key points include:

[0126] Step 1: Calculate the similarity between the coordinates of each tone and mouth shape key point and the coordinates of its corresponding basic key point;

[0127] Step 2: Select tone and mouth shape key points with a similarity of less than 0.618, and use the center point between the coordinates of the tone and mouth shape key points and the coordinates of their corresponding basic key points as the coordinates of the basic key points to update the coordinates of the basic key points.

[0128] Step 3: Calculate the similarity of the updated base keypoint coordinates. If the similarity result is less than 0.618, iterate through the following steps:

[0129] The coordinates of the basic key points are updated by using the center point between the coordinates of the lip-sync key points and the coordinates of their corresponding basic key points; until the similarity is greater than or equal to 0.618.

[0130] In the above technical solution, S90: as needed, the three-dimensional image reads aloud the text with tone change markers, and generates an output lip shape model sequence based on the lip shape model of the three-dimensional virtual image after tone adjustment using the phoneme lip shape driven method, which serves as the lip shape of the three-dimensional image with tone.

[0131] Step S90 specifically includes:

[0132] Step 1: Obtain the text that needs to be read aloud with tone change markers in 3D;

[0133] Step 2: Establish a phoneme set based on the phonemes corresponding to the characters in the text;

[0134] Step 3: Based on the phoneme set, search for and preload the corresponding 3D image mouth shape in the preset 3D mouth shape library as the basic mouth shape;

[0135] Step 4: Replace the lip movements corresponding to the tone change markers in the text with the 3D virtual avatar lip movement model after tone adjustment;

[0136] Step 5: Obtain the lip-sync sequence of the three-dimensional image with intonation.

[0137] A second aspect of the present invention provides a computer-readable storage medium, wherein computer program instructions are stored on the computer-readable storage medium; when the computer program instructions are executed by a processor, they implement the above-described method for simulating the lip movements of a three-dimensional image with intonation.

[0138] A third aspect of the present invention provides a three-dimensional image lip-syncing system with intonation, comprising: a computer-readable storage medium as described above.

[0139] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for simulating lip movements in a three-dimensional image with intonation, characterized in that, Includes the following steps: S10: Testers read aloud text marked with tone changes, and video recordings of the testers reading aloud are captured simultaneously. S20: Establish a three-dimensional coordinate system and acquire a first image of the test subject's face taken at the moment of tone change and a second image of the test subject's face taken at the moment of tone stability; S30: Determine a first change detection region of the first image and a second change detection region of the second image; wherein, the first change detection region is the lip region in the first image, and the second change detection region is the lip region in the second image; S40: Obtain all first patches of the first change detection region and all second patches of the second change detection region; S50: Perform deformation matching between the first all facets and the second all facets and preset representative facets respectively to obtain the first facet matching result of the first change detection area and the second facet matching result of the second change detection area; S60: Perform local feature matching on the first patch matching result and the second patch matching result to obtain the change value of the first patch matching result compared with the second patch matching result; S70: Convert the first image into a mask image, and input the mask image into a Gaussian mixture background model to obtain the Gaussian category of the lip-shape key points in the first change detection region output by the Gaussian mixture background model. The Gaussian category includes foreground and background. S80: Establish a three-dimensional virtual image lip-sync model, use all Gaussian categories as foreground key points as tone lip-sync key points, and adjust the tone of the three-dimensional virtual image lip-sync model using tone lip-sync key points; S90: As needed, the three-dimensional image reads aloud text with tone change markers. The phoneme mouth shape driving method generates a mouth shape model sequence based on the three-dimensional virtual image mouth shape model after tone adjustment, which serves as the mouth shape sequence of the three-dimensional image with tone. Specifically, step S60 includes: Local feature matching is performed on the first patch matching result and the second patch matching result to obtain the change value of the first patch matching result compared with the second patch matching result, specifically: The first set of all feature points of the first patch matching result and the second set of all feature points of the second patch matching result are generated using the Hessian matrix. By convolving the first patch matching result and the second patch matching result with box filters of different sizes, the first scale space of the first patch matching result and the second scale space of the second patch matching result are obtained. Based on all the first feature points of the first patch matching result and the first scale space of the first patch matching result, feature points are located to obtain a first stable feature point set; and based on all the second feature points of the second patch matching result and the second scale space of the second patch matching result, feature points are located to obtain a second stable feature point set. Using the first stable feature point set and the second stable feature point set as the center, statistically analyze the Harr wavelet features within a preset radius to obtain the first principal direction set of the first stable feature point set and the second principal direction set of the second stable feature point set; A first feature point description subset is generated based on the first principal direction set, and a second feature point description subset is generated based on the second principal direction set; Based on the first feature point description subset and the second feature point description subset, and using the least squares method to calculate the matching degree between any point in the first stable feature point set and all points in the second stable feature point set, the corresponding point in the second stable feature point set for any point in the first stable feature point set is obtained based on the matching degree. A matching point pair is formed by any point in the first stable feature point set and its corresponding point in the second stable feature point set; The change in the first patch matching result compared to the second patch matching result is calculated based on the matching point pairs.

2. The method for simulating lip movements in a three-dimensional image with intonation according to claim 1, characterized in that, Step S20 specifically includes: Establish a three-dimensional coordinate system according to the MPEG-4 standard; The acquisition of a first image of the test subject's face taken at the moment of tone change and a second image of the test subject's face taken at the moment of tone stability includes: Acquire the original images of the test subject's face at the moment of tone change and the original images of the test subject's face at the moment of tone stability; Features are extracted from the original images of the test subject's face at moments of change in tone of voice and from the original images of the test subject's face at moments of stable tone of voice; wherein, the features include facial key points and texture features; Based on the aforementioned features, determine the tone of the test subject's face at the moment of tone change and the tone of the test subject's face at the moment of tone stability; Based on the tone of the test subject's face at the moment of tone change, the original image of the test subject's face at the moment of tone change is subjected to corresponding dehazing or noise reduction image enhancement processing to obtain the first image of the test subject's face at the moment of tone change. Based on the tone of the test subject's face at the moment of stable tone, the original image of the test subject's face at the moment of stable tone is subjected to corresponding dehazing or noise reduction image enhancement processing to obtain a second image of the test subject's face taken at the moment of stable tone.

3. The method for simulating lip movements in a three-dimensional image with intonation according to claim 1, characterized in that, Step S30 specifically includes: Image enhancement is performed on the first change detection region and the second change detection region respectively to obtain the enhanced image of the first change detection region and the enhanced image of the second change detection region. The enhanced images of the first change detection region and the second change detection region are respectively filtered by a mean filter to obtain the filtered images of the first change detection region and the second change detection region. Edge detection and patch finding are performed on the filtered images of the first change detection region and the second change detection region, respectively, to obtain the first all patches of the first change detection region and the second all patches of the second change detection region.

4. The method for simulating lip movements in a three-dimensional image with intonation according to claim 1, characterized in that, Step S50 specifically includes: Calculate the first normalized central moments of all the first facets and the second normalized central moments of all the second facets, respectively; The first patch feature is calculated based on the first normalized central moment; and the second patch feature is calculated based on the second normalized central moment; wherein, both the first patch feature and the second patch feature include a center, an arc length, and an area; The first similarity between the first set of all faces and the representative face is calculated based on the first normalized central moments and the first face features; and the second similarity between the second set of all faces and the representative face is calculated based on the second normalized central moments and the second face features. The first patch matching result of the first change detection region is obtained based on the first similarity, and the second patch matching result of the second change detection region is obtained based on the second similarity.

5. The method for simulating lip movements in a three-dimensional image with intonation according to claim 1, characterized in that, Step S80 specifically includes: Step 1: Based on the MPEG-4 standard, use blendshape to create a 3D virtual lip-sync model; Step 2: Select the basic key points in the 3D virtual image lip shape model corresponding to the key points of tone and mouth shape, and update the coordinates of the basic key points using the coordinates of the key points of tone and mouth shape. Step 3: Replace the coordinates of the corresponding key points in the 3D virtual image lip-sync model with the updated coordinates of the basic key points.

6. The method for simulating lip movements in a three-dimensional image with intonation according to claim 5, characterized in that, The specific steps for updating the coordinates of basic key points using the coordinates of lip-sync key points include: Step 1: Calculate the similarity between the coordinates of each tone and mouth shape key point and the coordinates of its corresponding basic key point; Step 2: Select tone and mouth shape key points with a similarity of less than 0.618, and use the center point between the coordinates of the tone and mouth shape key points and the coordinates of their corresponding basic key points as the coordinates of the basic key points to update the coordinates of the basic key points. Step 3: Calculate the similarity of the updated base keypoint coordinates. If the similarity result is less than 0.618, iterate through the following steps: The coordinates of the basic key points are updated by using the center point between the coordinates of the lip-sync key points and the coordinates of their corresponding basic key points; until the similarity is greater than or equal to 0.

618.

7. The method for simulating lip movements in a three-dimensional image with intonation according to claim 6, characterized in that, Step S90 specifically includes: Step 1: Obtain the text that needs to be read aloud with tone change markers in 3D; Step 2: Establish a phoneme set based on the phonemes corresponding to the characters in the text; Step 3: Based on the phoneme set, search for and preload the corresponding three-dimensional image mouth shape in the preset three-dimensional mouth shape library as the basic mouth shape; Step 4: Replace the lip movements corresponding to the tone change markers in the text with the 3D virtual avatar lip movement model after tone adjustment; Step 5: Obtain the lip-sync sequence of the three-dimensional image with intonation.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, they implement a method for simulating the lip movements of a three-dimensional image with intonation as described in any one of claims 1 to 7.

9. A three-dimensional lip-syncing system with expressive features, characterized in that, include: The computer-readable storage medium as described in claim 8.

Images