A desktop digital human interactive broadcasting method based on JavaFX

Abstract

The present application provides a JavaFX-based desktop digital human interactive broadcasting method, comprising: scanning a real person's face with a depth camera to obtain three-dimensional shape data of facial tissue, segmenting and annotating the data in combination with medical anatomical knowledge, and establishing a digital human facial model, including the geometric shape and texture features of the facial tissue structures of the lips, tongue, and teeth; establishing tooth visibility control rules based on the mapping relationship between tongue position and phonemes during broadcasting, and dynamically adjusting the visibility of the digital person's teeth during broadcasting based on the tooth visibility parameters of the phonetic sequence, so that the tooth visibility changes match the characteristics of the real person's pronunciation during broadcasting; and tracking the shape and texture features of the digital person's lips, tongue, and teeth areas in real time during broadcasting, comparing them with the target broadcast lip shape, tongue position, and tooth visibility parameters, dynamically adjusting the digital human facial model parameters, and performing coordinated control of the lips, tongue, and teeth during broadcasting.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a JavaFX-based desktop digital human interactive broadcasting method. Background Art

[0002] In researching interactive presentation methods for desktop digital humans based on JavaFX, building a high-precision facial model and achieving realistic lip animation for presentation presents a complex technical challenge. The key challenge lies in accurately capturing and reproducing the subtle changes in the human face during articulation, particularly the coordinated movements of the lips, tongue, and teeth. This requires precise modeling of multiple dimensions, including geometry, texture features, and anatomical structure. Furthermore, a precise correspondence must be established between phonemes and lip movements to achieve natural synchronization of lip animation with speech. However, the movement of facial muscles during articulation is extremely complex, and the transitions between different phonemes are also subtle. Ensuring smooth and natural animation while also capturing the subtle differences in pronunciation of different phonemes presents a significant challenge. Addressing these technical challenges is crucial to enhancing the interactive experience and application value of digital humans. Summary of the Invention

[0003] The present invention provides a JavaFX-based desktop digital human interactive presentation method, which mainly includes:

[0004] Scanning a real person's face with a depth camera to obtain 3D shape data of facial tissues, then segmenting and annotating the data based on medical anatomical knowledge to build a digital human face model, including the geometric shape and texture features of the facial tissue structures of the lips, tongue, and teeth;

[0005] Obtain the lip shape changes of different phonemes in the digital human performance, establish amplitude, speed and trajectory templates, use JavaFX animation and timeline functions to obtain the lip shape parameter sequence corresponding to the sound sequence, and realize the synthesis of the performance lip shape animation synchronized with the phonemes by controlling the lip shape changes of the digital human facial model.

[0006] Based on the obtained digital human lip shape animation, a phoneme-to-lip shape parameter mapping model is trained to obtain a digital human lip shape animation with high lip shape control accuracy and natural and coordinated changes. Through comparative learning, the similarity between the digital human lip shape animation and the lip shape of a real person is determined. If the similarity is lower than a preset threshold, the lip shape control parameters are fine-tuned to make the lip shape changes during the broadcast more similar to those of a real person, thereby improving the visual realism of the lip shape animation and obtaining the optimized broadcast lip shape animation.

[0007] Based on the optimized digital human lip animation, the lip shape feature vectors of the key frames are extracted to obtain lip shape change templates representing different phonemes in the pronunciation state. The templates are used to guide the generation of lip shape animation to ensure the coherence of the digital human's lip shape changes when performing different phonemes.

[0008] Based on the lip shape change template, a tongue position control model is constructed to obtain the tongue position parameters of different phonemes when the digital human is performing pronunciation. By controlling the changes in the shape and position of the digital human's tongue through the parameters, the tongue position animation is synchronized with the phonemes during the performance, and the mapping relationship between the tongue position and the phonemes during the performance is obtained.

[0009] Based on the mapping relationship between tongue position and phonemes during broadcasting, a tooth visibility control rule is established. According to the tooth visibility parameters of the sound sequence, the visibility of the digital human's teeth is dynamically adjusted during broadcasting, so that the changes in tooth visibility match the characteristics of the real person's pronunciation.

[0010] The shape and texture features of the digital human's lips, tongue and teeth are tracked in real time during the broadcast, compared with the target broadcast lip shape, tongue position and teeth visibility parameters, and the digital human facial model parameters are dynamically adjusted to perform coordinated control of the lips, tongue and teeth during the broadcast.

[0011] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0012] The present invention discloses a desktop digital human interactive broadcasting method based on JavaFX. A depth camera is used to scan a real person's face to obtain three-dimensional shape data, and a precise digital human facial model is established in combination with medical anatomical knowledge. JavaFX is used to achieve synchronization of phonemes and lip shape animation, and lip shape control parameters are optimized through comparative learning. Key frame feature vectors are extracted to construct a lip shape change template to guide the generation of broadcast lip shape animation. At the same time, a refined tongue position control model and tooth visibility control rules are established to achieve precise synchronization of tongue position and tooth animation with phonemes. The facial model parameters are dynamically adjusted to achieve refined coordinated control of lips, tongue and teeth. The present invention significantly improves the realism and naturalness of digital human broadcast lip animation, and provides key technical support for applications such as virtual anchors and digital human interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 The present invention is a flowchart of a JavaFX-based desktop digital human interactive presentation method.

[0014] Figure 2 The figure is a schematic diagram of a JavaFX-based desktop digital human interactive presentation method of the present invention.

[0015] Figure 3 This is another schematic diagram of a JavaFX-based desktop digital human interactive presentation method of the present invention. DETAILED DESCRIPTION

[0016] To further understand the content of the present invention, the present invention is described in detail with reference to the accompanying drawings and examples. The present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the invention are shown in the accompanying drawings.

[0017] like Figure 1-3 In this embodiment, a JavaFX-based desktop digital human interactive presentation method may specifically include:

[0018] Step S101: Scan a real person's face with a depth camera to obtain three-dimensional shape data of facial tissue. Combined with medical anatomical knowledge, the data is segmented and annotated to create a digital human face model, including the geometric shape and texture features of the facial tissue structure of the lips, tongue, and teeth.

[0019] Acquire point cloud information carrying multi-angle three-dimensional scanning data of a real person's face, the point cloud information is obtained by collecting facial tissue with a non-contact 3D laser scanner; determine the boundary areas of the lip, tongue, and tooth facial tissues based on the point cloud information and a pre-established medical anatomical knowledge base; use an octree voxelization method to convert the point cloud information into a voxel grid structure to obtain voxelized data of the facial tissue; use a region growing algorithm to segment the lip, tongue, and tooth regions based on the voxelized data, if the lip region is segmented, apply a Loop subdivision algorithm to the lip region; if the tongue region is segmented, use a deformation field simulation to the tongue region; if the tooth region is segmented, apply a Machi cube algorithm to the tooth region; receive facial tissue data containing texture image information and feature descriptors, the feature descriptors are obtained by calculating the curvature and normal vector geometric features of the facial tissue and constructing them using a point feature histogram; use a PointNet++ network to perform fine segmentation and annotation of the facial tissue based on the facial tissue data to obtain a digital human facial model containing the lip, tongue, and tooth facial tissues.

[0020] For example, a non-contact 3D laser scanner is used to perform a multi-angle 3D scan of a real face to obtain point cloud data of facial tissues including lips, tongue, and teeth, which are then merged through a multi-view registration algorithm to obtain complete 3D facial shape information. The acquired 3D facial point cloud data is subdivided based on medical anatomical knowledge to delineate the boundary areas of facial tissues such as lips, tongue, and teeth, and the octree voxelization method is used to convert the point cloud data into a voxel grid structure. Based on the voxelized data, the region growing algorithm is used to segment the lips, tongue, and teeth regions. The Loop subdivision algorithm is used to increase geometric details of the lips, the deformation field is used to simulate the soft characteristics of the tongue, and the March cube algorithm is applied to the teeth to reconstruct the precise geometric shape. Feature extraction is performed on the voxelized facial tissue structure, and geometric features such as the curvature and normal vector of each facial tissue are calculated. The feature descriptor of the facial tissue is constructed using the point feature histogram (PFH). Combining texture image information and feature descriptors, the PointNet++ network is used to perform fine segmentation and annotation of facial tissues, constructing a high-precision three-dimensional geometric model of facial tissues such as lips, tongue, and teeth, and using the pix2pix generative adversarial network to combine the segmentation results with the original texture image to generate a high-precision texture map and map it to the model surface.

[0021] To construct a high-precision digital facial model, a non-contact 3D laser scanner with a resolution of 0.1mm was used to perform a 360-degree scan of the real face, with each scan interval of 30 degrees, generating a total of 12 sets of point cloud data. Point cloud registration was performed using the Iterative Closest Point (ICP) algorithm, with a registration error threshold of 0.05mm, to achieve accurate stitching of multi-view point clouds. The merged point cloud data was then subjected to an octree voxelization algorithm with a voxel size of 0.5mm, converting the continuous point cloud into a discrete voxel structure. Facial tissue was then segmented using a density-based region growing algorithm with a growth threshold of 0.8mm, successfully extracting the lips, tongue, and teeth. A three-iteration loop subdivision algorithm was applied to the lips, increasing the number of triangular facets by fourfold and significantly improving geometric detail. The tongue was simulated using a mass-spring system with a spring constant of 0.5N / m to achieve softness. The teeth were reconstructed using the Machi cube algorithm with a mesh resolution of 0.1mm to accurately reproduce the sharp edges of the teeth. In the feature extraction stage, the average curvature and Gaussian curvature of each vertex are calculated, and the normal vector is estimated using the K-nearest neighbor method (K=10). A 153-dimensional feature descriptor is constructed using a point feature histogram (PFH) with a radius of 3 mm. These features are input into the PointNet++ network, which contains 4 layers of Set Abstraction layers and 2 layers of Feature Propagation layers to achieve semantic segmentation of the point cloud. Finally, the pix2pix generative adversarial network is used. Both the discriminator and the generator use a U-Net structure. The input is the original texture image and segmentation label map with a resolution of 256x256, and the output is a fine texture map of the same resolution. The generated texture is accurately mapped to the surface of the three-dimensional model through parametric mapping, completing the construction of a high-precision digital human face model.

[0022] Step S102: Obtain the lip shape changes of different phonemes during the digital human performance, establish amplitude, speed and trajectory templates, use JavaFX's animation and timeline functions to obtain the lip shape parameter sequence corresponding to the phonetic sequence, and realize the synthesis of the performance lip shape animation synchronized with the phonemes by controlling the lip shape changes of the digital human facial model to obtain the performance lip shape animation.

[0023] A high-speed camera is used to capture the lip shape change sequence corresponding to different phonemes during real-person pronunciation. The lip shape change sequence contains image data at 240 frames per second. Edge detection is performed on the lip shape change sequence to obtain the lip contour, and feature points are extracted from the lip contour to obtain key point coordinates. The key point coordinates are fitted using the least squares method to obtain a lip shape change curve. The amplitude, velocity, and acceleration parameters of the lip shape change are calculated based on the lip shape change curve. Once the lip shape change parameters are obtained, a phoneme lip shape change template library is constructed; the phoneme lip shape change template library includes the amplitude, velocity, and trajectory template corresponding to each phoneme. A lip shape parameter sequence is read from the phoneme lip shape change template library, which contains key point coordinates, change amplitude, and velocity information. The lip shape parameter sequence is mapped to the vertex coordinates of the digital human face model using the radial basis function interpolation method. The changes in the vertex coordinates are controlled by the animation function to obtain a lip shape animation synchronized with the phonemes.

[0024] For example, a high-speed camera is used to film a live voice at 240 frames per second to capture the lip shape change sequences corresponding to different phonemes. The lip contour is extracted using the Canny edge detection algorithm, and the SURF feature point detection algorithm is then applied to obtain keypoint coordinates. The lip shape change curve is fitted using the least squares method to calculate the amplitude, velocity, and acceleration parameters of the lip shape change. Based on the extracted lip shape change parameters, a phoneme lip shape change template library is constructed. For each phoneme, a refined amplitude, velocity, and trajectory template is created and stored as a JSON file for improved access efficiency. Using the JavaFX timeline feature, the phoneme sequence data is read and phoneme recognition is performed using a hidden Markov model. A dynamic time warping algorithm is used to achieve temporal alignment between the phoneme and the lip shape template, generating a corresponding lip shape parameter sequence containing keypoint coordinates, amplitude, and velocity information. Using JavaFX animation capabilities, the generated lip shape parameter sequence is mapped to the vertex coordinates of a digital human face model using radial basis function interpolation. Keyframes are set using the JavaFX KeyFrame and KeyValue classes, and animation playback is controlled using the Timeline class, achieving lip shape animation synthesis synchronized with the phonemes.

[0025] To achieve lip shape animation synthesis for a digital human, a high-speed camera at 240 fps was used to capture a real-person pronunciation process, capturing a video sequence containing 20 basic phonemes. The Canny edge detection algorithm was applied to each frame, with dual thresholds set to 50 and 150, to extract the lip contour. The SURF feature point detection algorithm was then used, with a Hessian matrix threshold set to 300, to extract 64 key points within the lip region. These key points were fitted using the least squares method to obtain a lip shape variation curve, and the amplitude, velocity, and acceleration parameters of the curve were calculated. Lip shape variation templates were created for each of the 20 basic phonemes. Each template contained 10 key frames, storing the coordinates, amplitude, and velocity information of the 64 key points per frame in JSON format. A 5-state hidden Markov model was used to process the input speech data for phoneme recognition, achieving an accuracy of 95%. A dynamic time warping algorithm with a window size of 5 was used to achieve temporal alignment between the phonemes and the lip shape templates, generating a complete lip shape parameter sequence. In JavaFX, radial basis function interpolation was used to select 50 control points and map the lip shape parameters to a digital facial model consisting of 5,000 vertices. Keyframes were set using the JavaFX KeyFrame class at 60 frames per second. The KeyValue class defined the interpolation of vertex coordinates, and the Timeline class controlled a 30-second animation clip, achieving smooth lip animation synthesis. The entire process, from video capture to animation generation, took approximately two minutes on a 3.5GHz processor, and the resulting animation file was approximately 20MB in size.

[0026] In step S103, a phoneme-to-lip shape parameter mapping model is trained based on the obtained digital human lip shape animation to obtain a digital human lip shape animation with high lip shape control accuracy and natural and coordinated changes. The similarity between the digital human lip shape animation and the lip shape of a real person is determined through comparative learning. If the similarity is lower than a preset threshold, the lip shape control parameters are fine-tuned to make the lip shape changes during the broadcast closer to that of a real person, thereby improving the visual realism of the lip shape animation and obtaining an optimized broadcast lip shape animation.

[0027] The system obtains a phoneme sequence and constructs a phoneme-to-lip shape parameter mapping model using a temporal convolutional network. The optimized model parameters are obtained through a backpropagation algorithm. A live-action speech video is received and a 3D facial model is reconstructed using a 3D morphable model. Key features such as lip contour, opening and closing, and lip corner position are extracted to establish a live-action lip shape feature database. Based on the digital human's lip shape sequence and the live-action lip shape sequence, a dynamic time warping algorithm is used to determine the temporal alignment relationship. The visual similarity of each frame is then determined using a structural similarity index. A similarity score is calculated between the digital human's lip shape animation and the live-action lip shape features. If the similarity score falls below a preset threshold, the parameter fine-tuning process begins. A genetic algorithm is used to fine-tune the lip shape control parameters. The genetic algorithm involves chromosome encoding of the lip shape control parameters; generating new parameter combinations through roulette wheel selection, single-point crossover, and Gaussian mutation; and iterative optimization using the similarity score as the fitness function. When the similarity score exceeds a preset threshold or the maximum number of iterations is reached, the optimized live-action lip shape animation is output.

[0028] For example, based on the digital human lip animation and the corresponding phoneme sequence, a temporal convolutional network is used to construct a mapping model from phoneme to lip shape parameters. The model parameters are optimized through the back-propagation algorithm to obtain a digital human lip animation with high lip shape control accuracy and natural and coordinated changes. The 3D facial model in the real-person pronunciation video is reconstructed using a 3D deformable model, and key features such as lip shape contour, opening and closing degree, and lip corner position are extracted to establish a real-person pronunciation lip shape feature database. The time alignment of the digital human and real-person lip shape sequences is calculated using a dynamic time warping algorithm, and the visual similarity of each frame is evaluated in combination with the structural similarity index. The digital human lip animation and the real-person pronunciation lip shape features are compared to calculate the similarity score between the two. If the similarity is lower than the preset threshold of 0.85, the parameter fine-tuning process is entered. A genetic algorithm is used to fine-tune the lip shape control parameters. The chromosome encoding represents the numerical range of the lip shape control parameters. New parameter combinations are generated through roulette wheel selection, single-point crossover and Gaussian mutation operations. The similarity score is used as the fitness function. Through iterative optimization, the lip shape changes during broadcasting are made closer to real people. When the similarity exceeds the preset threshold or reaches the maximum number of iterations, the optimized broadcast lip animation is output.

[0029] To optimize the lip-syncing animation of a digital human, a phoneme-to-lip-shape parameter mapping model consisting of an 8-layer temporal convolutional network was first constructed. The model took 40-dimensional phoneme features as input and produced 68 lip keypoint coordinates as output. Using the Adam optimizer with a learning rate of 0.001, the model was trained for 50 epochs on 100,000 phoneme-lip shape pairs, resulting in a high-precision lip shape control model with an average error of less than 0.5 mm. Subsequently, a 3D morphable model was used to reconstruct 3D faces from the live pronunciation videos, using 199 shape parameters and 40 expression parameters to accurately capture lip shape variations. 68 lip keypoints were extracted, and 10 feature parameters, such as opening and closing degree and lip corner position, were calculated to construct a 50,000-frame live lip shape feature database. Using the dynamic time warping algorithm with a window size of 15 frames, the optimal alignment path between the digital human and live lip shape sequences was calculated. The visual similarity of each frame was evaluated using a structural similarity index with weights of 0.3 for brightness, 0.3 for contrast, and 0.4 for structure. If the average similarity falls below 0.85, a genetic algorithm is initiated for parameter fine-tuning. In this genetic algorithm, the chromosome encoding uses 10 floating-point numbers to represent the lip shape control parameters. The population size is set to 100, and roulette wheel selection, single-point crossover with a crossover probability of 0.8, and Gaussian mutation with a mutation probability of 0.1 are used. Iterative optimization is performed for 50 generations or until the similarity exceeds 0.9. Finally, the optimized lip-sync animation is output, ensuring that the digital human's lip shape changes closely match the real person's pronunciation.

[0030] In step S104, based on the optimized digital human lip animation, the lip shape feature vectors of the key frames are extracted to obtain lip shape change templates representing different phonemes in the pronunciation state. The templates are used to guide the generation of the lip shape animation to ensure the consistency of the lip shape changes of the digital human when performing different phonemes.

[0031] A video sequence of lip-syncing animation is obtained, and key frames are extracted using a motion amplitude calculation method based on the optical flow method. The extraction threshold is set at 1.5 times the average motion amplitude. A hierarchical clustering algorithm is used to process the lip shape feature vectors, and Ward's minimum variance method is used as the linking criterion. Lip shape change templates representing different phonemes are obtained by calculating the cluster center. A mapping relationship between phonemes and lip shape change templates is established, and a dynamic programming algorithm is used to search for the optimal path based on the continuous phoneme sequence. The state is defined as the lip shape template corresponding to the current phoneme. For the selected lip shape change template sequence, a multidimensional cubic spline interpolation algorithm is used to generate a smooth lip shape transition curve. Each dimension of the lip shape feature is interpolated, and the interpolation result is mapped to the digital human face model. The generated lip shape sequence is processed using a Kalman filter to eliminate mutations and jitter in the lip shape sequence, resulting in a coherent and natural lip shape animation for broadcast.

[0032] For example, based on the optimized digital human lip animation, keyframes are extracted using motion amplitude calculation based on the optical flow method, with a threshold set at 1.5 times the average motion amplitude. Dimensionality reduction is performed using principal component analysis to obtain a low-dimensional lip shape feature vector for each keyframe. A hierarchical clustering algorithm is used to group the extracted lip shape feature vectors. Ward's minimum variance method is used as the linkage criterion, automatically determining the number of clusters based on the number of phonemes. Cluster centers are calculated to obtain lip shape templates representing different phonemes in the state of pronunciation. A mapping relationship between phonemes and lip shape templates is established, and a dynamic programming algorithm is used to search for the optimal path for a continuous phoneme sequence. The state is defined as the lip shape template corresponding to the current phoneme. The transfer equation considers the contextual information of the preceding and following phonemes to select the most appropriate lip shape template sequence. Based on the selected lip shape template sequence, a multidimensional cubic spline interpolation algorithm is used to generate a smooth lip shape transition curve. Each dimension of the lip shape feature is interpolated separately, and a time weight is added to adjust the transition speed. The interpolation result is then mapped to the digital human facial model. A Kalman filter is used to filter the generated lip shape sequences, eliminating sudden changes and jitter, resulting in coherent and natural lip shape animation. To optimize the digital human's lip shape animation, the Horn-Schunck optical flow algorithm is first used to calculate the motion amplitude between adjacent frames, setting a threshold of 1.5 times the average motion amplitude to extract 200 keyframes. Principal component analysis is then applied to the coordinates of the 68 lip keypoints in each keyframe, retaining the top 20 principal components that explain 95% of the variance to generate a 20-dimensional lip shape feature vector. Hierarchical clustering is then performed using Ward's minimum variance method, with the number of clusters set to 40. Assuming there are 40 phonemes, 40 lip shape change templates are generated. After establishing the mapping between phonemes and lip shape templates, a Viterbi algorithm is used for dynamic programming. The state space is composed of 40 lip shape templates, with transition probabilities based on phoneme co-occurrence frequencies and observation probabilities determined by the phoneme recognition confidence. For a 10-second speech clip containing approximately 300 phonemes, the algorithm completes the optimal path search within 0.1 seconds. After selecting the lip template sequence, the 20-dimensional feature vector was interpolated using a cubic spline interpolation algorithm, with the number of interpolation points tripled compared to the original sequence. A time weighting function, w(t) = sin(πt / 2), was introduced to adjust the transition speed. Finally, a Kalman filter was applied, with the measurement noise covariance R set to 0.01 and the process noise covariance Q set to 0.001, to smooth the generated lip sequence and eliminate high-frequency jitter, resulting in a coherent and natural lip animation.

[0033] Step S105: Based on the lip shape change template, a tongue position control model is constructed to obtain the tongue position parameters of different phonemes when the digital human is performing pronunciation. The changes in the shape and position of the digital human's tongue are controlled by the parameters to achieve synchronization between the tongue position animation and the phonemes during the performance, and obtain the mapping relationship between the tongue position and the phonemes during the performance.

[0034] NURBS surface modeling technology is used to construct a digital human tongue position control model, and a control point grid is set according to the lip shape change template. A bidirectional LSTM network is used to perform phoneme recognition on the broadcast voice to obtain the phoneme sequence and timestamp information. Tongue movement is simulated through a physical simulation method based on a mass-spring system. If the lip shape changes, the tongue position parameters are optimized according to the phoneme sequence. A deep deterministic policy gradient algorithm is used to construct an Actor-Critic network structure, and the tongue position control strategy is trained based on the naturalness of the tongue position animation and the phoneme synchronization. The tongue position parameters are interpolated using the cubic Hermite interpolation algorithm, and the tongue position parameter sequence is obtained by adding interpolation points.

[0035] For example, a digital human tongue model was constructed using NURBS surface modeling technology based on a lip shape template. A 30x20 control point grid was set up, and a spline curve was used to describe the tongue contour. The tongue position changes were parameterized using the control points to obtain a refined tongue position control model. A bidirectional LSTM network was used to perform phoneme recognition on the broadcast speech, extracting the phoneme sequence and its timestamp information. This information was then combined with tongue position descriptions from a phonetic knowledge base to establish an initial mapping between phonemes and tongue position parameters. A physical simulation method based on a mass-spring system was used to simulate tongue movement based on lip shape changes and phoneme sequences, optimizing tongue position parameters and achieving precise synchronization between tongue position animation and phoneme pronunciation. A deep deterministic policy gradient algorithm was used to construct an actor-critic network architecture. The tongue position control strategy was trained using the naturalness of tongue position animation and phoneme synchronization as reward functions, optimizing the mapping between tongue position and phonemes during broadcast. The tongue position parameters were interpolated using a cubic Hermite interpolation algorithm, adding 100 interpolation points to improve the continuity and naturalness of tongue movement, resulting in a final refined tongue position parameter sequence.

[0036] In constructing the digital human tongue position control model, NURBS surface modeling technology was first used to create a 3D tongue model. A 30x20 grid of control points with a surface order of 3 was set. The tongue shape was accurately described by adjusting the weights and positions of the 600 control points. Subsequently, phoneme recognition was performed using a four-layer bidirectional LSTM network, each layer containing 256 hidden units. The input was a 40-dimensional Mel-spectrogram feature, and the output was a phoneme probability distribution. The recognition accuracy reached 95%. Based on the recognition results, a tongue position parameter mapping table containing 44 phonemes was established, with each phoneme corresponding to 20 tongue position control parameters. Next, a mass-spring system was used to simulate tongue movement. The tongue was divided into 500 mass points, connected by 1500 springs with a spring constant of 100 N / m, a damping coefficient of 0.5 Ns / m, and a time step of 0.001 s. The tongue deformation was calculated through numerical integration. During the optimization phase, the DDPG algorithm was used to train the tongue position control strategy. The Actor network adopted a three-layer fully connected structure (256-128-64), and the Critic network had four layers (256-256-128-1). The learning rate was set to 0.0001, and the discount factor was 0.99. After 100,000 iterations, a stable control strategy was obtained. Finally, cubic Hermite interpolation was applied to the optimized tongue position parameter sequence, interpolating from the original 30 frames per second to 300 frames per second to ensure C1 continuity and generate smooth and natural tongue position animation. The entire process was run on a workstation equipped with an RTX 3080 GPU and 32GB of RAM. The tongue position parameter optimization of a single phoneme took approximately 0.5 seconds, achieving real-time performance.

[0037] In step S106, a tooth visibility control rule is established based on the mapping relationship between tongue position and phonemes during the broadcast. The tooth visibility of the digital human during the broadcast is dynamically adjusted according to the tooth visibility parameters of the phonetic sequence, so that the tooth visibility changes match the characteristics of the real person's pronunciation during the broadcast.

[0038] Obtain tongue position parameters and phoneme types, and use the random forest algorithm to construct tooth visibility control rules based on the tongue position parameters and phoneme types to obtain a tooth visibility prediction model; receive a broadcast voice signal, and perform phoneme recognition on the broadcast voice signal through a convolutional neural network-long short-term memory network hybrid model to extract the phoneme sequence and its duration; generate a tooth visibility parameter sequence corresponding to each phoneme based on the tooth visibility prediction model and the phoneme sequence; use subdivision surface technology to construct a three-dimensional tooth model of the digital human, and if a tooth visibility parameter sequence is received, control the transparency of the three-dimensional tooth model through a vertex shader and a pixel shader combined with a masking technology; perform cubic spline interpolation on the tooth visibility parameter sequence to obtain a smoothed tooth visibility parameter sequence; if the smoothed tooth visibility parameter sequence does not match the preset real person pronunciation characteristics, use a dynamic time warping algorithm to fine-tune the smoothed tooth visibility parameter sequence to obtain a tooth visibility parameter sequence that matches the preset real person pronunciation characteristics.

[0039] For example, based on the mapping relationship between tongue position and phonemes in broadcasting, a random forest algorithm is used to construct a tooth visibility control rule. The tongue position parameters and phoneme types are used as input features to train a tooth visibility prediction model. A convolutional neural network-long short-term memory network hybrid model is used to perform phoneme recognition on the broadcast voice, extract the phoneme sequence and its duration, and combine it with the tooth visibility prediction model to generate a tooth visibility parameter sequence corresponding to each phoneme. Subdivision surface technology is used to construct a refined three-dimensional tooth model of the digital human. The transparency of the tooth model is controlled by vertex shaders and pixel shaders combined with masking technology to achieve dynamic and fine adjustment of tooth visibility. The tooth visibility parameter sequence is smoothed using the cubic spline interpolation algorithm to eliminate mutations. The transparency of the tooth model is updated in real time through the OpenGL graphics API. The generated tooth visibility sequence is fine-tuned using a dynamic time warping algorithm to make the tooth visibility changes match the characteristics of real-person broadcast pronunciation. To implement digital human tooth visibility control, a prediction model was first constructed using a random forest algorithm. Input features included 20-dimensional tongue position parameters and 44 phoneme types. A forest of 500 decision trees was constructed, with each tree having a maximum depth of 10 and a minimum leaf node size of 5. The training sample size was 100,000, resulting in a tooth visibility prediction model with an accuracy of 95%. Subsequently, a hybrid 1D-CNN and bidirectional LSTM network was used for phoneme recognition. The CNN consisted of three convolutional layers with kernel sizes of 3, 5, and 7, respectively. The LSTM layer contained 256 hidden units. The input was 40-dimensional Mel-spectrogram features, and the output was a phoneme probability distribution. Training on the LibriSpeech dataset yielded a phoneme recognition accuracy of 97%. A high-precision tooth model was constructed using NURBS surface technology. The control points were 20x30x10 and the subdivision level was set to 4, resulting in a detailed tooth model with approximately 1 million vertices. In the OpenGL environment, tooth transparency control was implemented through vertex and pixel shaders, using alpha blending with the blending function set to (GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA). Cubic spline interpolation was applied to the generated tooth visibility parameter sequence, increasing the original frame rate from 30 frames per second to 300 frames per second to ensure C2 continuity. Finally, the generated tooth visibility sequence was fine-tuned using the FastDTW algorithm with a window size of 20 and a tolerance threshold of 0.1. The sequence was then compared with a reference sequence extracted from a live-action video. Five iterations of optimization were performed, resulting in an average Euclidean distance between the generated sequence and the live-action features of the sequence less than 0.05. The entire process was run on a workstation equipped with an RTX3090 GPU and 64GB of RAM, achieving real-time rendering performance of 60 frames per second.

[0040] Step S107: Track the shape and texture features of the digital human's lips, tongue, and teeth in real time during the broadcast, compare them with the target broadcast lip shape, tongue position, and teeth visibility parameters, dynamically adjust the digital human's facial model parameters, and perform coordinated control of the lips, tongue, and teeth during the broadcast.

[0041] A facial key point detection algorithm is used to obtain key point information of the digital human facial image, and the key point information includes position data of the lips, tongue and teeth areas. Based on the key point information obtained, a mapping relationship between two-dimensional image features and three-dimensional model parameters is established. If it is detected that there is a deviation between the target broadcast lip shape, tongue position and teeth visibility parameters and the actual feature parameters, the adjustment amount of the facial model control point is calculated. The adjustment amount is minimized through an iterative optimization algorithm to obtain the adjustment result of the digital human facial model. Based on the adjustment result, a mass spring system is introduced to connect the key control points of the lips, tongue and teeth. The mass spring system is used to achieve coordinated control of the lips, tongue and teeth. The spring force action data is obtained from the mass spring system, and the final adjustment parameters of the digital human facial model are determined based on the spring force action data.

[0042] For example, the Dlib library's 68-point facial landmark detection algorithm is used to extract key points from the digital human facial image. Combined with a 3D morphable model, these 2D landmarks are mapped to 3D space to locate the lip, tongue, and teeth regions. Principal component analysis is then used to extract the shape and texture feature parameters of these regions. Based on preset target lip shape, tongue position, and tooth visibility parameters, an active appearance model is used to establish a mapping relationship between 2D image features and 3D model parameters. The Euclidean distance between the extracted actual feature parameters and the target parameters is calculated to obtain feature deviation values. The Levenberg-Marquardt algorithm is used to calculate the adjustment amount for the facial model control points based on the feature deviation values. This adjustment amount is minimized through iterative optimization, achieving refined adjustments to the digital human facial model. An extended Kalman filter is used to smooth the facial model adjustment process, setting appropriate state transition matrices and observation matrices to eliminate jitter caused by feature extraction and parameter comparison. A mass-spring system is introduced as a physical constraint model to connect the key control points of the lips, tongue, and teeth. The spring force achieves coordinated control of the lips, tongue, and teeth, ensuring natural and consistent movements. To achieve refined control of the digital human face, the 68-point facial landmark detection algorithm from the Dlib library was first used to locate landmarks on a 1920x1080 resolution image, with an average detection time of 15ms. Subsequently, the BaselFaceModel, a 3D morphable model consisting of 199 shape parameters and 40 expression parameters, was used to map the 2D landmarks to 3D space with an error of less than 0.5mm. Principal component analysis was used to extract shape and texture features of the lips, tongue, and teeth, retaining the top 20 principal components that explained 95% of the variance. An active appearance model was used to map the 2D features to the 3D parameters, using a training sample of 100,000 images and a convergence threshold of 0.001. The Euclidean distance between the actual features and the target parameters was calculated to obtain a 68-dimensional feature deviation vector. Optimization was performed using the Levenberg-Marquardt algorithm, with an initial damping factor λ set to 0.01, a maximum number of iterations of 50, and a convergence criterion of less than 1e-6 relative error. An extended Kalman filter was used for smoothing, with a state vector dimension of 239 (199 + 40) and an observation vector dimension of 68. The process noise covariance Q and the observation noise covariance R were set to 0.01 times the unit matrix through debugging. A mass-spring system was introduced, with 20 mass points set on the lips, 10 on the tongue, and 8 on the teeth. The spring stiffness coefficient k = 100 N / m and the damping coefficient c = 0.5 Ns / m. The entire process ran on an RTX 3090 GPU, achieving real-time performance of 60 fps. The facial model control accuracy reached 0.1 mm, and the similarity score with the real-life performance reached 0.95, with a maximum score of 1.

[0043] The above is only a preferred embodiment of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and supplements without departing from the principles of the present invention. These improvements and supplements should also be regarded as the scope of protection of the present invention.

Claims

1. A desktop digital human interactive presentation method based on JavaFX, characterized in that: The method comprises: Scanning a real person's face with a depth camera to obtain 3D shape data of facial tissues, then segmenting and annotating the data based on medical anatomical knowledge to build a digital human face model, including the geometric shape and texture features of the facial tissue structures of the lips, tongue, and teeth; Obtain the lip shape changes of different phonemes in the digital human performance, establish amplitude, speed and trajectory templates, use JavaFX animation and timeline functions to obtain the lip shape parameter sequence corresponding to the sound sequence, and realize the synthesis of the performance lip shape animation synchronized with the phonemes by controlling the lip shape changes of the digital human facial model. Based on the obtained digital human lip shape animation, a phoneme-to-lip shape parameter mapping model is trained to obtain a digital human lip shape animation with high lip shape control accuracy and natural and coordinated changes. Through comparative learning, the similarity between the digital human lip shape animation and the lip shape of a real person is determined. If the similarity is lower than a preset threshold, the lip shape control parameters are fine-tuned to make the lip shape changes during the broadcast more similar to those of a real person, thereby improving the visual realism of the lip shape animation and obtaining the optimized broadcast lip shape animation. Based on the optimized digital human lip animation, the lip shape feature vectors of the key frames are extracted to obtain lip shape change templates representing different phonemes in the pronunciation state. The templates are used to guide the generation of lip shape animation to ensure the coherence of the digital human's lip shape changes when performing different phonemes. Based on the lip shape change template, a tongue position control model is constructed to obtain the tongue position parameters of different phonemes when the digital human is performing pronunciation. By controlling the changes in the shape and position of the digital human's tongue through the parameters, the tongue position animation is synchronized with the phonemes during the performance, and the mapping relationship between the tongue position and the phonemes during the performance is obtained. Based on the mapping relationship between tongue position and phonemes during broadcasting, a tooth visibility control rule is established. According to the tooth visibility parameters of the sound sequence, the visibility of the digital human's teeth is dynamically adjusted during broadcasting, so that the changes in tooth visibility match the characteristics of the real person's pronunciation. Track the shape and texture features of the digital human's lips, tongue, and teeth in real time during the broadcast, compare them with the target broadcast's lip shape, tongue position, and teeth visibility parameters, dynamically adjust the digital human's facial model parameters, and perform coordinated control of the lips, tongue, and teeth during the broadcast; Based on the mapping relationship between tongue position and phonemes during broadcasting, a tooth visibility control rule is established. According to the tooth visibility parameters of the phonetic sequence, the visibility of the digital human's teeth is dynamically adjusted during broadcasting, so that the changes in tooth visibility match the characteristics of the real person's pronunciation during broadcasting, including: Obtaining tongue position parameters and phoneme types, and constructing tooth visibility control rules using a random forest algorithm based on the tongue position parameters and phoneme types to obtain a tooth visibility prediction model; Receiving a speech signal, performing phoneme recognition on the speech signal using a convolutional neural network-long short-term memory network hybrid model, and extracting a phoneme sequence and its duration; Generate a tooth visibility parameter sequence corresponding to each phoneme according to the tooth visibility prediction model and the phoneme sequence; The 3D tooth model of the digital human is constructed using subdivision surface technology. If a tooth visibility parameter sequence is received, the transparency of the 3D tooth model is controlled by combining vertex shaders and pixel shaders with masking technology. Performing cubic spline interpolation processing on the tooth visibility parameter sequence to obtain a smoothed tooth visibility parameter sequence; If the smoothed tooth visibility parameter sequence does not match the preset real-person pronunciation characteristics, the dynamic time warping algorithm is used to fine-tune the smoothed tooth visibility parameter sequence to obtain a tooth visibility parameter sequence that matches the preset real-person pronunciation characteristics.

2. The JavaFX-based desktop digital human interactive presentation method according to claim 1, characterized in that: Scanning a real person's face with a depth camera to obtain 3D shape data of facial tissue, segmenting and annotating the data in combination with medical anatomical knowledge to create a digital human face model, including the geometric shape and texture features of the facial tissue structure of the lips, tongue, and teeth. This includes: obtaining point cloud information containing multi-angle 3D scanning data of a real person's face, which is obtained by collecting facial tissue using a non-contact 3D laser scanner; Determine the boundary areas of the lips, tongue, and dental facial tissues based on the point cloud information and a pre-established medical anatomical knowledge base; An octree voxelization method is used to convert the point cloud information into a voxel grid structure to obtain voxelized data of facial tissue; For the voxelized data, a region growing algorithm is used to segment the lip, tongue and teeth regions. If the lip region is segmented, a loop subdivision algorithm is applied to the lip region. If the tongue region is obtained by segmentation, deformation field simulation is used for the tongue region; If the tooth region is obtained by segmentation, the March cube algorithm is applied to the tooth region; Receiving facial tissue data including texture image information and feature descriptors, wherein the feature descriptors are constructed by calculating curvature and normal vector geometric features of the facial tissue and using a point feature histogram; Based on the facial tissue data, the PointNet++ network is used to perform fine segmentation and annotation of facial tissue, and a digital human facial model including lips, tongue and teeth facial tissue is obtained.

3. The JavaFX-based desktop digital human interactive presentation method according to claim 1, characterized in that: The lip shape changes of different phonemes in the digital human performance are obtained, and the amplitude, speed and trajectory templates are established. The lip shape parameter sequence corresponding to the sound sequence is obtained by using the animation and timeline functions of JavaFX. By controlling the lip shape changes of the digital human facial model, the performance lip shape animation synchronization with the phonemes is achieved. The performance lip shape animation includes: A high-speed camera is used to capture the lip shape change sequence corresponding to different phonemes during the pronunciation of a real person, wherein the lip shape change sequence includes image data at 240 frames per second; Performing edge detection according to the lip shape change sequence to obtain a lip contour, and extracting feature points from the lip contour to obtain key point coordinates; Fitting the key point coordinates using the least squares method to obtain a lip shape change curve, and calculating the amplitude, speed, and acceleration parameters of the lip shape change according to the lip shape change curve; If the lip shape change parameters are obtained, a phoneme lip shape change template library is constructed; The phoneme lip shape change template library includes the amplitude, speed and trajectory templates corresponding to each phoneme; Reading a lip shape parameter sequence from the phoneme lip shape change template library, wherein the lip shape parameter sequence includes key point coordinates, change amplitude and speed information; The lip shape parameter sequence is mapped to the vertex coordinates of the digital human face model by using a radial basis function interpolation method, and the change of the vertex coordinates is controlled by an animation function to obtain a lip shape animation synchronized with the phonemes.

4. The JavaFX-based desktop digital human interactive presentation method according to claim 1, characterized in that: Based on the obtained digital human lip animation, a phoneme-to-lip parameter mapping model is trained to obtain a digital human lip animation with high lip control accuracy and natural and coordinated changes. Through comparative learning, the similarity between the digital human lip animation and the lip shape of a real person is determined. If the similarity is lower than a preset threshold, the lip shape control parameters are fine-tuned to make the lip shape changes during the broadcast more similar to that of a real person, improving the visual realism of the lip animation. The optimized broadcast lip animation is obtained, including: Obtain a phoneme sequence, use a temporal convolutional network to build a phoneme-to-lip shape parameter mapping model, and obtain optimized model parameters through a back-propagation algorithm; Receive a real-person pronunciation video, use a 3D deformable model to reconstruct the 3D facial model in the video, extract key features such as lip contour, opening and closing degree, and lip corner position, and establish a real-person pronunciation lip feature database; Based on the lip movement sequences of the digital human and the real person, the dynamic time warping algorithm is used to determine the time alignment relationship, and the structural similarity index is used to determine the visual similarity of each frame. Calculate the similarity score between the digital human's lip animation and the real person's lip shape features. If the similarity score is lower than a preset threshold, enter the parameter fine-tuning process. A genetic algorithm is used to fine-tune the lip shape control parameters, and the genetic algorithm includes chromosome encoding of the lip shape control parameters; Generate new parameter combinations through roulette wheel selection, single-point crossover and Gaussian mutation operations; The similarity score is used as the fitness function for iterative optimization; When the similarity score exceeds the preset threshold or reaches the maximum number of iterations, the optimized lip-sync animation is output.

5. The JavaFX-based desktop digital human interactive presentation method according to claim 1, characterized in that: Based on the optimized digital human lip animation, the lip shape feature vectors of the key frames are extracted to obtain lip shape change templates representing different phonemes in the pronunciation state. The templates are used to guide the generation of lip shape animation to ensure the coherence of the digital human's lip shape changes when performing different phonemes, including: Obtain a video sequence of lip-sync animation, extract key frames using a motion amplitude calculation method based on the optical flow method, and set the extraction threshold at 1.5 times the average motion amplitude; Hierarchical clustering algorithm is used to process lip shape feature vectors, Ward's minimum variance method is used as the linking criterion, and lip shape change templates representing different phonemes are obtained by calculating the cluster center. Establish a mapping relationship between phonemes and lip shape change templates, and use a dynamic programming algorithm to search for the optimal path based on the continuous phoneme sequence. The state is defined as the lip shape template corresponding to the current phoneme. For the selected lip shape change template sequence, a multi-dimensional cubic spline interpolation algorithm is used to generate a smooth lip shape transition curve, interpolate each dimension of the lip shape feature, and map the interpolation result to the digital human face model; The generated lip sequence is processed by Kalman filter to eliminate the mutation and jitter in the lip sequence and obtain a coherent and natural lip animation.

6. The JavaFX-based desktop digital human interactive presentation method according to claim 1, characterized in that: Based on the lip shape change template, a tongue position control model is constructed to obtain the tongue position parameters of different phonemes when the digital human is performing pronunciation. The changes in the shape and position of the digital human's tongue are controlled by the parameters to achieve synchronization between the tongue position animation and the phonemes during the performance. The mapping relationship between the tongue position and the phonemes during the performance is obtained, including: NURBS surface modeling technology is used to build a digital human tongue position control model, and the control point grid is set according to the lip shape change template; Use a bidirectional LSTM network to perform phoneme recognition on the broadcast voice and obtain phoneme sequence and timestamp information; The tongue movement is simulated by a physical simulation method based on a mass-spring system. If the lip shape changes, the tongue position parameters are optimized according to the phoneme sequence. A deep deterministic policy gradient algorithm is used to build an Actor-Critic network structure, and the tongue position control strategy is trained based on the naturalness of tongue position animation and phoneme synchronization. The tongue position parameters are interpolated using the cubic Hermite interpolation algorithm, and the tongue position parameter sequence is obtained by adding interpolation points.

7. The JavaFX-based desktop digital human interactive presentation method according to claim 1, characterized in that: Track the shape and texture features of the digital human's lips, tongue, and teeth in real time during the broadcast, compare them with the target broadcast's lip shape, tongue position, and teeth visibility parameters, dynamically adjust the digital human's facial model parameters, and perform coordinated control of the lips, tongue, and teeth during the broadcast, including: Using a facial key point detection algorithm to obtain key point information of a digital human face image, the key point information includes position data of the lips, tongue and teeth areas; Based on the acquired key point information, a mapping relationship between two-dimensional image features and three-dimensional model parameters is established; If it is detected that the target broadcast lip shape, tongue position and teeth visibility parameters deviate from the actual feature parameters, the adjustment amount of the facial model control points is calculated; The adjustment amount is minimized through iterative optimization algorithm to obtain the adjustment result of the digital human face model; Based on the adjustment results, a mass spring system is introduced to connect the key control points of the lips, tongue and teeth. The mass spring system is used to achieve coordinated control of the lips, tongue and teeth. Spring force action data is obtained from the mass-spring system, and final adjustment parameters of the digital human face model are determined according to the spring force action data.

Images