Adaptive rendering method and system for characters and images in AI digital human virtual and real scene fusion

Abstract

The invention discloses an adaptive rendering method and system for characters and images in AI digital human virtual and real scene fusion, and the method comprises the following steps: S1, collecting Chinese text and image data, and extracting semantic and regional features; s2, constructing a Shenchang differential equation converter model, and inputting text and image features to generate fusion representation; s3, extracting an attention weight matrix, and determining the spatial arrangement position of the Chinese text in the image; s4, optimizing the structural parameter vector by adopting a luminous insect colony optimization algorithm to obtain a fusion model; s5, generating an image-text rendering instruction set based on the preliminary arrangement and optimization model; s6, in combination with the digital human action state and background image information, attitude mapping and illumination adjustment are completed to generate a rendering frame; and S7, inputting the rendering frame into a rendering engine, and outputting the video content fused by the Chinese characters and the images. According to the method, accurate fusion of Chinese semantics and images is realized, and the naturalness and interactivity of AI digital human image-text presentation are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of virtual human rendering technology, and in particular to an adaptive rendering method and system for AI digital human virtual and real scene fusion Chinese characters and images. Background Art

[0002] In the context of the rapid development of artificial intelligence and digital content generation, AI digital humans, as a combination of computer vision, speech generation, natural language processing and other technologies, have been widely used in virtual live broadcasts, educational broadcasts, cultural communication and other scenarios. As application needs continue to expand, AI digital humans not only need to have the ability to generate speech and movements, but also need to be able to achieve deep integration of graphic content with real or virtual scenes. Especially in the Chinese context, how to achieve accurate and natural integration of Chinese characters and background images has become one of the key challenges.

[0003] Most existing image-text fusion methods rely on static templates or rule-driven methods to arrange text in images. They lack an in-depth understanding of text semantics and are difficult to adapt to Chinese expression structures in complex contexts. At the same time, some systems use simple matching strategies based on image segmentation to locate text in image areas, but fail to combine the dynamic correspondence between text semantics and image content, resulting in a lack of semantic consistency and visual coordination in the fusion effect. In addition, in the image-text fusion process, existing methods often ignore the impact of AI digital humans in different actions, postures, and different scene lighting conditions on the image-text fusion display results, and are unable to achieve dynamic, coherent, and high-quality rendering output.

[0004] In terms of model construction, traditional image and text processing methods mainly rely on convolutional neural networks or Transformer structures, which have natural limitations in processing the temporal continuity and multi-scale dynamic features of Chinese semantics and image information, especially in dynamically changing scenes, which can easily cause context loss or semantic fusion ambiguity. In terms of optimization, most current model structures rely on fixed parameter settings, lacking adaptive adjustment mechanisms for structural elements such as model depth, attention mechanisms, and time control, and are unable to dynamically optimize fusion effects based on different data distributions.

[0005] In response to the above problems, there is an urgent need for a text-and-image fusion rendering method and system that can achieve accurate alignment of Chinese semantic information with image area content, combine attention mechanisms with continuous time modeling capabilities, have adjustable structural optimization capabilities, and can adapt to the dynamic behavior of digital humans and scene lighting changes. The system should support the joint modeling of Chinese text and images, have semantic perception and visual linkage capabilities, and be able to automatically determine the text layout position, generate text-and-image rendering instructions, and synchronize posture and lighting adjustments, and present the fusion effect in the form of high-quality rendering frame output, thereby improving the intelligent text-and-image expression capabilities and natural interaction of AI digital humans in multiple scenarios.

[0006] Therefore, how to provide an adaptive rendering method and system for text and images in the virtual-real scene fusion of AI digital humans is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0007] An object of the present invention is to propose a method for fusing and rendering text and images in an AI digital human based on a neural ordinary differential equation transformer and a glowworm swarm optimization algorithm. The present invention makes full use of natural language processing, multi-modal feature modeling and continuous-time neural network technologies, and details the joint modeling, layout optimization and dynamic rendering processes of Chinese text semantics and image region features, and has the advantages of accurate semantic alignment, natural visual fusion and strong ability to adapt to dynamic scenes.

[0008] According to the embodiment of the present invention, the adaptive rendering method for text and images in the virtual-real scene fusion of AI digital humans includes the following steps: S1. Collect Chinese text and corresponding image data, perform word segmentation and semantic encoding on the Chinese text to generate a text feature vector; perform region segmentation and texture extraction on the image data to generate an image region feature vector; S2. Construct a neural ordinary differential equation transformer model, including a multi-head attention module and a variable time-step ODE calculation module. The structural parameter vector of the neural ordinary differential equation transformer model includes network depth, number of attention heads and time-step control parameters. Input the text feature vector and the image region feature vector into the neural ordinary differential equation transformer model to generate a fused feature representation; S3. Extract the attention weight matrix generated by the multi-head attention module from the fused feature representation, and determine the spatial arrangement position of the Chinese text in the image according to the attention weight matrix to form a preliminary text-image fusion layout; S4. Adopt the glowworm swarm optimization algorithm to jointly optimize the network depth, number of attention heads and time-step control parameters to obtain a fused model with optimized structure; S5. Based on the preliminary text-image fusion layout, generate a text-image rendering instruction set based on the fused model with optimized structure; S6. Combine the current action state of the digital human and the background image information, perform pose mapping and lighting adjustment on the text-image rendering instruction set to generate a virtual-real fusion rendering frame; S7. Input the virtual-real fusion rendering frame into the digital human rendering engine to complete the output of the video content of the fusion of Chinese text and images.

[0009] Optionally, the S1 specifically includes: S11. Collect Chinese text and corresponding image data. The Chinese text is short text content with a specific expression intention, and the image data is an image containing a scene or object related to the text semantics; S12. Perform Chinese language word segmentation on each piece of Chinese text. Adopt a word segmentation method that combines rules and data-driven approaches to divide the original text into a sequence of words with word boundaries. S13. On the basis of completing language word segmentation, encode the word sequence, extract semantic features including context-dependent relationships, and form a text feature vector with a fixed dimension. S14. Perform region segmentation processing on each piece of image data. Adopt a region division algorithm based on texture and boundary features to divide the image data into several image regions with consistent texture patterns. S15. Perform texture feature operations on the divided image data regions, extract the color distribution, edge direction, and local structure features of each region in the image, and form an image region feature vector.

[0010] Optionally, the specific steps of S2 include: S21. Use the text feature vector as the language input sequence and the image region feature vector as the visual input sequence, and perform unified encoding operations on the two types of input sequences. S22. Input the unified encoded input sequence into the multi-head attention module. The multi-head attention module contains several parallel attention sub-heads, and each attention sub-head calculates the correlation between the text and the image to generate several attention representation matrices. S23. Perform splicing and linear mapping operations on each attention representation matrix to construct a fused attention output sequence as the output of the multi-head attention module. S24. Use the fused attention output sequence as the input feature and input it into the variable time-step ODE calculation module. Model the change of the input feature state over time based on the continuous-time dynamic function. The continuous-time dynamic function consists of the following structure: The first linear transformation unit jointly transforms the input feature and the time variable and then outputs an intermediate representation. Between the first linear transformation unit and the second linear transformation unit, a time-aware gating control unit is set to generate an input gate and a reset gate based on the current time variable and the state of the input feature, and dynamically adjust the channel information of the intermediate representation. The second linear transformation unit maps the gated feature representation and then inputs it into the activation function unit. The activation function unit adopts a time-adjustable activation mechanism, and the adjustable coefficient generated according to the time variable controls the response intensity of the non-linear output. The variable time-step ODE calculation module uses the time-step control parameter as the integration granularity control input, integrates the continuous-time dynamic function at the time step, and outputs a multi-layer dynamic representation evolving over time. S25. The overall depth of the neural ordinary differential equation transformer model is determined by the continuous stacking layers of the variable time-step ODE calculation modules included. The stacking layers, as the network depth parameter, control the step size of each integration layer in the variable time-step ODE calculation module, and balance the feature calculation accuracy and calculation efficiency through the time-step control parameter. S26. Use the multi-layer dynamic representations output by the variable time-step ODE calculation module as the fused feature representations.

[0011] Optionally, the specific steps of S3 are as follows: S31. Extract the attention weight matrix generated by the multi-head attention module from the fused feature representations. The attention weight matrix characterizes the matching strength between each word in the text feature vector and each region in the image region feature vector. S32. Determine the image region corresponding to each Chinese text according to the maximum weight value corresponding to each text in the attention weight matrix, and establish a one-to-one mapping relationship between the Chinese text and the image region. S33. According to the one-to-one mapping relationship, extract the spatial coordinate range of each corresponding image region, determine the arrangement position of the Chinese text on the image, and generate a preliminary graphic-text fusion layout including the display positions of all Chinese texts. S34. Perform conflict detection and position adjustment processing on the preliminary graphic-text fusion layout, and set a minimum spacing threshold between any two adjacent Chinese words to meet the readability of text arrangement and non-overlap of regions.

[0012] Optionally, the specific steps of S4 are as follows: S41. Initialize the glowworm swarm, and each glowworm individual corresponds to a set of structural parameter vectors. The structural parameter vectors are composed of three parts: the network depth, the number of attention heads, and the time-step control parameter in the neural ordinary differential equation transformer model. S42. Set a brightness function for each glowworm individual. The brightness function takes the comprehensive performance index of the neural ordinary differential equation transformer model on the validation set as the objective function value. The comprehensive performance index is composed of three weighted combinations: the graphic-text arrangement accuracy, the sparsity of attention distribution, and the stability of fused features. The sum of the weighting coefficients of each index is one. S43. In each generation of iteration, according to the relative difference between the individual brightness value and the brightness values of other individuals in the neighborhood, and based on the difference between the current individual's structural parameter vector and the structural parameter vector of the brighter individual in the neighborhood, construct a movement function through the brightness difference weight and the spatial distance factor, and update the network depth, the number of attention heads, and the time-step control parameter of the current individual. S44. Set a step size decay mechanism and parameter boundary control conditions during the movement process to prevent the structural parameter vector from exceeding the boundary and maintain the adaptability and convergence of the search area. S45. When the preset maximum number of iterations or the convergence condition of the brightness function is reached, select the glowworm individual with the optimal brightness value, and the corresponding structure parameter vector is the optimal combination of the network depth, the number of attention heads, and the time step control parameter; S46. Based on the optimal combination, reconstruct the neural ordinary differential equation transformer model as the fusion model after structure optimization.

[0013] Optionally, the S5 graphic and text rendering instruction set includes: determining the font, color, transparency, and display coordinates in the image of each Chinese text based on the fusion model after structure optimization, constructing graphic and text rendering instructions including text style parameters and spatial position parameters, and combining them to generate a graphic and text rendering instruction set.

[0014] Optionally, S6 specifically includes: obtaining the skeletal pose parameters of the digital human and the illumination distribution characteristics of the background image in the current frame, and performing coordinate transformation and brightness adjustment based on the pose angle and illumination direction according to the graphic and text rendering instruction set, and outputting the processed image frame as the virtual-real fusion rendering frame.

[0015] The adaptive rendering system for text and image in the virtual-real scene of the AI digital human according to the embodiment of the present invention includes the following modules: A text feature extraction module, configured to perform word segmentation and semantic encoding on Chinese text to generate a text feature vector; An image feature extraction module, configured to perform region segmentation and texture extraction on image data to generate an image region feature vector; A fusion modeling module, configured to input the text feature vector and the image region feature vector into a neural ordinary differential equation transformer model to generate a fusion feature representation; An attention arrangement module, configured to extract an attention weight matrix from the fusion feature representation and generate a preliminary graphic and text fusion layout; A parameter optimization module, configured to optimize the network depth, the number of attention heads, and the time step control parameter based on the glowworm swarm optimization algorithm; An instruction generation module, configured to generate a graphic and text rendering instruction set based on the preliminary graphic and text fusion layout; A pose illumination mapping module, configured to generate a virtual-real fusion rendering frame by combining the action state of the digital human and the background image information; A graphic and text rendering engine module, configured to output the virtual-real fusion rendering frame as video content of the fusion of Chinese text and image.

[0016] The beneficial effects of the present invention are: First, the present invention constructs a neural ordinary differential equation transformer model, effectively integrating Chinese semantic features and image region features, solving the problems of inaccurate text-image semantic alignment and layout dependence on templates in existing methods, achieving dynamic inference of the position of Chinese text in the image and spatially arranged semantic consistency, and improving the naturalness and intelligence level of text-image fusion.

[0017] Secondly, the present invention introduces a glowworm swarm optimization algorithm to jointly optimize the network depth, number of attention heads, and time step control parameters of the neural ordinary differential equation transformer model, significantly enhancing the adaptive ability of the model structure to the complexity of different text-image samples, improving the comprehensive performance of the fusion model in terms of layout accuracy, attention sparsity, and feature stability, and effectively overcoming the defects of fixed structure and poor generalization ability in existing methods.

[0018] In addition, the present invention combines the dynamic behavior of an AI digital human and scene background information to perform pose mapping and lighting adjustment on the text-image rendering instruction set, generating a dynamically rendered frame of virtual-real fusion, achieving high coordination among Chinese content, image background, and digital human behavior, adapting to various application scenarios such as virtual live broadcast and intelligent interaction, and enhancing the immersion and interactivity of the text-image expression of the AI digital human. Description of the Drawings

[0019] The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation to the present invention. In the drawings: Figure 1 is the overall flowchart of the adaptive rendering method for text and image in the virtual-real scene fusion of an AI digital human proposed by the present invention; Figure 2 is the flowchart of the structural parameter vector optimization based on the glowworm swarm optimization algorithm for the adaptive rendering method for text and image in the virtual-real scene fusion of an AI digital human proposed by the present invention. Detailed Embodiment

[0020] Now, the present invention will be further described in detail with reference to the drawings. These drawings are all simplified schematic diagrams, only illustrating the basic structure of the present invention in a schematic manner, so they only show the components related to the present invention.

[0021] Refer to Figure 1 and Figure 2 , the adaptive rendering method for text and image in the virtual-real scene fusion of an AI digital human includes the following steps: S1. Collect Chinese text and corresponding image data, perform word segmentation and semantic encoding on the Chinese text to generate a text feature vector; perform region segmentation and texture extraction on the image data to generate an image region feature vector; S2. Construct a neural ordinary differential equation transformer model, including a multi-head attention module and a variable time-step ODE calculation module. The structural parameter vector of the neural ordinary differential equation transformer model includes the network depth, the number of attention heads, and the time-step control parameter. Input the text feature vector and the image region feature vector into the neural ordinary differential equation transformer model to generate a fused feature representation; S3. Extract the attention weight matrix generated by the multi-head attention module from the fused feature representation, and determine the spatial arrangement position of the Chinese text in the image according to the attention weight matrix to form a preliminary text-image fusion layout; S4. Use the glowworm swarm optimization algorithm to jointly optimize the network depth, the number of attention heads, and the time-step control parameter to obtain a fused model with optimized structure; S5. Based on the preliminary text-image fusion layout, generate a text-image rendering instruction set based on the fused model with optimized structure; S6. Combine the current action state of the digital human and the background image information to perform pose mapping and lighting adjustment on the text-image rendering instruction set to generate a virtual-real fusion rendering frame; S7. Input the virtual-real fusion rendering frame into the digital human rendering engine to complete the output of the video content of the fusion of Chinese text and image. Specifically: Use the virtual-real fusion rendering frame after pose mapping and lighting adjustment as the input and inject it into the digital human rendering engine. The rendering engine generates a video sequence containing the fusion effect of Chinese text and image frame by frame based on the frame content, realizing the synchronous presentation of text content and image visual information in the digital human broadcast scenario, and finally outputting the fusion video content with semantic consistency and visual coherence.

[0022] The present invention constructs a method for rendering the fusion of Chinese characters and images for AI digital humans by introducing a neural ordinary differential equation transformer model and a glowworm swarm optimization algorithm, which can achieve the precise fusion of text and image regions in a dynamic semantic scenario, overcome the problems of inaccurate semantic alignment and static and rigid rendering in the existing text-image arrangement methods, greatly improve the naturalness, matching degree, and rendering coherence of the fused content, and adapt to various virtual interaction scenarios.

[0023] In this embodiment, the specific content of S1 includes: S11. Collect Chinese text and corresponding image data. The Chinese text is short text content with specific expression intentions, and the image data is an image containing scenes or objects related to the text semantics; S12. Perform Chinese language word segmentation processing on each piece of Chinese text. Use a lexical analyzer to divide the continuous text into meaningful word units, and adopt a word segmentation method combining rules and data-driven to divide the original text into a word sequence with word boundaries; S13. On the basis of completing word segmentation of the language, encode the word sequence. For the segmented word sequence, use a pre-trained word vector model or a context embedding model to convert each word into a corresponding numerical vector representation, preserve semantic information and context dependencies, extract semantic features containing context dependencies, and form a text feature vector of a fixed dimension; S14. Perform region segmentation processing on each piece of image data. Adopt a region division algorithm based on texture and boundary features to divide the image data into several image regions with consistent texture patterns; S15. Perform texture feature operations on the divided image regions. Use a texture encoding network or a statistical feature algorithm to extract the texture pattern, edge structure, and local details of each region, generate features reflecting the visual characteristics of the region, extract the color distribution, edge direction, and local structure features of each region in the image, and form an image region feature vector.

[0024] The present invention refines and standardizes the text word segmentation, semantic encoding, and image region feature extraction processes for the preprocessing of Chinese text and image data, ensures the unity of the input data dimension and semantic feature structure, provides a stable and compatible feature basis for the subsequent model input, and solves the problem of difficult model fusion caused by mismatched feature dimensions in existing methods.

[0025] In this embodiment, the specific steps of S2 are as follows: S21. Use the text feature vector as the language input sequence and the image region feature vector as the visual input sequence. Perform unified encoding operations on the two types of input sequences. Map the two types of inputs to a shared representation space using the same-dimensional embedding transformation method to maintain the consistency of semantic and visual information; S22. Input the uniformly encoded input sequence into the multi-head attention module. The multi-head attention module includes several parallel attention sub-heads. Each attention sub-head calculates the relevance between the text and the image. Specifically, based on the attention mechanism, perform weighted evaluation on the matching strength between each text position and the image region, capture the semantic dependency relationship and corresponding degree between the two, and generate several attention representation matrices; S23. Perform splicing and linear mapping operations on each attention representation matrix. Specifically, concatenate the attention matrices output by each attention sub-head along the feature dimension to form a unified multi-head attention representation matrix, and perform a linear transformation through a fully connected layer to integrate the attention information of each sub-head and improve the expression ability of the representation, and construct a fused attention output sequence as the output of the multi-head attention module; S24. Take the fused attention output sequence as the input feature and input it into the variable time-step ODE calculation module. Model the change of the input feature state over time based on the continuous-time dynamic function, and the continuous-time dynamic function consists of the following structure: The first linear transformation unit jointly transforms the input feature and the time variable and then outputs an intermediate representation. Between the first linear transformation unit and the second linear transformation unit, a time-aware gating control unit is set to generate an input gate and a reset gate based on the current time variable and the state of the input feature, and dynamically adjust the channel information of the intermediate representation. The second linear transformation unit maps the gated feature representation and then inputs it into the activation function unit. The activation function unit adopts a time-adjusted activation mechanism, and the adjustable coefficient generated according to the time variable controls the response intensity of the non-linear output. The variable time-step ODE calculation module uses the time-step control parameter as the integration granularity control input, integrates the continuous-time dynamic function at the time step, and outputs the multi-layer dynamic representation evolving over time. S25. The overall depth of the neural ordinary differential equation transformer model is determined by the number of consecutive stacking layers of the variable time-step ODE calculation module included. The stacking layer number is used as the network depth parameter to control the step size of each integration layer in the variable time-step ODE calculation module, and balance the feature calculation accuracy and calculation efficiency through the time-step control parameter. S26. Take the multi-layer dynamic representation output by the variable time-step ODE calculation module as the fused feature representation.

[0026] Through the construction of the neural ordinary differential equation transformer model including the multi-head attention module and the variable time-step ODE calculation module, the present invention realizes the deep fusion of cross-modal Chinese text and image features in the continuous time domain, and further enhances the modeling ability of non-linear semantic associations through the improvement of the continuous-time dynamic function in the structure, significantly improving the expression accuracy and time consistency of the multi-modal fusion model.

[0027] In this embodiment, the specific content of S3 includes: S31. Extract the attention weight matrix generated by the multi-head attention module from the fused feature representation. The attention weight matrix characterizes the matching strength between each word in the text feature vector and each region in the image region feature vector. S32. Determine the image region corresponding to each Chinese text according to the maximum weight value corresponding to each text in the attention weight matrix, and establish a one-to-one mapping relationship between the Chinese text and the image region. S33. According to the one-to-one mapping relationship, extract the spatial coordinate range of each corresponding image region, and determine the arrangement position of the Chinese text on the image. Specifically: according to the one-to-one mapping relationship between the Chinese text and the image region constructed by the attention weight matrix generated by the multi-head attention module, extract the spatial coordinate range of each concerned image region, including the normalized position coordinates of its upper left corner and lower right corner in the image, and then match the position of the Chinese text according to the geometric center or high-attention density point of the region, determine the arrangement position of each text segment on the image, and generate a preliminary graphic-text fusion layout including the display positions of all Chinese texts. S34. Perform conflict detection and position adjustment processing on the preliminary graphic-text fusion layout, and set a minimum spacing threshold between any two adjacent Chinese words to meet the readability of text arrangement and the non-overlap of regions.

[0028] The present invention extracts attention weight information from the multi-head attention representation matrix and constructs a spatial arrangement strategy of text in the image, realizes the dynamic positioning matching between Chinese semantics and image regions, effectively solves the problems that traditional arrangement methods cannot understand language context and have serious graphic-text misalignment, provides a structured spatial basis for subsequent rendering instruction generation, and improves the intelligent level of graphic-text arrangement.

[0029] In this embodiment, the specific content of S4 includes: S41. Initialize the glowworm swarm, and each glowworm individual corresponds to a set of structure parameter vectors, which are composed of three parts: the network depth, the number of attention heads, and the time step control parameter in the neural ordinary differential equation transformer model. S42. Set a brightness function for each glowworm individual, and the brightness function takes the comprehensive performance index of the neural ordinary differential equation transformer model on the validation set as the objective function value. The comprehensive performance index is composed of three weighted combinations: graphic-text arrangement accuracy, attention distribution sparsity, and fusion feature stability, and the sum of the weighting coefficients of each index is one. S43. In each generation of iteration, according to the relative difference between the individual brightness value and the brightness values of other individuals in the neighborhood, based on the difference between the current individual's structure parameter vector and the structure parameter vector of the individual with higher brightness in the neighborhood, construct a movement function through the brightness difference weight and the spatial distance factor, and update the network depth, the number of attention heads, and the time step control parameter of the current individual. S44. Set a step size decay mechanism and parameter boundary control conditions during the movement. Specifically: Introduce a step size decay mechanism and parameter boundary control conditions during the movement of glowworm individuals. The step size decay mechanism dynamically reduces the movement distance of individuals according to the number of iterations, realizing the transition from coarse-grained search to fine-tuning and improving the convergence stability. The parameter boundary control conditions are used to constrain that the parameter values of individuals after each update must be kept within the legal interval of the preset structural parameter vector, preventing the model structure from having invalid or out-of-design-range abnormal configurations, ensuring the effectiveness and controllability of the optimization process, preventing the structural parameter vector from exceeding the boundary, and maintaining the adaptability and convergence of the search area. S45. When the preset maximum number of iterations or the convergence condition of the brightness function is reached, select the glowworm individual with the optimal brightness value, and the corresponding structural parameter vector is the optimal combination of the network depth, the number of attention heads, and the time step control parameter. S46. Based on the optimal combination, reconstruct the neural ordinary differential equation transformer model as the fusion model after structure optimization.

[0030] The present invention uses the glowworm swarm optimization algorithm to dynamically update the structural parameter vector of the neural ordinary differential equation transformer model. Through the brightness function to guide the structure evolution and combined with the information transfer mechanism between adjacent individuals, it realizes the adaptive adjustment of the fusion model structure in the search space, effectively overcomes the problems of strong rigidity and weak generalization ability of the traditional model structure, and improves the global optimal performance ability of the fusion model.

[0031] In this embodiment, the S5 graphic and text rendering instruction set includes: determining the font, color, transparency, and display coordinates in the image of each Chinese text based on the fusion model after structure optimization. On the basis of the fusion model after structure optimization, combined with the attention weight distribution and the semantic style information carried in the fusion feature representation, determine the visual attributes and spatial attributes of each Chinese text respectively: the font is mapped by the semantic emphasis intensity, the color and transparency are set according to the text emotional tendency and the background image contrast, and the display coordinates are calculated from the attention center position of its corresponding image area. Construct graphic and text rendering instructions containing text style parameters and spatial position parameters, and combine them to generate a graphic and text rendering instruction set. The graphic and text rendering instructions are generated by the fusion model and are used to drive the final graphic and text presentation, including the text style parameters of each Chinese text, such as font type, font size, color, transparency, and spatial position parameters, including the display coordinates and alignment method on the image. This instruction set is used to guide the rendering engine to accurately overlay the text content on the image area in a way that adapts to the visual context, realizing the natural fusion display of text and image.

[0032] In this embodiment, S6 specifically includes: obtaining the skeletal pose parameters of the digital human and the illumination distribution characteristics of the background image in the current frame, and performing coordinate transformation and brightness adjustment based on the pose angle and illumination direction according to the graphic rendering instruction set. The coordinate transformation is based on the current three-dimensional pose angle information of the digital human, and performs affine transformation or perspective transformation on the text display coordinates defined in the graphic rendering instruction, so that the text position is visually adjusted synchronously with the digital human's actions, ensuring that it fits the dynamic perspective. The brightness adjustment calculates the light and dark degree that the text may be affected by light in the target area according to the main light source direction and intensity of the current background image, and adjusts the text color or transparency up or down, so that it remains clearly visible in the rendered frame without being overly obtrusive, improving the overall visual fusion naturalness, and outputting the processed image frame as the virtual-real fusion rendering frame.

[0033] The adaptive rendering system for text and image in the virtual-real scene fusion of the AI digital human according to the embodiment of the present invention includes the following modules: The text feature extraction module is used to perform word segmentation and semantic encoding on the Chinese text to generate a text feature vector; The image feature extraction module is used to perform region segmentation and texture extraction on the image data to generate an image region feature vector; The fusion modeling module is used to input the text feature vector and the image region feature vector into the neural ordinary differential equation transformer model to generate a fusion feature representation; The attention arrangement module is used to extract the attention weight matrix from the fusion feature representation and generate a preliminary text-image fusion layout; The parameter optimization module is used to optimize the network depth, the number of attention heads, and the time step control parameters based on the glowworm swarm optimization algorithm; The instruction generation module is used to generate a graphic rendering instruction set based on the preliminary text-image fusion layout; The pose illumination mapping module is used to generate a virtual-real fusion rendering frame by combining the digital human's action state and the background image information; The graphic rendering engine module is used to output the virtual-real fusion rendering frame as the video content of the fusion of Chinese text and image.

[0034] Embodiment 1: To verify the feasibility of the present invention in implementation, the present invention is applied to an AI digital human virtual explanation system for the immersive broadcast task of presenting digital cultural relics content to the public. In this task, the system needs to fuse high-resolution exhibit images from the cultural relic image database with explanatory Chinese sentences to achieve the effect of synchronous broadcast by the AI digital human and synchronous display of text and images. The traditional system mainly sets the display position of text on the image through static templates or manual configuration, and it is difficult to adapt to situations such as changes in the length of the text, differences in the complexity of the image area, and dynamic changes in the broadcast rhythm. Eventually, serious text-image misalignment, rigid layout, and poor user experience are caused.

[0035] The implementation of the present invention is based on the above virtual explanation scenario. Taking the exhibit description text and the exhibit image as inputs, first, a semantic encoding model is used to segment and encode the Chinese sentences to generate a text feature vector with a dimension of 512. For the image part, regional convolutional processing and texture encoding are used to extract the local features of 8 key regions in the image, forming an image region feature tensor with a size of 8256. Subsequently, a neural ordinary differential equation transformer model is constructed, with the initial network depth set to 4 layers, the number of attention heads set to 6, and the time step control parameter set to 0.1. Then, the above text and image features are input into the model to generate a fused feature representation tensor, capturing the cross-modal attention correlation between the text and the image.

[0036] By extracting the attention weight matrix generated by the multi-head attention module, the model automatically infers the correspondence between each phrase in the text and the image region, and calculates the layout position of the text in the image based on the attention center position and the relative weight size. After the preliminary layout is generated, the glowworm swarm optimization algorithm is further used to optimize the structural parameters. Eventually, the model structure depth converges to 6 layers, and the number of attention heads converges to 8, with a significant improvement in the fusion accuracy. Using the model with optimized structure, the system generates rendering instructions for each piece of text according to the fusion layout, including font (FangSong or Microsoft YaHei or KaiTi), font size (16–24), transparency (0.4–1.0), coordinates (normalized based on the width and height of the image), and synchronizes with the digital human broadcast, and combines the skeletal actions and real-time lighting state to complete the generation of the rendering frame.

[0037] In this implementation process, the effects of the traditional template layout method and the method described in the present invention are compared in terms of text-image alignment, readability, and rendering coherence. The table is shown as follows: Table 1: Comparative experimental table of text-image fusion effects between the method of the present invention and the traditional method

[0038] The above "Comparison Experiment Table of the Graphic and Text Fusion Effects between the Method of the Present Invention and Traditional Methods" shows the evaluation data of 10 groups of graphic and text samples in actual applications, comprehensively reflecting the performance advantages of the method of the present invention in graphic and text fusion. First of all, from the perspective of the graphic and text alignment index, the alignment of all samples numbered from 01 to 10 exceeds 90%, with the highest being 95.0% of sample 07, and the average alignment reaching 92.66%. This result indicates that the present invention has high accuracy in dealing with the spatial correspondence between Chinese text and image regions, significantly superior to the traditional template layout method.

[0039] In terms of text readability, the scores of all samples are above 4.4, with the highest reaching 4.8, and the average readability score being 4.53. This shows that the text styles generated by the present invention have high visual clarity and readability in terms of font, font size, and transparency control, etc., conform to the reading habits of users, and enhance the interaction experience.

[0040] The rendering coherence score reflects the stability and dynamic consistency of the graphic and text display during continuous broadcast of the system. The scores of each sample generally remain above 89%, with an average score of 91.26%, showing strong fusion coherence and content consistency. The improvement of this index is mainly due to the dynamic adaptation mechanism of the present invention in the pose mapping and lighting adjustment stages, effectively solving the problems of abrupt graphic and text display and hierarchical fragmentation in traditional systems.

[0041] In addition, the model structure parameters also show good optimization stability. The structure depth is mostly 6 layers, and the number of attention heads is mostly stable at 8. Some samples are slightly adjusted to adapt to the changes in image complexity, reflecting that the luminous insect swarm optimization algorithm has good adaptive adjustment ability while maintaining the high performance of the model.

[0042] This table verifies the advantages of the present invention in Chinese graphic and text fusion rendering from multiple perspectives. It not only comprehensively surpasses the traditional method in terms of alignment, readability, and coherence, but also reflects the effectiveness of the model structure design and optimization strategy, providing a reliable and efficient technical path for the graphic and text display of AI digital humans.

[0043] The above is only the preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, makes equivalent substitutions or changes, and should be covered by the protection scope of the present invention.

Claims

1. An adaptive rendering method for text and images in the fusion of virtual and real scenarios of AI digital humans, characterized in that, It includes the following steps: S1. Collect Chinese text and corresponding image data, perform word segmentation and semantic encoding on the Chinese text to generate a text feature vector; perform region segmentation and texture extraction on the image data to generate an image region feature vector; S2. Construct a neural ordinary differential equation transformer model, including a multi-head attention module and a variable time-step ODE calculation module. The structural parameter vector of the neural ordinary differential equation transformer model includes network depth, number of attention heads, and time-step control parameters. Input the text feature vector and the image region feature vector into the neural ordinary differential equation transformer model to generate a fused feature representation; S3. Extract the attention weight matrix generated by the multi-head attention module from the fused feature representation, and determine the spatial arrangement position of the Chinese text in the image according to the attention weight matrix to form a preliminary text-image fusion layout; S4. Adopt the glowworm swarm optimization algorithm to jointly optimize the network depth, number of attention heads, and time-step control parameters to obtain a fused model with optimized structure; S5. Based on the preliminary text-image fusion layout, generate a text-image rendering instruction set based on the fused model with optimized structure; S6. Combine the current action state of the digital human and the background image information, perform pose mapping and lighting adjustment on the text-image rendering instruction set to generate a virtual-real fusion rendering frame; S7. Input the virtual-real fusion rendering frame into the digital human rendering engine to complete the output of the video content of the Chinese text and image fusion.

2. The adaptive rendering method of text and image in the virtual-real scene fusion of AI digital humans according to claim 1, wherein The specific content of S1 includes: S11. Collect Chinese text and corresponding image data. The Chinese text is short text content with specific expression intentions, and the image data is an image containing scenes or objects related to the text semantics; S12. Perform Chinese language word segmentation on each Chinese text. Adopt a word segmentation method combining rules and data-driven to divide the original text into a word sequence with word boundaries; S13. On the basis of completing language word segmentation, encode the word sequence, extract semantic features containing context-dependent relationships, and form a text feature vector with a fixed dimension; S14. Perform region segmentation processing on each image data. Adopt a region division algorithm based on texture and boundary features to divide the image data into several image regions with consistent texture patterns; S15. Perform texture feature operations on the divided image data regions, extract the color distribution, edge direction, and local structure features of each region in the image, and form an image region feature vector.

3. The adaptive rendering method of text and image in the virtual-real scene fusion of AI digital humans according to claim 2, wherein, The specific content of S2 includes: S21. Use the text feature vector as the language input sequence and the image region feature vector as the visual input sequence to perform unified encoding operations on the two types of input sequences; S22. Input the uniformly encoded input sequences into the multi-head attention module. The multi-head attention module contains several parallel attention sub-heads, and each attention sub-head calculates the correlation between the text and the image to generate several attention representation matrices; S23. Perform splicing and linear mapping operations on each attention representation matrix to construct a fused attention output sequence as the output of the multi-head attention module; S24. Take the fused attention output sequence as the input feature and input it into the variable time-step ODE calculation module. Model the change of the input feature state over time based on the continuous-time dynamic function, which consists of the following structure: The first linear transformation unit jointly transforms the input feature and the time variable and then outputs an intermediate representation. Between the first linear transformation unit and the second linear transformation unit, a time-aware gating control unit is set to generate an input gate and a reset gate based on the current time variable and the state of the input feature, dynamically adjusting the channel information of the intermediate representation. The second linear transformation unit maps the gated feature representation and then inputs it into the activation function unit, which adopts a time-adjustable activation mechanism and uses the adjustable coefficient generated according to the time variable to control the response intensity of the non-linear output. The variable time-step ODE calculation module uses the time-step control parameter as the integration granularity control input and integrates the continuous-time dynamic function at the time step, outputting a multi-layer dynamic representation evolving over time. S25. The overall depth of the neural ordinary differential equation transformer model is determined by the continuously stacked number of variable time-step ODE calculation modules included. The stacked number serves as the network depth parameter to control the step size of each integration layer in the variable time-step ODE calculation module, achieving a balance between the feature calculation accuracy and the calculation efficiency through the time-step control parameter. S26. Take the multi-layer dynamic representation output by the variable time-step ODE calculation module as the fused feature representation.

4. The adaptive rendering method of text and image in the fusion of virtual and real scenarios of AI digital humans according to claim 3, wherein The specific content of S3 includes: S31. Extract the attention weight matrix generated by the multi-head attention module from the fused feature representation. The attention weight matrix characterizes the matching strength between each word in the text feature vector and each region in the image region feature vector. S32. According to the maximum weight value corresponding to each text in the attention weight matrix, determine the image region corresponding to each Chinese text, and establish a one-to-one mapping relationship between the Chinese text and the image region. S33. According to the one-to-one mapping relationship, extract the spatial coordinate range of each corresponding image region, determine the layout position of the Chinese text on the image, and generate a preliminary text-image fusion layout including the display positions of all Chinese texts. S34. Perform conflict detection and position adjustment processing on the preliminary text-image fusion layout, and set a minimum spacing threshold between any two adjacent Chinese words to meet the readability of the text layout and the non-overlap of regions.

5. The adaptive rendering method of text and image in the fusion of virtual and real scenarios of AI digital humans according to claim 4, characterized in that The specific content of S4 includes: S41. Initialize the glowworm swarm, where each glowworm individual corresponds to a set of structure parameter vectors, which are composed of three parts: the network depth, the number of attention heads, and the time-step control parameter in the neural ordinary differential equation transformer model. S42. Set a brightness function for each glowworm individual. The brightness function takes the comprehensive performance index of the neural ordinary differential equation transformer model on the validation set as the objective function value. The comprehensive performance index is composed of a weighted combination of three items: the text-image layout accuracy, the sparsity of the attention distribution, and the stability of the fused features. The sum of the weighting coefficients of each index is one. S43. In each generation of iteration, according to the relative difference between the individual brightness value and the brightness values of other individuals in the neighborhood, based on the difference between the current individual's structural parameter vector and the structural parameter vector of the individual with higher brightness in the neighborhood, a movement function is constructed through the brightness difference weight and the spatial distance factor to update the network depth, the number of attention heads, and the time step control parameters of the current individual; S44. During the movement process, a step size decay mechanism and parameter boundary control conditions are set to prevent the structural parameter vector from exceeding the boundary and maintain the adaptability and convergence of the search area; S45. When the preset maximum number of iterations or the convergence condition of the brightness function is reached, the glowworm individual with the optimal brightness value is selected, and the corresponding structural parameter vector is the optimal combination of the network depth, the number of attention heads, and the time step control parameters; S46. Based on the optimal combination, the neural ordinary differential equation transformer model is reconstructed as the fused model after structure optimization.

6. The adaptive rendering method of text and image in the virtual-real scene fusion of AI digital humans according to claim 5, characterized in that The S5 graphic and text rendering instruction set includes: determining the font, color, transparency, and display coordinates in the image of each Chinese text based on the fused model after structure optimization, constructing graphic and text rendering instructions containing text style parameters and spatial position parameters, and combining them to generate a graphic and text rendering instruction set.

7. The adaptive rendering method of text and image in the virtual-real scene fusion of AI digital humans according to claim 6, wherein The S6 specifically includes: obtaining the skeletal pose parameters of the digital human and the illumination distribution characteristics of the background image in the current frame, and performing coordinate transformation and brightness adjustment based on the pose angle and illumination direction according to the graphic and text rendering instruction set, and outputting the processed image frame as the virtual-real fusion rendering frame.

8. An adaptive rendering system for text and images in the integration of virtual and real scenarios of AI digital humans, which is applied to the adaptive rendering method for text and images in the integration of virtual and real scenarios of AI digital humans according to any one of claims 1 to 7, characterized in that, It includes the following modules: A text feature extraction module, which is used to perform word segmentation and semantic encoding on Chinese text to generate text feature vectors; An image feature extraction module, which is used to perform region segmentation and texture extraction on image data to generate image region feature vectors; A fusion modeling module, which is used to input the text feature vectors and image region feature vectors into the neural ordinary differential equation transformer model to generate fusion feature representations; An attention arrangement module, which is used to extract the attention weight matrix from the fusion feature representation and generate a preliminary graphic and text fusion layout; A parameter optimization module, which is used to optimize the network depth, the number of attention heads, and the time step control parameters based on the glowworm swarm optimization algorithm; An instruction generation module, which is used to generate a graphic and text rendering instruction set based on the preliminary graphic and text fusion layout; A pose illumination mapping module, which is used to generate a virtual-real fusion rendering frame by combining the action state of the digital human and the background image information; A graphic and text rendering engine module, which is used to output the virtual-real fusion rendering frame as video content with Chinese characters and images fused.

Images