Digital human full-media production method

Abstract

This invention provides a multimedia production method for digital humans, comprising: acquiring multimodal media asset input and establishing a connection with a multidimensional tagging system; responding to the needs of digital human production tasks, recommending multimodal media assets required by different digital humans through the multidimensional tagging system; receiving raw content instructions input by users, generating structured scripts, and automatically generating a digital human recommendation list; selecting the required digital human based on the recommendation list, and automatically generating voice-over, action sequences, and camera movement instructions based on the recommended multimodal media assets and structured scripts, aligning and integrating them on a unified timeline, and outputting a multimodal composite content package; adjusting the specifications of the multimodal composite content package based on the target terminal type before distributing it to the corresponding target terminal. This invention solves the technical problems of fragmented multimodal element collaboration, insufficient cross-terminal adaptation and compatibility, and lack of data closure in production decision-making in existing digital human production.

Description

Technical Field

[0001] This invention relates to the field of multimedia processing, and in particular to a method for producing digital humans using multimedia. Background Technology

[0002] A digital human is a digitally created avatar that closely resembles a human, produced using digital technology. They are widely used in video creation, live streaming, news broadcasting, voice prompts, and other scenarios. Currently, users can generate multimedia resources corresponding to digital humans, such as digital human videos and animations, through digital human generation platforms according to their own needs.

[0003] Digital human technology has been widely applied in multimedia processing, but it suffers from shortcomings in technical implementation, scenario adaptation, model optimization, and user experience. Core technical deficiencies include insufficient multimodal fusion capabilities and shortcomings in the spatiotemporal synchronization and semantic matching of voice, vision, and motion data. For example, the synchronization error between speech recognition results and lip-syncing animation / body movements is too large, and the matching degree between text and action generation is insufficient, leading to a "disconnect between speech and action" phenomenon during interaction. Furthermore, bottlenecks exist in encoding and allocation rates for large-screen TV broadcasts, as well as in the processing performance of real-time rendering. Finally, in model optimization, there are data dependencies and annotation challenges; massive amounts of multimodal training data (text, video, audio) require support from large media asset libraries, text libraries, and audio libraries. Regarding user experience, there is a lack of methods and product forms for cross-stage integration and end-to-end production. Summary of the Invention

[0004] This application provides a method for producing multimedia digital humans to address the problems in the background art.

[0005] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0006] According to a first aspect of the embodiments of this application, a method for producing multimedia digital humans is provided, comprising: Acquire multimodal media asset input, perform semantic understanding and alignment on the multimodal media assets, and establish the association between multimodal media assets and a multidimensional tag system; respond to the needs of digital human production tasks, recommend multimodal media assets required by different digital humans through the multidimensional tag system; Receive raw content instructions from the user, generate a structured script document, and automatically generate a digital human recommendation list based on the structured script document; Select the desired digital human from the digital human recommendation list, and automatically generate voice-over, action sequences and camera movement instructions based on the multimodal media assets and structured scripts recommended by the selected digital human. Align and integrate them on a unified timeline to output a multimodal composite content package. Based on the target terminal type, the specifications of the multimodal synthesized content package are adjusted before it is distributed to the corresponding target terminal.

[0007] According to one embodiment of this application, the multi-dimensional tagging system includes content theme, sentiment tendency, scene type, character role, action semantics, and regional cultural characteristics, wherein the regional cultural characteristics include semantic modeling and knowledge graph embedding of local characteristic elements.

[0008] According to one embodiment of this application, the step of performing semantic understanding and alignment on the multimodal media assets to establish the association between the multimodal media assets and the multidimensional tagging system specifically includes: Based on a lightweight cross-modal pre-trained model, basic semantic vectors of text, images, audio, and video in multimodal media assets are extracted respectively; A cross-modal attention mechanism is adopted to enhance the semantic vector association between different modalities; The associated semantic vectors are enhanced by stacking Transformer encoders; Then, dynamic calibration is performed in conjunction with digital human production tasks to ultimately establish the connection between multimodal media assets and a multidimensional tagging system.

[0009] According to one embodiment of this application, the step of recommending multimodal media resources required by different digital humans through a multi-dimensional tagging system in response to the needs of digital human production tasks specifically includes: Responding to the needs of digital human production tasks, it automatically identifies the required tag combinations; Based on the required tag combination, retrieve and recommend multimodal media assets with a semantic matching degree of ≥0.85 and a tag coverage completeness of ≥80%.

[0010] According to one embodiment of this application, the step of receiving the user's input of raw content instructions and generating a structured script document specifically includes: Receive raw content instructions from the user; By calling a domain-adaptive large language model and combining digital human role settings, broadcast scene types and target platform specifications, an AI script with complete structure, appropriate tone and reasonable rhythm is automatically generated. The generated AI script text is formatted and standardized, with unified timestamp markers, sentiment annotations, accent marks, and action trigger markers, resulting in a standardized structured script text.

[0011] According to one embodiment of this application, the automatic generation of a digital human recommendation list based on the structured script document specifically includes: Semantic parsing is performed on the structured script to extract core tags; these core tags cover the script's theme, core content, stylistic features, domain, and emotional tone. By using vector matching algorithms to quickly search and compare multiple dimensions in the digital human tag library, a set of digital human candidates with the highest overlap with core tags and the strongest semantic fit is selected. Sentiment analysis is performed on structured script texts using a pre-trained language model to obtain sentiment analysis results; Based on the sentiment analysis results, the digital humans in the candidate set are sorted in descending order to generate a digital human recommendation list.

[0012] According to one embodiment of this application, the step of selecting a desired digital human from the digital human recommendation list, and automatically generating voice-over, action sequences, and camera movement instructions based on the multimodal media assets and structured scripts recommended by the selected digital human, aligning and integrating them on a unified timeline, and outputting a multimodal composite content package, specifically includes: Obtain a structured script; the structured script includes segmented text content, preset emotion types, key action prompts, and shot intention tags; Based on each paragraph of text in the structured script, the speech synthesis service is invoked to generate the corresponding digital human voiceover, and the start and end times and duration of the voiceover are recorded simultaneously. Based on the timeline of the dubbing and the preset emotion types and key action cues in the structured script, the corresponding facial expressions, lip movements and body movement segments are matched from the digital human action library and spliced ​​into a complete action sequence in chronological order. Based on the shot intent tags in the structured script, select the appropriate virtual shot scheme, including shot type, shot transition rhythm and camera movement, and generate corresponding shot scheduling instructions; The voice-over, motion sequences, and camera movement instructions are aligned and integrated on a unified timeline to output a multimodal composite content package for video rendering or real-time playback.

[0013] According to one embodiment of this application, when targeting a presenter-level digital human, the process further includes optimizing the voice-over and lip-sync: The system receives the text to be broadcast, performs deep semantic analysis on the text, identifies and marks polyphonic characters, ambiguous phrases and domain-specific terms; based on semantic dependency relations and a preset host prosody rule library, it automatically generates prosodic text with hierarchical pause marks, stress positions and intonation curves. The prosodic text is input into the speech synthesis model, which binds the contextual features and standard pronunciation of polyphonic characters through an attention mechanism, and dynamically adjusts the inter-frame transition parameters according to hierarchical pause markers to generate a target speech signal and corresponding phoneme time sequence that conforms to the host's broadcasting habits. A multimodal encoder is used to extract the acoustic features of the target speech signal and the three-dimensional features of the digital human face benchmark model. Cross-modal feature fusion is performed through a deep feature perception model to generate enhanced lip-speech synchronization features. The phoneme time series and enhanced lip-sync features are input into a conditional generative adversarial network to complete the mouth shape sequence optimization. The target speech signal and the optimized lip-sync sequence are time-stamped and then output to the digital human rendering engine for the generation of multimodal synthetic content for presenter-level digital humans.

[0014] According to one embodiment of this application, adjusting the specification parameters of the multimodal synthesized content package based on the target terminal type before distributing it to the corresponding target terminal specifically includes: Based on the terminal capability detection protocol, obtain the target terminal device's capabilities and software decoding support list; the target terminal includes at least small-screen mobile terminals and large-screen broadcast-grade equipment. Based on the target terminal's capabilities and the software decoding support list, the encoding format, resolution, bit rate, frame rate, and color space of the multimodal synthesized content package are adaptively converted to generate a broadcast file or data stream that precisely matches the target terminal.

[0015] According to one embodiment of this application, it further includes collecting full-link data after broadcasting, performing data analysis to generate optimization suggestions, and feeding the optimization suggestions back to the preceding stages to achieve full-process iteration.

[0016] Compared with existing technologies, the beneficial effects of adopting the above technical solution are as follows: This invention can solve the technical problems of fragmented multimodal element collaboration, insufficient cross-terminal adaptation and compatibility, and lack of data closed loop in production decision-making in existing digital human production, and realize the integrated multimedia production of the entire process from "production element input - content fusion and synthesis - multi-terminal adaptation output - post-broadcast feedback optimization". Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0018] Figure 1This is a flowchart of a digital human multimedia production method according to an embodiment of this application.

[0019] Figure 2 This is a schematic diagram illustrating the implementation of the multimedia production method for digital humans according to an embodiment of this application.

[0020] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0021] The embodiments of this application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar modules or modules having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. Rather, the embodiments of this application include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.

[0022] To address the issues of fragmented multimodal element collaboration, insufficient cross-terminal adaptation and compatibility, and lack of data closure in production decision-making in existing digital human production, this application proposes a digital human multimedia production method. This method achieves integrated multimedia production across the entire process, from "production element input - content fusion and synthesis - multi-terminal adaptation output - post-broadcast feedback optimization." Please refer to [reference needed]. Figure 1 , Figure 2 The specific steps are as follows: S101. Obtain multimodal media asset input, perform semantic understanding and alignment on the multimodal media assets, and establish the association between multimodal media assets and multidimensional tag system; in response to the needs of digital human production tasks, recommend multimodal media assets required by different digital humans through multidimensional tag system.

[0023] This embodiment employs cross-modal semantic mapping technology and multi-dimensional tag system construction technology. Based on the pre-trained multimodal model CLIP, it achieves text-image-audio feature association. Through a hierarchical weighted matching algorithm, it enables the collaborative input of lightweight media assets (video, audio, images, newspapers), multimedia articles, digital humans, and synchronous sound. This solves the technical problems of single tag dimensions, difficulty in cross-modal matching, and difficulty in identifying and reusing local content in traditional media asset systems. Multimedia articles include images, text, and video. Different types of articles can be created using channels such as WeChat, Weibo, and Douyin. The article editing page has image and video materials and supports image and text layout. Synchronous sound refers to the sound recorded synchronously with the footage during filming, not added later. It includes human voices and ambient sounds; all camera footage stored in the intelligent media asset library contains synchronous sound.

[0024] Specifically, this embodiment constructs a multi-dimensional tagging system for digital human content production, including content theme, emotional tendency, scene type, character role, action semantics, and regional cultural characteristics. Among them, regional cultural characteristics include semantic modeling and knowledge graph embedding of local characteristic elements.

[0025] To establish a multimodal media asset and multidimensional tagging system, it is necessary to perform cross-modal deep semantic understanding and alignment on the input multimodal media assets.

[0026] First, modality-specific feature extraction is required for the input multimodal media assets: feature extraction is completed using a quantized cross-modal pre-trained model (CLIP fine-tuned version), and basic semantic vectors of text (BERT semantic encoding), image (CNN visual features), audio (MFCC+Librosa temporal features), and video (frame-level visual features + action semantic features) are extracted through modality-specific branches.

[0027] Then, cross-modal attention fusion is performed: In this embodiment, a cross-modal attention mechanism is introduced to strengthen the semantic vector association between different modalities (such as the semantic binding between "plum picking action" in the video and "dialect explanation" in the audio, and "processing technology" in the text), so as to solve the problem of incomplete semantic expression in a single modality.

[0028] Next, semantic enhancement and dynamic calibration are performed: the fused semantic vector is enhanced by stacking Transformer encoders to mine implicit related information and extract deep semantics of "regional culture + action semantics + scene type" from the video synchronously.

[0029] Finally, dynamic calibration is performed in conjunction with the context of digital human production tasks, and cosine similarity calculation is performed with the multi-dimensional labeling system to complete cross-modal matching and establish the association between the multi-dimensional labeling system and multi-modal media assets.

[0030] Based on the association between multimodal media assets and a multidimensional tagging system, this embodiment proposes another context-aware intelligent retrieval and recommendation method to proactively recommend materials. Specifically, in response to the scenario requirements of digital human production tasks (such as cultural tourism promotion), it automatically identifies the required tag combinations, retrieves and recommends multimodal candidate materials with "semantic matching degree ≥ 0.85 + tag coverage completeness ≥ 80%", thereby optimizing the process from "manual query" to "proactive recommendation", significantly improving the efficiency of material retrieval, and increasing the reuse rate of local characteristic content.

[0031] In practical applications, a multimedia tagging system can be used to establish a connection between multimedia media assets and digital humans, and corresponding multimedia media assets can be directly recommended after selecting a digital human.

[0032] S102. Receive the user's input of the original content instruction, generate a structured script document, and automatically generate a digital human recommendation list based on the structured script document.

[0033] In practical applications, users provide raw content instructions. These instructions typically include information such as the scenario, keywords, a thematic outline, the target audience, or reference examples. Based on these raw content instructions, this embodiment utilizes AI to automate the generation and standardization of the document, supporting subsequent multimodal synthesis.

[0034] Specifically, after obtaining the original content instructions, the system invokes a domain-adaptive large language model, combining the digital human role setting, broadcast scenario type, and target platform specifications to automatically generate an AI script that is structurally complete, has an appropriate tone, and a reasonable rhythm. The digital human role setting is primarily determined based on the user's original content instructions, representing the user's desired digital human style. For example, a government propaganda document corresponds to a professional and rigorous style; while a sales-oriented document corresponds to a lively style with large gestures. In some embodiments, users can also customize the digital human role style.

[0035] Then, the generated AI script is standardized in format, and the timestamp markers, emotion annotations, accent marks, and action trigger markers are unified. The output is a structured script, which serves as a unified input source for subsequent speech synthesis, expression-driven processing, and action choreography.

[0036] The structured script document in this embodiment can effectively solve the problems of low efficiency, inconsistent style, and lack of multimodal driving signals in manual script writing.

[0037] After generating a structured script, a digital human recommendation list can be generated based on this script. In this embodiment, a technical closed loop of "script semantic parsing - precise digital human matching - topic selection reverse empowerment" supports efficient collaboration in subsequent production stages and provides support for script creation in terms of topic selection angles, content directions, and prediction of popular topics. This addresses the pain points of the traditional "write the script first, then select the digital human" model, which suffers from "blind creation, low content-digital human fit, and missed popular topics." The recommendation process mainly involves script tag matching, sentiment analysis, and multi-dimensional scoring, forming an efficient and accurate recommendation closed loop and providing a fundamental guarantee for the recommendation effect.

[0038] Specifically, the structured script is first subjected to deep semantic analysis to extract core tags. These core tags comprehensively cover key information such as the script's theme, core content, stylistic features, domain, and emotional tone. In the tag extraction stage, several advanced technologies in Natural Language Processing (NLP) are integrated. By constructing a lexical semantic association network, the deep logical relationships between words in the text are accurately captured, improving the semantic accuracy and content matching of tag extraction.

[0039] The digital human tag library contains a pre-defined set of tags for each digital human that match their characteristics, areas of expertise, language style, and emotional inclinations. After extracting the core tags, a highly efficient vector matching algorithm is used to quickly search and compare the tags across multiple dimensions within the digital human tag library, ultimately selecting the candidate set of digital humans with the highest overlap and semantic fit with the document's tags.

[0040] Text sentiment analysis technology, as a key support for achieving "sentiment-based recommendation" in this embodiment, adopts a multimodal fusion analysis framework. Based on the Transformer architecture, it incorporates pre-trained language models such as BERT and GPT series. Through deep semantic understanding of user input text, it can not only accurately determine the overall sentiment tendency of the text (positive, negative, neutral), but also further subdivide it into specific sentiment categories such as joy, anger, sadness, and surprise, providing refined sentiment basis for subsequent personalized recommendations.

[0041] Based on sentiment analysis results, this embodiment uses a multi-dimensional scoring model to complete the final recommendation ranking of digital humans. The recommendation results include key information such as the digital human's name, core tags, and emotional fit, presented in descending order of score. The recommendation strategy fully reflects emotional fit: when a user is detected to be in a positive and excited emotional state, digital humans with lively appearances, bright visual colors, and strong expressive movements are prioritized; if the user's emotional tendency is low or sad, digital humans with gentle appearances, concerned expressions, and soft, soothing voices are matched. In terms of interaction details, voice style and speech expression also achieve emotional linkage—when the user is anxious, the digital human uses a concise, direct, and slightly faster speech response; when the user is calm and relaxed, the interaction style is calm and peaceful with a moderate speech speed; when facing an angry user, the digital human first uses soothing language to calm the emotion before solving the problem; for a joyful user, the interaction uses positive and uplifting language to achieve emotional resonance.

[0042] In this embodiment, when constructing the multi-dimensional scoring model, core dimensions such as content relevance, user historical preferences, and digital human interaction performance are comprehensively considered. A weight allocation strategy combining the Analytic Hierarchy Process (AHP) and the entropy weight method is adopted. Through dual calibration of "experience-based assignment + data-driven approach," the industry rationality of the weight settings is ensured, and the model's adaptability is improved through dynamic updates of real-time interactive data, ensuring the objectivity of the scoring results and the accuracy of the recommendations.

[0043] S103. Select the desired digital human from the digital human recommendation list, and automatically generate voice-over, action sequences and camera movement instructions based on the multimodal media assets and structured scripts recommended by the selected digital human. Align and integrate them on a unified timeline to output a multimodal composite content package.

[0044] In practical applications, you can select the desired digital human from the digital human recommendation list. When selecting a digital human, multimodal media assets associated with that digital human will be automatically recommended.

[0045] Once the digital human is identified, an end-to-end multimodal content generation process can be executed based on standardized structured scripts, specifically including: S1031. Receive a structured script, which includes segmented text content, preset emotion types, key action cues, and shot intention tags.

[0046] S1032. Based on each paragraph of the structured script, automatically call the speech synthesis service to generate the corresponding digital human voiceover, and simultaneously record the start and end times and duration of the voiceover. S1033. Based on the timeline of the dubbing and the preset emotion types and key action prompts in the structured script, automatically match the corresponding facial expressions, lip movements and body movement segments from the digital human motion library, and splice them into a complete action sequence in chronological order.

[0047] S1034. Based on the shot intent tags and script context semantics marked in the structured script document, automatically select the appropriate virtual shot scheme, including shot type, shot switching rhythm and camera movement method, and generate corresponding shot scheduling instructions. S1035. Align and integrate voice-over, motion sequences, and camera movement instructions on a unified timeline to output a multimodal composite content package that can be directly used for video rendering or real-time playback.

[0048] It should be added that when aligning and integrating voice-over, motion sequences, and camera movement instructions, if the duration of the motion and voice-over differs by more than 10%, it can be corrected by stretching / compressing the motion and inserting transition frames, thereby outputting a composite content package that can be directly rendered.

[0049] Furthermore, this embodiment also provides an optimization scheme for voice-over and lip-sync coordination for presenter-level digital humans, including the following process: (1) Text preprocessing: Receive the text to be broadcast, call the pre-trained language model that integrates the host's broadcast corpus, perform deep semantic analysis on the text, identify and mark polyphonic characters, ambiguous phrases and domain-specific terms; based on semantic dependency relations and the preset host prosody rule library, automatically generate prosodic text containing hierarchical pause markers (between sentences, between meaning groups, before emphasis), stress position and intonation change curve.

[0050] In one embodiment, the host prosody rule base is generated by collecting at least 100 hours of different types of host broadcast videos (news, interviews, commentary), manually annotating prosody parameters, and then training with a transfer learning algorithm. It can be dynamically adapted according to the broadcast scenario (such as political news, entertainment information).

[0051] (2) Accurate speech synthesis: The prosodic text is input into the speech synthesis model. The model binds the contextual features and standard pronunciation of polyphonic characters through an attention mechanism, and dynamically adjusts the inter-frame transition parameters according to the hierarchical pause markers to generate the target speech signal and corresponding phoneme time sequence that conforms to the host's broadcasting habits.

[0052] (3) Simultaneous generation and optimization of lip movements: A multimodal encoder is used to extract the acoustic features (including fundamental frequency and spectral envelope) of the target speech signal and the three-dimensional features of the digital human face reference model. Cross-modal feature fusion is performed through a deep feature perception module to generate enhanced lip-speech synchronization features.

[0053] Phoneme time series and enhanced lip-sync features are input into a conditional generative adversarial network (GAN). The generator produces lip-sync animation sequences based on standard lip-sync templates corresponding to the phonemes (containing BlendShape parameters for all Chinese phonemes). The discriminator evaluates the similarity and naturalness of the generated lip-sync to that of a real presenter, and optimizes lip-sync details using a deep feature perception loss function. In one embodiment, the generator of the GAN employs a three-level decoding architecture, performing layered parsing of the fused features and enhancing local parameters for lip expansion during vowel pronunciation and lip-tooth closure during consonant pronunciation, thereby improving the smoothness of inter-frame transitions in the lip-sync animation sequence by more than 30%.

[0054] A speech-lip-sync timing alignment model is constructed to perform timestamp calibration on the target speech signal and the optimized lip-sync animation sequence with millisecond-level precision to eliminate accumulated delay. The calibrated speech and lip-sync data are synchronously output to the digital human rendering engine to drive the digital human to complete presenter-level broadcasting.

[0055] S104. Based on the target terminal type, adjust the specifications of the multimodal synthesized content package before distributing it to the corresponding target terminal.

[0056] In this embodiment, the target terminal's capabilities and software decoding support list are first obtained based on a terminal capability detection protocol. In this embodiment, the terminal capability detection protocol obtains the terminal's hardware capabilities and software decoding support list via SNMP (Simple Network Management Protocol) or UPnP (Universal Plug and Play) protocol. The target terminal includes at least small-screen mobile terminals and large-screen broadcast-grade equipment.

[0057] Then, based on the target terminal's capabilities and the software decoding support list, the encoding format, resolution, bit rate, frame rate, and color space of the multimodal synthesized content package are adaptively converted to generate a broadcast file or data stream that precisely matches the target terminal.

[0058] In one embodiment, parameters can be preset, with H.264 (AVC) High Profile encoding used by default to ensure compatibility. Specific preset parameters include: Video encoding: H.265 / HEVC, conforming to the SMPTE ST 2110 uncompressed video stream over IP standard; Resolution: 4K (3840×2160); Frame rate: 50fps (PAL standard); Bitrate range: 15-25 Mbps; Color space: BT.2020 wide color gamut; Audio encoding: AAC-LC or PCM, sampling rate 48kHz, bitrate not less than 384kbps; Physical interface: supports 12G-SDI signal output to ensure seamless integration with broadcast production systems, i.e., default adaptation to large-screen target terminals. When the preset parameters do not meet the broadcast specifications of the target terminal, such as when the target terminal is a small-screen mobile terminal, the adaptive conversion processing target parameters are: video encoding: H.264 / AVC, compliant with 3GPP standards; resolution: adaptive 720P (1280×720) or 1080P (1920×1080).

[0059] This embodiment synchronously implements "broadcast-grade parameter preset" on the cloud synthesis platform. Through the processing method of "preset parameters + dynamic fine-tuning", it achieves efficient distribution of synthesized media content (including digital human images), which avoids secondary transcoding loss and takes into account scene adaptability.

[0060] Please continue to refer to this. Figure 1 The digital human multimedia production method proposed in this application further includes: S105, collecting full-link data after broadcasting, performing data analysis to generate optimization suggestions, and feeding the optimization suggestions back to the preceding stages to achieve full-process iteration.

[0061] In this embodiment, the collected data includes dissemination data (i.e., target terminal) and production data. Dissemination data includes playback volume, completion rate (segmented statistics, calculated in 10-second intervals), interaction rate (likes / comments / shares), and sentiment analysis based on user comments / bullet screen text. Production data includes material reuse rate, digital human call rate, percentage of abnormal actions (segmentation with action alignment error > 8ms), and tag matching accuracy.

[0062] Then, data analysis is performed on the collected data. In this embodiment, a traceable three-dimensional dashboard of "production efficiency - dissemination effect - audience preference" is constructed, specifically including: (1) Production efficiency dimension: display material reuse rate, digital human call rate, and abnormal action ratio. Set thresholds: material reuse rate <10% is "low frequency material", digital human call rate <5% is "low frequency digital human", and abnormal action ratio >8% is "digital human that needs optimization".

[0063] (2) Dissemination effect dimension: Analyze user interaction data and content completion rate.

[0064] (3) Audience preference dimension: Identify users' content preference tags (scene and topic preferences), digital human preference type (such as friendly type, professional type), and positive or negative emotions.

[0065] Optimization of multimodal media assets and digital human assets can be achieved using 3D dashboards, including: For low-frequency content / digital humans: reduce their recommendation weight by 30% and remove them from the "priority recommendation pool".

[0066] For digital humans with high abnormal movements: optimize the lip-sync algorithm to reduce the proportion of abnormal movements below the threshold.

[0067] For digital human assets with high negative impact: mark them as "sensitive assets", suspend automatic recommendations, and synchronize them to the tag library.

[0068] Finally, based on the data analysis results, precise iterations are carried out, which mainly include the following processes.

[0069] (1) Adjust tag weights: Extract the core tag combination of highly interactive content (interaction rate ≥ 1.5 times the average of the same topic), and then increase the weight of the multi-dimensional tag system and the digital human multi-dimensional scoring model by 20%; (2) Trial production verification: Based on the adjusted tag weights, generate 3-5 test contents and record the three-dimensional data of "production efficiency-dissemination effect-audience preference". The following conditions must be met: completion rate, negative feedback rate, and tag matching accuracy rate reach the threshold to verify the effectiveness of optimization.

[0070] (3) Based on the parameters (core tag combination, digital human type, recommendation weight) that have passed the trial production verification, a standardized recommendation strategy is generated.

[0071] (4) The optimized new recommendation strategy is synchronized to the media asset retrieval recommendation, digital human recommendation and multi-dimensional tag system architecture process to achieve full-link collaborative updates.

[0072] Based on the same technical concept, embodiments of this application also provide an electronic device that can implement the multimedia production method process for digital humans provided in the above embodiments of the present invention. In one embodiment, the electronic device can be a server, a terminal device, or other electronic devices. Figure 3 As shown, the electronic device may include: At least one processor and a memory connected to the at least one processor. In this embodiment of the invention, the specific connection medium between the processor and the memory is not limited. Figure 3 The example used is the connection between the processor and memory via a bus. The bus... Figure 3 The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. Buses can be divided into address buses, data buses, control buses, etc., but for ease of representation, [the specific bus type is not shown here]. Figure 3 The processor is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, a processor can also be called a controller; there are no restrictions on the name.

[0073] In this embodiment of the invention, the memory stores instructions executable by at least one processor. By executing the instructions stored in the memory, the at least one processor can perform the multimedia production method for a digital human described above. The processor can implement... Figure 3 The functions of each module in the device shown.

[0074] The processor is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory and calling data stored in memory, it can monitor the device's various functions and process data, thereby enabling overall monitoring of the device.

[0075] In an alternative design, the processor may include one or more processing units. The processor may integrate an application processor and a modem processor, wherein the application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may also not be integrated into the processor. In some embodiments, the processor and memory may be implemented on the same chip; in some embodiments, they may also be implemented separately on separate chips.

[0076] The processor can be a general-purpose processor, such as a CPU, digital signal processor, application-specific integrated circuit, field-programmable array, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the digital human multimedia production method disclosed in the embodiments of this invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0077] Memory, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory can include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures, and accessible by a computer, but is not limited thereto. In embodiments of the present invention, memory can also be a circuit or any other device capable of implementing storage functions, used to store program instructions and / or data.

[0078] By designing and programming the processor, the code corresponding to the multimedia production method of the digital human described in the foregoing embodiments can be embedded into the chip, enabling the chip to execute the steps of the methods described in the foregoing embodiments during operation. How to design and program the processor is a technique well-known to those skilled in the art and will not be elaborated upon here.

[0079] Based on the same inventive concept, embodiments of the present invention also provide a storage medium storing computer instructions that, when executed on a computer, cause the computer to perform a multimedia production method for a digital human described above.

[0080] In some alternative embodiments, the present invention also provides a digital human multimedia production method in which various aspects can also be implemented as a program product comprising program code that, when the program product is run on a device, causes the control device to perform the steps in a digital human multimedia production method according to various exemplary embodiments of the present invention as described in this specification.

[0081] It should be noted that although several units or sub-units of the apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the invention, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units. Furthermore, although the operation of the method of the invention is described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0082] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0083] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a server, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0084] Program code for performing the operations of this invention can be written using any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0085] In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0086] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0087] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0088] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimedia production method for digital humans, characterized in that, include: Acquire multimodal media asset input, perform semantic understanding and alignment on the multimodal media assets, and establish the association between multimodal media assets and a multidimensional tag system; In response to the needs of digital human production tasks, a multi-dimensional tagging system is used to recommend multimodal media resources required for different digital humans; Receive raw content instructions from the user, generate a structured script document, and automatically generate a digital human recommendation list based on the structured script document; Select the desired digital human from the digital human recommendation list, and automatically generate voice-over, action sequences and camera movement instructions based on the multimodal media assets and structured scripts recommended by the selected digital human. Align and integrate them on a unified timeline to output a multimodal composite content package. Based on the target terminal type, the specifications of the multimodal synthesized content package are adjusted before it is distributed to the corresponding target terminal.

2. The multimedia production method for digital humans according to claim 1, characterized in that, The multi-dimensional tagging system includes content theme, sentiment, scene type, character role, action semantics, and regional cultural characteristics. Among them, regional cultural characteristics include semantic modeling and knowledge graph embedding of local characteristic elements.

3. The multimedia production method for digital humans according to claim 1 or 2, characterized in that, The semantic understanding and alignment of the multimodal media assets, and the establishment of the association between the multimodal media assets and the multidimensional tag system, specifically include: Based on a lightweight cross-modal pre-trained model, basic semantic vectors of text, images, audio, and video in multimodal media assets are extracted respectively; A cross-modal attention mechanism is adopted to enhance the semantic vector association between different modalities; The associated semantic vectors are enhanced by stacking Transformer encoders; Then, dynamic calibration is performed in conjunction with digital human production tasks to ultimately establish the connection between multimodal media assets and a multidimensional tagging system.

4. The multimedia production method for digital humans according to claim 1, characterized in that, In response to the needs of digital human production tasks, a multi-dimensional tagging system is used to recommend multimodal media resources required for different digital humans, specifically including: Responding to the needs of digital human production tasks, it automatically identifies the required tag combinations; Based on the required tag combination, retrieve and recommend multimodal media assets with a semantic matching degree of ≥0.85 and a tag coverage completeness of ≥80%.

5. The multimedia production method for digital humans according to claim 1, characterized in that, The step of receiving the user's input of raw content instructions and generating a structured script document specifically includes: Receive raw content instructions from the user; By calling a domain-adaptive large language model and combining digital human role settings, broadcast scene types and target platform specifications, an AI script with complete structure, appropriate tone and reasonable rhythm is automatically generated. The generated AI script text is formatted and standardized, with unified timestamp markers, sentiment annotations, accent marks, and action trigger markers, resulting in a standardized structured script text.

6. The multimedia production method for digital humans according to claim 1 or 5, characterized in that, The automatic generation of a digital human recommendation list based on the structured script document specifically includes: Semantic parsing is performed on the structured script to extract core tags; these core tags cover the script's theme, core content, stylistic features, domain, and emotional tone. By using vector matching algorithms to quickly search and compare multiple dimensions in the digital human tag library, a set of digital human candidates with the highest overlap with core tags and the strongest semantic fit is selected. Sentiment analysis is performed on structured script texts using a pre-trained language model to obtain sentiment analysis results; Based on the sentiment analysis results, the digital humans in the candidate set are sorted in descending order to generate a digital human recommendation list.

7. The multimedia production method for digital humans according to claim 1, characterized in that, The process involves selecting the desired digital human from the recommended list, and automatically generating voice-over, action sequences, and camera movement instructions based on the multimodal media assets and structured scripts recommended by the selected digital human. These are then aligned and integrated on a unified timeline to output a multimodal composite content package. Specifically, this includes: Obtain a structured script; the structured script includes segmented text content, preset emotion types, key action prompts, and shot intention tags; Based on each paragraph of text in the structured script, the speech synthesis service is invoked to generate the corresponding digital human voiceover, and the start and end times and duration of the voiceover are recorded simultaneously. Based on the timeline of the dubbing and the preset emotion types and key action cues in the structured script, the corresponding facial expressions, lip movements and body movement segments are matched from the digital human action library and spliced ​​into a complete action sequence in chronological order. Based on the shot intent tags in the structured script, select the appropriate virtual shot scheme, including shot type, shot transition rhythm and camera movement, and generate corresponding shot scheduling instructions; The voice-over, motion sequences, and camera movement instructions are aligned and integrated on a unified timeline to output a multimodal composite content package for video rendering or real-time playback.

8. The multimedia production method for digital humans according to claim 7, characterized in that, When targeting digital avatars at the presenter level, the process also includes optimizing voice-over and lip-syncing: The system receives the text to be broadcast, performs deep semantic analysis on the text, identifies and marks polyphonic characters, ambiguous phrases and domain terms; based on semantic dependency relations and a preset host prosody rule library, it automatically generates prosodic text with hierarchical pause marks, stress positions and intonation curves. The prosodic text is input into the speech synthesis model, which binds the contextual features and standard pronunciation of polyphonic characters through an attention mechanism, and dynamically adjusts the inter-frame transition parameters according to hierarchical pause markers to generate a target speech signal and corresponding phoneme time sequence that conforms to the host's broadcasting habits. A multimodal encoder is used to extract the acoustic features of the target speech signal and the three-dimensional features of the digital human face benchmark model. Cross-modal feature fusion is performed through a deep feature perception model to generate enhanced lip-speech synchronization features. The phoneme time series and enhanced lip-sync features are input into a conditional generative adversarial network to complete the mouth shape sequence optimization. The target speech signal and the optimized lip-sync sequence are time-stamped and then output to the digital human rendering engine for the generation of multimodal synthetic content for presenter-level digital humans.

9. The multimedia production method for digital humans according to claim 1, characterized in that, The process of adjusting the specifications of the multimodal synthesized content package based on the target terminal type before distributing it to the corresponding target terminal specifically includes: Based on the terminal capability detection protocol, obtain the target terminal device's capabilities and software decoding support list; the target terminal includes at least small-screen mobile terminals and large-screen broadcast-grade equipment. Based on the target terminal's capabilities and the software decoding support list, the encoding format, resolution, bit rate, frame rate, and color space of the multimodal synthesized content package are adaptively converted to generate a broadcast file or data stream that precisely matches the target terminal.

10. The multimedia production method for digital humans according to claim 1, characterized in that, It also includes collecting full-link data after broadcasting, performing data analysis to generate optimization suggestions, and feeding these optimization suggestions back to the preceding stages to achieve full-process iteration.

Images