Real-time video processing method and system based on artificial intelligence

Abstract

The invention discloses a real-time video processing method and system based on artificial intelligence, and belongs to the technical field of computers. Through deep fusion of multi-source information and a regional adaptive coding mechanism, the problem of collaboration between content generation and network resource allocation in a high-interactivity live broadcast scene is solved. In the feature extraction link, video, audio and text streams are synchronously processed, so that three-dimensional perception of a live broadcast scene is realized; in the multi-mode driving instruction set generation process, audio event features and text interaction features are subjected to semantic association, so that content generation and real-time interaction are highly synchronized, and response delay caused by information splitting in a traditional scheme is avoided; according to the regional differentiation coding processing, resource allocation is dynamically adjusted according to the regional importance of visual semantic feature recognition and the network state, and the coding quality of a digital human region is preferentially guaranteed, so that the stability of core visual elements is maintained under the network fluctuation condition, and the intelligent level of real-time video processing and the user experience are improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a real-time video processing method and system based on artificial intelligence. Background Technology

[0002] Real-time video processing technology plays an increasingly important role in interactive media scenarios such as live streaming and video conferencing. Existing solutions can typically achieve certain visual enhancements, such as background blurring or simple texture overlay. However, these methods fall short when dealing with dynamic and highly interactive live streaming scenarios.

[0003] Therefore, there is a need in the field for a smarter video processing technology that can intelligently and differentially allocate encoding resources based on the semantic importance of the image content and network status, thereby providing users with a smoother and more immersive visual experience under various conditions. Summary of the Invention

[0004] This application provides a real-time video processing method and system based on artificial intelligence, which can intelligently and differentially allocate encoding resources according to the semantic importance of the screen content and the network status, thereby providing users with a smoother and more immersive visual experience under various conditions. The technical solution is as follows: On the one hand, an artificial intelligence-based real-time video processing method is provided, the method comprising: Feature extraction is performed on the original video stream, audio stream, and bullet screen text in the live streaming scene to obtain the visual semantic features of the original video stream, the audio event features of the audio stream, and the text interaction features of the bullet screen text. Based on the audio event features and text interaction features, a multimodal driving instruction set is generated, which includes background generation instructions and digital human activation instructions. Based on the visual semantic features and the multimodal driving instruction set, the original video stream is subjected to real-time enhancement processing to obtain an enhanced video stream. The real-time enhancement processing includes background replacement based on the text interaction features and digital human generation based on the audio event features. Based on the visual semantic features, the digital human activation instructions, and the real-time network status, the enhanced video stream is processed by an adaptive encoder to perform regional differential encoding, and the target video stream is output. During the regional differential encoding process, more encoding resources are allocated to the visual regions containing the digital human.

[0005] On the one hand, an artificial intelligence-based real-time video processing system is provided, the system comprising: The feature extraction module is used to extract features from the original video stream, audio stream, and bullet screen text in the live streaming scene, and obtain the visual semantic features of the original video stream, the audio event features of the audio stream, and the text interaction features of the bullet screen text. The generation module is used to generate a multimodal driving instruction set based on the audio event features and text interaction features. The multimodal driving instruction set includes background generation instructions and digital human activation instructions. An enhancement module is used to perform real-time enhancement processing on the original video stream based on the visual semantic features and the multimodal driving instruction set to obtain an enhanced video stream. The real-time enhancement processing includes background replacement based on the text interaction features and digital human generation based on the audio event features. The encoding module is used to perform regional differential encoding processing on the enhanced video stream through an adaptive encoder based on the visual semantic features, the digital human activation instructions, and the real-time network status, and output the target video stream. In the regional differential encoding process, more encoding resources are allocated to the visual regions containing the digital human.

[0006] On one hand, a computer device is provided, the computer device including one or more processors and one or more memories, the one or more memories storing at least one computer program, the computer program being loaded and executed by the one or more processors to implement the artificial intelligence-based real-time video processing method.

[0007] On one hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the artificial intelligence-based real-time video processing method.

[0008] On the one hand, a computer program product or computer program is provided, which includes program code stored in a computer-readable storage medium. The processor of a computer device reads the program code from the computer-readable storage medium and executes the program code, causing the computer device to perform the above-mentioned real-time video processing method based on artificial intelligence. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1This is a schematic diagram of the implementation environment of a real-time video processing method based on artificial intelligence provided in an embodiment of this application; Figure 2 This is a flowchart of a real-time video processing method based on artificial intelligence provided in an embodiment of this application; Figure 3 This is a partial flowchart of a real-time video processing method based on artificial intelligence provided in an embodiment of this application; Figure 4 This is a partial flowchart of another real-time video processing method based on artificial intelligence provided in an embodiment of this application; Figure 5 This is a partial flowchart of another real-time video processing method based on artificial intelligence provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of a real-time video processing system based on artificial intelligence provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0012] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first", "second", and "n", nor are there any restrictions on the quantity or execution order.

[0013] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain better results.

[0014] Machine Learning (ML) is a multidisciplinary field that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge sub-models to continuously improve their performance.

[0015] Attention weights represent the importance of a piece of data during training or prediction. Importance indicates the magnitude of the influence of input data on output data. Data with high importance corresponds to higher attention weights, while data with low importance corresponds to lower attention weights. The importance of data varies in different scenarios, and training the model to assign attention weights is essentially the process of determining data importance.

[0016] Digital human: refers to a virtual character model created through technologies such as computer graphics and artificial intelligence. In the embodiments of this application, the digital human is not a pre-rendered static image, but a virtual entity that can be activated in real time based on audio events (such as specific voice commands) or interactive text (such as bullet comments), and can dynamically interact with real people or the environment in the video stream according to preset behavioral parameters (such as action sequences and interaction levels).

[0017] An adaptive encoder is a video encoder that can dynamically adjust its encoding parameters based on the characteristics of the input content and external conditions (such as network status). In the embodiments of this application, it does not uniformly encode the entire frame, but rather receives instructions from upstream processing (such as region segmentation results and allocated target bitrates), and uses differentiated quantization parameters, frame rates, etc., for different visual regions (such as key regions and secondary key regions of a digital human), thereby achieving optimal allocation of encoding resources.

[0018] Acoustic event detection and speech content parsing: These are two techniques for parallel processing of audio streams. Acoustic event detection aims to identify non-speech acoustic event types in the audio (such as applause, laughter, specific sound effects) to determine what event occurred; speech content parsing focuses on recognizing speech signals and converting them into text to understand their semantic content, i.e., what was said. The combination of these two techniques provides the system with more comprehensive audio contextual information.

[0019] Real-time keyword extraction and intent analysis: This is a technology for rapidly understanding interactive text streams such as bullet comments. Real-time keyword extraction quickly filters out core words that represent the current scene or topic from massive, fast-moving text streams. Intent analysis, based on this, further determines the purpose or intent behind the user's posting of these texts (such as asking questions, praising, or requesting content), thereby understanding what the user wants to do.

[0020] Multi-head attention encoder: A neural network architecture whose core mechanism is to use multiple parallel attention heads to process different aspects of the input data simultaneously. In the embodiments of this application, it is used for audio feature processing. Different attention heads can focus on the patterns of acoustic events (such as burstiness, spectral features) and the semantic patterns of speech content, respectively, thereby extracting richer and more accurate heterogeneous features from the same audio segment.

[0021] Real-time clustering: a technique for grouping dynamically flowing data online without pre-labeling. In this text processing, it is applied to quickly group lexical-level features generated in real time, automatically merging semantically similar words into the same set (i.e., semantic clusters), thereby discovering current hot topics of discussion from discrete words.

[0022] Intent evolution analysis results refer to the conclusions drawn about the changing trends of user interaction intents over a continuous time period. It not only reflects the dominant intent at the current moment but also describes the changes in the intensity and confidence of different intents over time, thereby capturing the dynamic migration process of user group attention.

[0023] Semantic clusters: Lexical groups formed through real-time clustering techniques. Words within the same semantic cluster share a high degree of semantic similarity or correlation, collectively pointing to a specific subtopic or concept. Semantic clusters are the basic units from which structured information is extracted from raw text.

[0024] Core vocabulary: The most representative words in a semantic cluster that best embody the core semantics of that cluster. It is usually selected from all words within the cluster through calculation (such as based on word frequency, position, or semantic vector distance) and can serve as a label or summary for that semantic cluster.

[0025] Dominant intent: At a specific moment, the most prominent intent, most likely representing the current interaction purpose of most users, is determined by comprehensively evaluating factors such as the strength and confidence of multiple intents. Subsequent system decisions (such as generating instructions) will prioritize responding to this dominant intent.

[0026] Compatibility Score: A quantitative evaluation value used to measure the degree of visual matching between the target background and the area to be replaced (such as background area, transition area) in the original video stream. Evaluation factors include edge sharpness, texture detail, spatial consistency, etc. A higher score indicates that the visual fusion effect after replacement is likely to be better.

[0027] Interaction level: A predefined or dynamically determined parameter used to quantify the expected intensity of interaction between the digital human and a real person or environment. A higher level generally means more complex behavior of the digital human and a closer spatial relationship with the real person (e.g., requiring closer distance and more frequent interaction), thus affecting the calculation of its integrated position and motion model.

[0028] Spatial registration: an image processing technique aimed at precisely aligning two or more image / video streams acquired at different times, from different perspectives, or under different conditions in space, giving them a unified coordinate system. In the embodiments of this application, it is used to ensure that the background replacement video stream and the digital human embedded video stream are pixel-level aligned before compositing, which is the basis for subsequent consistent lighting rendering and seamless compositing.

[0029] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by the respective parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0030] In traditional real-time video processing technologies, the processing of real-time video streams faces key challenges in multi-source information fusion and encoding resource allocation. Existing technologies process the raw video stream, audio stream, and bullet screen text independently, failing to establish a cross-modal feature association mechanism. This results in content generation failing to respond to dynamic interactive needs. At the same time, encoding strategies lack the ability to recognize the semantic importance of screen regions. When network conditions fluctuate, core visual elements suffer quality degradation due to insufficient bitrate resource allocation, directly affecting the continuity and immersion of the user's visual experience.

[0031] For example, in game live streaming scenarios, viewers send background replacement requests in real time via bullet comments, and the system needs to simultaneously parse voice commands in the audio stream to activate the digital human character. However, because the text interaction features of the bullet comments and the audio event features of the audio stream are not analyzed in conjunction, the background replacement operation cannot meet the timeliness requirements of the user's intent; when the anchor's voice triggers the digital human's behavior, the digital human generation experiences a response delay because the visual semantic features are not processed in conjunction with the audio event features; during periods of sudden drops in network bandwidth, the encoder does not allocate resources to the visual region containing the digital human, resulting in block effects or blurring in that region, while secondary background regions occupy redundant bitrate.

[0032] If this problem is not resolved, the fragmentation of multimodal information will lead to a continuous disconnect between content generation and user interaction intent. The cumulative effect of interaction delay will significantly deteriorate the system's real-time performance indicators. The uniformization of regional coding strategies will cause the loss of quality stability in semantically critical regions. The degradation of visual quality under network fluctuations will directly interrupt the user's immersive experience, ultimately reducing the overall reliability and availability of the video processing system (server).

[0033] Figure 1 This is a schematic diagram illustrating the implementation environment of a real-time video processing method based on artificial intelligence, as provided in an embodiment of this application. See also... Figure 1This implementation environment may include node 110 and server 140.

[0034] Node 110 is connected to server 140 via a wireless or wired network. Optionally, node 110 can be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these. Node 110 has an application installed and running that supports real-time video processing based on artificial intelligence.

[0035] Server 140 is a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms. Server 140 can provide background services for applications running on node 110.

[0036] This application proposes a real-time video processing method based on artificial intelligence, see [link to relevant documentation]. Figure 2 Taking the server as the executing entity as an example, the following steps are included.

[0037] 201. Extract features from the original video stream, audio stream, and bullet screen text in the live streaming scene to obtain the visual semantic features of the original video stream, the audio event features of the audio stream, and the text interaction features of the bullet screen text. 202. Based on audio event features and text interaction features, generate a multimodal driving instruction set, which includes background generation instructions and digital human activation instructions; 203. Based on visual semantic features and multimodal driving instruction set, the original video stream is enhanced in real time to obtain an enhanced video stream. The real-time enhancement process includes background replacement based on text interaction features and digital human generation based on audio event features. 204. Based on visual semantic features, digital human activation instructions, and real-time network status, an adaptive encoder is used to perform regional differential coding processing on the enhanced video stream to output the target video stream. In the regional differential coding process, more coding resources are allocated to the visual regions containing the digital human.

[0038] This application relates to an artificial intelligence-based real-time video processing method, in which feature extraction of the original video stream, audio stream, and bullet screen text in a live broadcast scene refers to the process of obtaining structured representations from multi-source input data. Specifically, the visual semantic features of the original video stream can be extracted as vector representations of object categories and spatial relationships in the image, which can be achieved using traditional computer vision algorithms, such as using convolutional neural networks for object detection and scene segmentation of keyframes, or matching and mapping based on manually annotated semantic tag libraries; the audio event features of the audio stream can be identified as quantitative indicators of predefined events in acoustic signals, which can be achieved using fixed threshold detection methods, such as detecting applause or laughter through energy mutation analysis, or using preset acoustic templates for pattern matching; the text interaction features of the bullet screen text can be parsed as semantic summaries of real-time user feedback, which can be achieved using basic natural language processing techniques, such as using a bag-of-words model to count high-frequency interactive words, or extracting combined features of emoticons and keywords based on a rule engine.

[0039] Furthermore, generating a multimodal driving instruction set based on audio event features and text interaction features refers to the process of generating control signals based on multimodal inputs. Background generation instructions can be configured as static resource call instructions, such as selecting corresponding materials from a background image library based on a preset keyword mapping table, or triggering background switching through a user-defined scene template. Digital human activation instructions can be implemented as behavior trigger signals, such as activating a preset digital human action sequence when a specific voice command is detected, or generating an activation signal based on a combination of continuously appearing interactive symbols in the bullet comments. Therefore, real-time enhancement processing of the original video stream based on visual semantic features and the multimodal driving instruction set refers to the operation of dynamically modifying the screen content. Background replacement based on text interaction features can be implemented as a screen area coverage operation, such as directly associating detected user preference keywords with a background material library for replacement, or selecting an appropriate background color tone based on the emotional polarity value of the bullet comments. Digital human generation based on audio event features can be implemented as a virtual element embedding process, such as calling a pre-rendered digital human model based on the voice command type, or dynamically adjusting the duration of the digital human's appearance based on the audio energy intensity.

[0040] Furthermore, region-differentiated encoding based on visual semantic features, digital human activation commands, and real-time network status refers to an encoding operation that implements non-uniform resource allocation for the video stream. Allocating more encoding resources to visual regions containing digital humans can be implemented as a dynamic bitrate scheduling mechanism. For example, high bitrate blocks can be fixedly allocated by preset region coordinates, or encoding parameters can be increased for dynamic regions based on the motion complexity detection results. This embodiment achieves the linkage between content generation and encoding strategies through the collaborative processing of multimodal information. The feature extraction stage uses non-deep learning methods to obtain multi-source features; the command generation stage relies on a predefined rule base to construct driving signals; the enhancement processing stage performs image modification based on simple semantic mapping; and the encoding stage achieves resource allocation through static region division and network status threshold judgment. Therefore, this application transforms multi-source inputs in live streaming scenarios into executable enhancement commands and dynamically adjusts resource allocation based on semantic importance during the encoding stage. This solves the problem of coordinating content generation and network adaptability in highly interactive scenarios without relying on complex models, avoiding the quality degradation of core visual elements caused by uniform encoding of the entire frame in traditional technologies.

[0041] In real-time video processing, feature extraction is first performed simultaneously on the original video stream, audio stream, and bullet screen text in the live broadcast scene. This yields visual semantic features of the original video stream, audio event features of the audio stream, and text interaction features of the bullet screen text. Visual semantic features capture spatial layout and semantic information within the video frame, audio event features identify key acoustic events and speech content, and text interaction features directly reflect the audience's real-time interaction intent. Further, based on semantic association analysis of audio event features and text interaction features, a multimodal driving instruction set is generated, including background generation instructions and digital human activation instructions. The generation process of this instruction set dynamically configures parameters by integrating audio semantic content and scene keywords, ensuring that the background generation instructions respond to user modification requests reflected in the bullet screen text, while simultaneously synchronizing the digital human activation instructions with the audio event types. Subsequently, real-time enhancement processing is applied to the original video stream using the visual semantic features and the multimodal driving instruction set. This includes performing background replacement based on text interaction features to match the audience's interaction intent, and triggering the digital human generation process based on audio event features to achieve natural behavioral responses. Based on this, by combining the semantic regions identified by visual semantic features, the key region range specified by the digital human activation command, and the bandwidth fluctuation trend of the real-time network status, the enhanced video stream is processed by an adaptive encoder to perform regional differential encoding. The visual regions of the digital human are given priority in being allocated more encoding resources, thereby ensuring the quality stability of core visual elements under limited bandwidth conditions.

[0042] As one implementation method, in a game live-streaming scenario, when viewers send keywords to switch to a sci-fi background via bullet comments and summon a digital human assistant with voice commands, the feature extraction stage can specifically employ a pre-trained convolutional neural network to process the video stream to obtain spatial layout information. Simultaneously, an acoustic model is used to analyze the audio stream to extract speech content features, and a natural language processing module extracts keywords from the bullet comment text in real time. Further, the multimodal driven instruction set generation stage semantically matches the bullet comment keyword "sci-fi background" with the voice command to summon the digital human assistant, generating a background generation instruction specifying the starry sky scene type and a digital human activation instruction setting the virtual assistant's behavioral parameters. The real-time enhancement processing stage accordingly replaces the background area of ​​the original video stream with a dynamic starry sky background and renders a 3D digital human model matching the voice command on the right side of the character. Finally, an adaptive encoder dynamically improves the quantization accuracy and frame rate of the digital human area during bandwidth fluctuations, ensuring that the clarity of virtual elements is unaffected by network conditions.

[0043] Therefore, this method solves the coordination problem of content generation and network resource allocation in highly interactive live streaming scenarios through deep fusion of multi-source information and a regional adaptive coding mechanism. Specifically, the feature extraction stage achieves three-dimensional perception of the live streaming scene by synchronously processing video, audio, and text streams, providing a multi-dimensional input basis for instruction generation; the multimodal-driven instruction set generation process semantically associates audio event features with text interaction features, making content generation and real-time interaction highly synchronized, avoiding the response delay caused by information fragmentation in traditional solutions; the regional differentiated coding processing dynamically adjusts resource allocation based on the regional importance of visual semantic feature recognition and network status, prioritizing the coding quality of the digital human region, thereby maintaining the stability of core visual elements under network fluctuation conditions and improving the intelligence level of real-time video processing and user experience.

[0044] Specifically, in some of the embodiments described above in this application, a multimodal driving instruction set is proposed to drive real-time video enhancement processing. However, in its implementation, directly generating the instruction set based on audio event features and text interaction features may fail to fully explore the deep semantic relationships of multimodal data, resulting in the instruction set generation lacking adaptability to dynamic changes in the scene and failing to capture the user's real-time interaction intentions and the contextual information of audio events, thus causing problems such as delays or mismatches in enhancement effects in highly interactive live streaming scenarios.

[0045] For this, see Figure 3 Taking the server as the executing entity as an example, the following steps are included.

[0046] 301. Based on audio event features, perform acoustic event detection and speech content parsing to obtain audio event types and audio semantic content; 302. Real-time keyword extraction and intent analysis based on text interaction features to obtain scene keywords and user interaction intent; 303. Generate an initial instruction sequence based on the audio event type and user interaction intent; 304. Based on audio semantic content and scene keywords, configure parameters and trigger conditions for the initial instruction sequence to generate a multimodal driving instruction set.

[0047] Acoustic event detection and speech content parsing refer to identifying specific event categories and their semantic connotations in an audio stream. This can be achieved using an end-to-end model based on the Transformer architecture or a traditional Hidden Markov Model. The aim is to separate the dual information of event type and semantic content from the time-series signal, avoiding the lack of contextual understanding caused by relying solely on a single event classification. Real-time keyword extraction and intent analysis refer to dynamically capturing the core semantic units and user behavior tendencies in the bullet screen text. This can be achieved using unsupervised clustering algorithms or sequence labeling networks. The aim is to adapt to the rapid changes in bullet screen information in live streaming scenarios, ensuring that key words are extracted and interpreted accurately. The timeliness of word and intent extraction; initial instruction sequence generation refers to integrating discrete audio event types and user interaction intents into an executable instruction framework, which can be achieved based on a semantic matching rule engine or a sequence-to-sequence generation model. The aim is to establish an initial instruction structure that integrates multimodal information and overcomes the limitations of single-modal driven scenario adaptation; parameter configuration and trigger condition configuration refer to dynamically adjusting instruction parameters and activation thresholds based on semantic association. This can be achieved using a similarity-based weight allocation mechanism or a dynamic threshold adjustment strategy. The aim is to enable instruction parameters and trigger conditions to adapt to changes in content and improve the responsiveness of the instruction set.

[0048] Specifically, the proposed solution first performs deep analysis on audio event features to obtain dual outputs of event type and semantic content. Simultaneously, it performs multi-granular analysis on text interaction features to extract scene keywords and user intent. These two types of information are then semantically coupled to generate an initial instruction sequence. Finally, the instruction sequence is fine-tuned based on the correlation strength between the audio semantic content and scene keywords. This phased processing mechanism ensures that audio event features and text interaction features retain their original semantic integrity during their respective analysis stages. Subsequently, dynamic adjustment of semantic correlation achieves the organic fusion of multimodal information, avoiding semantic distortion caused by directly splicing features. Since the semantic matching of audio event type and user interaction intent forms the basic framework of the instruction sequence, and the weighted fusion of audio semantic content and scene keywords drives the dynamic optimization of parameter configuration, the entire process forms a closed-loop optimization link from feature analysis to instruction generation, effectively addressing the dynamic interactive characteristics of multimodal data in live streaming scenarios.

[0049] As a specific implementation method, the solution of this application is implemented as follows: In the acoustic event detection and speech content parsing steps, a pre-trained VGGish model is used to extract audio features, which are then input into a bidirectional long short-term memory network for event classification and semantic decoding; in the real-time keyword extraction and intent analysis steps, the BERT encoder is used to vectorize the bullet screen text, and a candidate keyword set is generated through a hierarchical clustering algorithm; in the initial instruction sequence generation step, semantic matching is performed based on a predefined instruction template library, and target templates are selected and basic parameters are filled in according to the matching degree score; in the parameter configuration and trigger condition configuration steps, the cosine similarity between the audio semantic content vector and the scene keyword vector is calculated, and the background style parameters and digital human trigger threshold are dynamically adjusted according to the similarity results.

[0050] Through the above technical solutions, this application can systematically mine the deep semantic relationship between audio event features and text interaction features, enabling the generation process of multimodal driving instruction sets to have dynamic adaptability, thereby capturing users' real-time interaction intentions and audio context information in live streaming scenarios, effectively eliminating technical obstacles such as delays or mismatches in enhancement effects, and significantly improving the scenario fit and user experience smoothness of real-time video enhancement processing.

[0051] In practical applications, some of the embodiments described above in this application propose acoustic event detection and speech content parsing based on audio event features. However, in the implementation process, traditional methods cannot effectively integrate the short-term burst features and long-term context features of audio signals, and acoustic event detection and speech content parsing are isolated from each other. They fail to utilize the semantic correlation between the two for dynamic collaboration, resulting in insufficient accuracy and limited real-time performance of the parsing results in live streaming scenarios.

[0052] In response, this application further proposes steps for acoustic event detection and speech content parsing based on audio event features to obtain audio event types and audio semantic content, including: Multi-scale temporal analysis is performed on the features of audio events to extract a fused audio representation, which includes short-term burst features and long-term context features. The fused audio representation is input into a multi-head attention encoder, which focuses on acoustic event patterns and speech semantic patterns through different attention heads to generate enhanced feature representations. Based on enhanced feature representation, audio event types and audio semantic content are output synchronously through parallel event classifiers and semantic decoders. The input weights of the event classifier and semantic decoder are dynamically adjusted based on the semantic correlation between the audio event types and audio semantic content.

[0053] Multi-scale temporal analysis refers to the technique of extracting features from audio signals at different time scales. It can be implemented using wavelet transform or short-time Fourier transform combined with recurrent neural networks. Specifically, it can capture the instantaneous changes and long-term trends of audio signals, aiming to provide a complete feature base that takes into account temporal granularity and avoid feature omissions. Multi-head attention encoders refer to structures that utilize multiple attention mechanisms to process input features in parallel. They can be configured so that some attention heads focus on acoustic event pattern recognition, while others focus on speech semantic pattern parsing. Specifically, they can separate interference information and strengthen the representation of key patterns, aiming to improve the targeting and robustness of feature extraction. Parallel event classifiers and semantic decoders refer to event classification... The architecture of parallel operation of the event classifier and semantic decoder can be implemented by combining a convolutional neural network as the event classifier and a Transformer as the semantic decoder. Specifically, it can eliminate the latency bottleneck of traditional serial processing. Its purpose is to achieve synchronous output of event detection and semantic parsing, thereby improving real-time performance. Dynamic adjustment of input weights refers to the mechanism of adjusting the input ratio of the event classifier and semantic decoder in real time according to the semantic correlation between audio event type and audio semantic content. This can be achieved by calculating the semantic correlation score and adjusting the weight parameters accordingly. Specifically, it can be dynamically optimized based on scenarios such as the correlation strength between laughter and lighthearted topics. Its purpose is to enable event detection and semantic parsing to correct and complement each other, thereby enhancing the coordination and accuracy of the output results. Specifically, the proposed solution first performs multi-scale temporal analysis on audio event features to extract a fused audio representation containing short-term burst features and long-term contextual features, ensuring the capture of instantaneous changes and long-term trends in the audio signal. Then, the fused audio representation is input into a multi-head attention encoder, which focuses on acoustic event patterns and speech semantic patterns using different attention heads to generate enhanced feature representations, effectively separating interference and strengthening key information. Finally, based on the enhanced feature representations, the audio event type and audio semantic content are simultaneously output through a parallel event classifier and semantic decoder. The input weights are dynamically adjusted according to semantic relevance, allowing for mutual correction and enabling real-time joint parsing in live streaming scenarios.

[0054] As a preferred embodiment, the solution of this application is implemented as follows: In a live streaming scenario, audio event features are processed by a multi-scale temporal analysis module, which uses wavelet transform to extract short-term burst features and utilizes a long short-term memory network to capture long-term contextual features; the fused audio representation is fed into a multi-head attention encoder, where some attention heads are dedicated to processing acoustic event patterns such as applause detection, and other attention heads focus on speech semantic patterns such as dialogue content understanding; the event classifier uses a residual network structure to identify audio event types, and the semantic decoder parses audio semantic content based on a Transformer architecture. The two work in parallel and optimize the input based on semantic relevance through a dynamic weight adjustment mechanism, for example, when a laughter event is detected, the parsing weight for lighthearted topic semantics is enhanced.

[0055] Through the above scheme, this application effectively integrates the short-term burst features and long-term context features of audio signals in live streaming scenarios, realizes dynamic coordination between acoustic event detection and speech content parsing, improves the accuracy and real-time performance of audio parsing, and thus provides a reliable foundation for the generation of multimodal driving instruction sets.

[0056] In some of the embodiments described above in this application, real-time keyword extraction and intent analysis based on text interaction features are proposed to generate scene keywords and user interaction intents. However, in the implementation process, due to the characteristics of high real-time performance, strong noise interference and rapid semantic drift of bullet screen text in live streaming scenarios, traditional methods rely solely on single-level text analysis, which makes it difficult to accurately capture key information and dynamically respond to changes in user intent. This results in the extracted keywords being out of touch with real interaction needs and the intent analysis being lagging behind, which in turn affects the generation of multimodal driving instruction sets and ultimately weakens the real-time performance and user experience of video enhancement processing.

[0057] In response, this application further proposes real-time keyword extraction and intent analysis based on text interaction features to obtain scene keywords and user interaction intents, including: Multi-granular semantic parsing is performed on text interaction features to obtain lexical-level features and sentence-level semantic representations; Lexical features are input into a dynamic semantic network for real-time clustering to generate a set of candidate keywords with initial weight scores. At the same time, sentence-level semantic representations are input into a temporal attention module for intent evolution analysis to obtain intent evolution analysis results. The candidate keyword set and intent evolution analysis results are interacted at the feature level to generate optimized scenario keywords and user interaction intents.

[0058] Among them, multi-granularity semantic parsing refers to the simultaneous deconstruction of text at both micro and macro levels. This can be achieved using a hierarchical neural network architecture, such as extracting local lexical features through convolutional neural networks and combining them with bidirectional long short-term memory networks to capture sentence-level contextual dependencies. Lexical features refer to the vectorized representation of basic semantic units in the text, which can be generated using pre-trained word embedding models such as GloVe. Specifically, this involves mapping words or phrases in the comments to fixed-dimensional semantic vectors, aiming to identify the distribution patterns of high-frequency keywords. Sentence-level semantic representation refers to the semantic abstraction of complete sentences, which can be achieved using Transformer-based encoders to output sentence vectors. Specifically, this involves aggregating lexical features and modeling the internal logical relationships of sentences, aiming to effectively represent the overall intent expressed by the user. Dynamic semantic networks refer to... A semantic clustering structure adaptable to the dynamic changes of text streams can be implemented using online learning clustering algorithms such as streaming hierarchical clustering. Specifically, it incrementally updates the semantic cluster centers to cope with the continuous influx of bullet comments, aiming to filter noise interference and dynamically capture hot topics. The temporal attention module is an attention mechanism component that specifically handles the temporal dependencies of sequential data. It can be implemented using a Transformer decoder structure with positional encoding. Specifically, it focuses on the temporal correlation of historical interactions through a self-attention mechanism, aiming to track the dynamic evolution of user intent. Feature-level interaction refers to the process of fusing and calculating features from different sources in vector space. It can be implemented using a gated attention mechanism. Specifically, it calculates the semantic correlation strength and weightedly fuses multi-source information, aiming to ensure a close association between keywords and the current dominant intent.

[0059] This application's solution performs multi-granular semantic parsing on text interaction features, simultaneously acquiring lexical-level features and sentence-level semantic representations, providing a feature foundation that combines fine-grained detail with overall comprehensiveness for subsequent processing. Based on this, lexical-level features are input into a dynamic semantic network for real-time clustering to generate a candidate keyword set, while sentence-level semantic representations are input into a temporal attention module for intent evolution analysis. These two processes are executed in parallel to fully utilize computational resources and reduce processing latency. Finally, through feature-level interaction between the candidate keyword set and the intent evolution analysis results, keyword weights and intent information are fused to generate optimized scene keywords and user interaction intents. This processing flow, through multi-granular feature extraction, parallel processing, and feature-level fusion mechanisms, enables lexical-level features to identify high-frequency keyword distribution patterns, sentence-level semantic representations to effectively model contextual relationships, a dynamic semantic network to adaptively divide semantic clusters and quantify keyword importance, a temporal attention module to track the dynamic evolution of intent, and feature-level interaction to ensure semantic consistency between keywords and the dominant intent. This systematically solves the technical challenges posed by the high real-time requirements, strong noise interference, and rapid semantic drift of bullet screen text in live streaming scenarios. As a specific implementation method, the solution of this application is implemented as follows: In the implementation of the dynamic semantic network, an online hierarchical clustering algorithm based on a sliding time window is adopted. This algorithm can dynamically adjust the semantic cluster structure as new bullet comments are continuously input, and automatically calculate the initial weight score of keywords according to the size of the semantic clusters. In the implementation of the temporal attention module, a Transformer decoder with relative position encoding is adopted. This decoder focuses on the temporal dependencies of historical bullet comments through a self-attention mechanism, and outputs analysis results containing the dominant intent feature vector and confidence. In the feature-level interaction process, a cosine similarity-based attention mechanism is used to calculate the semantic association strength between candidate keywords and the dominant intent, and a comprehensive optimized weight is generated by combining the initial weight score and confidence, thereby re-ranking and filtering the candidate keywords, and finally outputting the highest-ranked keyword subset as scene keywords. Through the above technical solution, this application can accurately capture the key information of bullet comment text in live streaming scenarios and dynamically respond to changes in user intent, avoiding the problems of keywords being out of sync with real interaction needs and intent analysis being lagging behind, thereby improving the generation accuracy of multimodal driven instruction sets and enhancing the real-time performance and user experience of video enhancement processing.

[0060] Specifically, in some of the embodiments described above in this application, a feature-level interaction is proposed to generate optimized scene keywords and user interaction intentions by combining the candidate keyword set with the intent evolution analysis results. However, in its implementation, the initial weight score is determined only based on the semantic cluster size and fails to dynamically associate the semantic changes of the user's current dominant intent. This results in the keyword weights being unable to respond in real time to the rapid evolution of user intent in the live interactive scenario, causing the generated candidate keyword set to become disconnected from the real-time interactive focus, thus reducing the accuracy of subsequent scene keyword extraction and system responsiveness.

[0061] In response, this application further proposes to perform feature-level interaction between the clustering results and the intent evolution analysis results to generate a set of candidate keywords containing weighted scores, including: Extract the central words of each semantic cluster from the clustering results, and obtain the initial weight of each central word based on the size of the semantic cluster; Extract the semantic vector representation of the current dominant intent from the intent evolution analysis results; The semantic association strength between each central word and the current dominant intent is determined based on an attention mechanism; The initial weights of each central word are adjusted based on the semantic association strength to obtain an optimized weight score. Based on the optimized weight score, all central words are sorted in descending order, and the top N central words are selected to form the candidate keyword set, where N is a positive integer.

[0062] In practical applications, the central vocabulary of a semantic cluster refers to the representative words that represent the core semantics of the semantic cluster during text clustering. It can be achieved by using the words with the highest TF-IDF weights or word vector cluster centers, with the aim of providing a semantic core representation of the semantic cluster. The semantic vector representation of the current dominant intent refers to the technical means of encoding user interaction intent into high-dimensional vectors through deep learning models. It can be achieved by using embedding vectors generated by pre-trained language models such as BERT, with the aim of quantitatively expressing the mainstream interaction intent of users in the current live streaming scenario. The attention mechanism refers to the technical method of calculating the correlation between different elements. It can be achieved by using dot product attention or scaled dot product attention mechanisms, with the aim of dynamically evaluating the semantic correlation between the central vocabulary and the dominant intent. The semantic association strength refers to the quantitative index that measures the semantic similarity between the central vocabulary and the dominant intent. It can be calculated by cosine similarity or Euclidean distance, with the aim of providing a basis for weight adjustment. The optimized weight score refers to the comprehensive score calculated by combining the initial weights and the semantic association strength. It can be generated by weighted average or nonlinear combination, with the aim of making the keyword weights dynamically respond to changes in user intent. Specifically, this scheme first extracts the central words of each semantic cluster and their initial weights based on the size of the semantic cluster from the clustering results, providing a basic measure of keyword importance. Simultaneously, it extracts the semantic vector representation of the current dominant intent from the intent evolution analysis results, capturing user focus in real time. Then, it calculates the semantic association strength between each central word and the dominant intent based on an attention mechanism, quantifying the relevance of keywords in the current interactive context. Next, it dynamically adjusts the initial weights using the semantic association strength, generating optimized weight scores. Finally, it ranks the central words according to the optimized weight scores and selects the top-ranked words to form a candidate keyword set. The entire process forms a complete chain from basic weight acquisition, intent semantic capture, association calculation, dynamic weight adjustment to keyword selection, enabling the candidate keyword set to reflect the real-time focus in the current live interactive scenario. As a specific implementation method, the solution of this application is implemented as follows: When a large number of bullet comments such as lottery and benefits appear in the live broadcast scene, the system first clusters the bullet comment text through a dynamic semantic network to obtain semantic clusters including lottery (large in scale and high in initial weight) and question (small in scale and low in initial weight); at the same time, the temporal attention module analyzes and finds that the current dominant intent is to participate in the event, and its semantic vector representation is biased towards promotional interaction; subsequently, the attention mechanism calculates that the semantic association strength between lottery and the dominant intent is significantly higher than that of question; accordingly, the system significantly enhances the initial weight of lottery, while moderately adjusting the weight of question; finally, after sorting and screening, words such as lottery that are highly related to the current event are given priority to be included in the candidate keyword set to support the efficient generation of subsequent scene keywords.

[0063] Through the above scheme, this application realizes dynamic optimization of keyword weights, enabling the candidate keyword set to respond in real time to the rapid evolution of user intent in live interactive scenarios. It solves the problem of real-time interaction disconnect caused by static keyword weights, improves the accuracy of scenario keyword extraction and system response speed, and enhances the degree of subsequent multimodal driving instruction generation.

[0064] Specifically, in some of the embodiments described above in this application, a feature-level interaction is proposed to generate optimized scene keywords and user interaction intentions by combining the candidate keyword set with the intent evolution analysis results. However, in the implementation process, due to the lack of a deep feature fusion mechanism between the candidate keywords and the intent evolution analysis results, the generated scene keywords rely only on text statistical features and fail to fully associate with the dynamic evolution of the current dominant intent, resulting in a semantic disconnect between the keywords and the user's real-time interaction intent, which in turn affects the generation and real-time response capability of multimodal driven instructions.

[0065] To this end, this application further proposes feature-level interaction between the candidate keyword set and the intent evolution analysis results to generate optimized scenario keywords and user interaction intents, including: From the results of intent evolution analysis, extract the feature vector and confidence score of the current dominant intent; The relevance of each keyword in the candidate keyword set to the current dominant intent feature vector is evaluated by semantic similarity calculation. By integrating the initial weight scores, intent relevance, and confidence of the dominant intent of each keyword, a comprehensive optimized weight for each keyword is obtained. A comprehensive optimization weight is used to reorder and filter multiple keywords in the candidate keyword set, and the highest-ranking subset of keywords is output as the scene keywords. The semantic representation of scene keywords is input into the intent classification model to obtain the user's interaction intent.

[0066] Among them, the feature vector of the current dominant intent refers to the high-dimensional numerical semantic representation of the intent, which can be implemented using the hidden state output of a recurrent neural network or transformer architecture. Its purpose is to capture the deep semantic structure of the intent and use it as a benchmark anchor for subsequent interactions. The confidence score is a quantitative indicator of the reliability of the current dominant intent, which can be implemented using the maximum value in the classification probability distribution or confidence interval estimation based on historical data. Its purpose is to filter out low-reliability intents to avoid decision bias. The semantic similarity calculation is a measure of the semantic association strength between keywords and intent, which can use cosine similarity, Euclidean distance, or pre-training-based methods. The semantic matching function of the language model aims to overcome the limitations of traditional text statistical features and establish dynamic semantic associations. The comprehensive optimization weight refers to the weighted scoring that integrates multi-dimensional information. It can be implemented using linear weighted fusion, gating mechanisms, or attention networks. Its purpose is to coordinate the three factors of keyword importance, intent matching depth, and intent reliability to ensure that the weights dynamically adapt to changes in the scenario. The intent classification model is a discriminative model that maps keyword semantics to user interaction intent. It can be implemented using support vector machines, random forests, or fine-tuned pre-trained classifiers. Its purpose is to improve the robustness and accuracy of intent recognition. Specifically, this application's solution extracts the feature vector and confidence score of the current dominant intent from the intent evolution analysis results, establishing a high-confidence semantic core representation of the intent and providing a reliable benchmark for feature-level interaction. Based on this, it evaluates the dynamic correlation strength between candidate keywords and intent feature vectors through semantic similarity calculation, elevating keyword selection from static statistical features to the semantic matching level. Furthermore, it integrates the initial weight score, intent relevance, and dominant intent confidence to generate a comprehensive optimized weight, ensuring that the keyword scoring mechanism simultaneously considers text saliency, intent fit, and intent reliability. Subsequently, based on the comprehensive optimized weight, candidate keywords are reordered and filtered to ensure that the output subset of scene keywords closely matches the current dominant intent. Finally, the optimized scene keywords are input into the intent classification model, utilizing pure semantic representation to improve the recognition accuracy of user interaction intent. This process constructs a closed-loop feedback between intent evolution and keyword generation through a feature-level interaction mechanism, enabling scene keywords to dynamically respond to real-time changes in user intent in live streaming scenarios and solving the semantic disconnect problem.

[0067] As a preferred embodiment, the specific implementation of this application's solution is as follows: The intent evolution analysis result is generated by a long short-term memory network processing user bullet screen sequences. The output hidden state is converted into a feature vector of the current dominant intent through a fully connected layer, and a confidence score is output through a softmax layer. Semantic similarity calculation uses the cosine similarity function in the vector space model to compare the word embedding vectors of candidate keywords with the intent feature vectors. The comprehensive optimization weight is implemented through a learnable weight allocation module, which receives the initial weight score, intent relevance, and confidence score as input, and outputs normalized weight values ​​after nonlinear transformation. Keyword re-ranking uses a fast sorting algorithm to sort the comprehensive optimization weights in descending order, and selects the top-ranked keyword subset based on a preset threshold. The semantic representation of scene keywords is generated into a fixed-dimensional vector through pooling operations and then input into a fine-tuned BERT classification model to output the user interaction intent category.

[0068] Through the above solution, this application achieves dynamic matching between scene keywords and users' real-time interactive intentions, avoids deviations in multimodal driven instruction generation caused by semantic disconnect, and improves the real-time response capability and interaction accuracy of content enhancement in live streaming scenarios.

[0069] Specifically, in some of the embodiments described above in this application, an initial instruction sequence based on audio event type and user interaction intent is proposed to construct a multimodal driven instruction set. However, in its implementation, due to the dynamic interactive characteristics of live streaming scenarios, directly generating the initial instruction sequence can easily lead to low instruction template matching efficiency and insufficient scenario adaptability. Specifically, the template selection lacks quantitative evaluation standards, and it is impossible to dynamically distinguish the applicable priority of different templates. At the same time, the parameter filling process does not fully integrate real-time semantic information, resulting in instruction generation delay or disconnection from the actual scenario, which in turn affects the real-time performance and accuracy of subsequent background replacement and digital human generation.

[0070] In response, this application further proposes generating an initial driver instruction set based on audio event type and user interaction intent, including: The audio event type and user interaction intent are input into the instruction template library for template matching to obtain a set of basic instruction templates; Determine the matching score between each basic instruction template in the basic instruction template set and the current scene; Based on the matching score, the basic instruction templates in the basic instruction template set are prioritized to obtain multiple target basic instruction templates; Based on audio semantic content and scene keywords, parameter placeholders in multiple target basic instruction templates are filled in real time to obtain the initial driving instruction set.

[0071] The instruction template library is a collection of predefined instruction templates, which can be implemented using a relational database or in-memory cache structure. Its purpose is to provide a structured foundation for template retrieval, ensuring the efficiency and scalability of the template matching process. The matching score is an indicator that quantitatively evaluates the suitability of a basic instruction template for the current live streaming scenario. It can be determined based on semantic similarity calculation or rule-based matching algorithms, aiming to objectively reflect the applicability of the template and avoid biases caused by subjective judgment. Priority sorting refers to the process of arranging basic instruction templates in descending order according to the matching score. This can be implemented using a quicksort algorithm or a priority queue data structure, aiming to efficiently select the most matching template and reduce redundant calculations. Real-time filling refers to the operation of dynamically filling audio semantic content and scene keywords into parameter placeholders. This can be accomplished using template engine technology or string interpolation methods, aiming to ensure that the generated instruction content is highly consistent with real-time semantic information and improve the scene adaptability of the instructions. Specifically, the proposed solution uses audio event types and user interaction intentions as input for template matching to quickly obtain a set of basic instruction templates relevant to the current live streaming scenario. Then, by calculating the matching score of each template, its suitability for the scenario is quantitatively evaluated, providing an objective basis for priority ranking. Based on the score ranking, high-matching target basic instruction templates are selected. Finally, audio semantic content and scenario keywords are integrated to fill parameter placeholders in real time, generating a semantically coherent initial driving instruction set. This process, through structured processing and quantitative evaluation, achieves dynamic optimization of the instruction generation process, ensuring the priority selection of high-matching templates and semantic consistency in parameter filling, thereby solving the core problems of low template matching efficiency and insufficient scenario adaptability. As a specific implementation method, the solution of this application is implemented as follows: In a game live streaming scenario, when the system detects that the audio event type is victory cheer and the user's interaction intention is celebration, this information is input into the instruction template library for matching to obtain a basic instruction template set containing victory animation and special effects display; the matching degree score of each template is calculated, and it is determined that the matching degree of the victory animation template is higher than that of the special effects display template; based on the matching degree score, the victory animation is selected as the target basic instruction template; subsequently, using the specific game name and scene keyword "exciting" in the audio semantic content, the parameter placeholders in the template are filled to generate an initial driving instruction set such as "Play victory animation: game name, special effects intensity: high".

[0072] Through the above scheme, this application effectively realizes the quantitative evaluation and dynamic priority management of the instruction template matching process, avoids the problem of parameter filling being disconnected from real-time semantic information, improves the real-time performance and accuracy of the initial driving instruction set generation, and provides a reliable technical foundation for subsequent background replacement and digital human generation.

[0073] Specifically, in some of the embodiments described above in this application, a multimodal driving instruction set is generated by configuring parameters and setting conditions for the initial driving instruction set. However, in its implementation, the parameter configuration lacks in-depth utilization of the dynamic semantic association between audio semantic content and scene keywords, resulting in the parameter setting of the background generation instruction being too static and unable to adapt to real-time scene changes. At the same time, the triggering conditions of the digital human activation instruction are fixed and it is difficult to dynamically adjust the semantic triggering threshold according to the evolution of user interaction intent. This causes the generated instruction set to be out of touch with the actual needs of the live streaming scene, ultimately affecting the performance of video enhancement processing and user experience.

[0074] In response, this application further proposes a process for configuring parameters and setting conditions for the initial driving instruction set based on audio semantic content and scene keywords to obtain a multimodal driving instruction set, including the following steps: The semantic relevance between audio semantic content and scene keywords is determined by an attention mechanism, and the audio semantic content and scene keywords are weighted and fused based on the semantic relevance to obtain a fused semantic vector; Based on the fused semantic vector, the background generation instructions in the initial instruction sequence are configured with parameters, including background style parameters and scene element parameters. The background style parameters are determined based on the matching results of scene keywords and preset style library, and the scene element parameters are determined based on scene-related entity information in the audio semantic content. Based on the fused semantic vector, the triggering conditions for the digital human activation instructions in the initial instruction sequence are configured. The triggering condition configuration includes setting the semantic triggering threshold for the appearance of the digital human. The semantic triggering threshold is dynamically adjusted based on the semantic association strength between scene keywords and audio semantic content. The initial instruction sequence after completing parameter configuration and trigger condition configuration will be output as a multimodal drive instruction set.

[0075] In practical applications, the attention mechanism refers to a computational model used to dynamically quantify the correlation between input elements. It can be implemented using the multi-head attention mechanism in the Transformer architecture or dot-product-based attention calculation. Its purpose is to capture the non-linear semantic dependency between audio semantic content and scene keywords, avoiding semantic distortion caused by fixed weights in traditional methods. Semantic relevance can be understood as a numerical indicator reflecting the real-time matching degree between audio semantic content and scene keywords. It can be calculated based on cosine similarity or dynamically weighted similarity, aiming to provide adaptive weighting for subsequent weighted fusion. Specifically, weighted fusion refers to the process of combining audio semantic content and scene keywords proportionally according to their semantic relevance. This can be implemented using gating mechanisms or linear interpolation methods, aiming to generate a fused language that adaptively reflects the core semantic focus of the current scene. Semantic vectors; background style parameters can be understood as configuration items that control the visual presentation characteristics of the background. They can be dynamically determined based on the semantic matching results between scene keywords and a preset style library. For example, style templates such as technological feel and natural scenery can be retrieved through vector similarity, with the aim of keeping the background style consistent with the semantics of the live broadcast scene; scene element parameters refer to configuration parameters that define dynamic elements in the background. They can extract scene-related entity information from the audio semantic content. For example, entities such as racing cars and fireworks can be captured through named entity recognition technology, with the aim of ensuring semantic coherence between background elements and audio content; semantic trigger threshold can be understood as the semantic sensitivity threshold for activating the digital human. It can be dynamically adjusted based on the correlation strength between scene keywords and audio semantic content. For example, when the correlation strength is high, the threshold can be lowered to improve the response speed, with the aim of matching the timing of digital human activation with the evolution trend of user interaction intent. Specifically, the proposed solution first dynamically calculates the semantic relevance between audio semantic content and scene keywords through an attention mechanism. This relevance serves as a weighting factor to guide the weighted fusion process, generating a fused semantic vector that adaptively reflects the core semantic focus of the current scene. Subsequently, the fused semantic vector is uniformly applied to the configuration process of background generation instructions and digital human activation instructions. The background style parameter dynamically determines the background presentation style through semantic matching between scene keywords and a preset style library, while the scene element parameter extracts relevant entity information from the audio semantic content to fill background elements. Simultaneously, the semantic trigger threshold adjusts the activation conditions in real time based on the semantic relevance strength. Finally, the initial instruction sequence after configuration is output as a multimodal driven instruction set, ensuring that background replacement and digital human generation are semantically coordinated, thereby enabling the instruction set to respond to dynamic changes in the live streaming scene.

[0076] As a preferred embodiment, the solution of this application is implemented as follows: When the audio semantic content in the live broadcast scene indicates that the racing competition has ended and the scene keyword is cheering, the attention mechanism calculates that the semantic correlation between the two is high, and the semantic vector is fused to indicate the theme of victory celebration; the background style parameters are matched with a dynamic celebration style template, and the scene element parameters extract entity information such as ribbons and trophies and embed them into the background; at the same time, the semantic trigger threshold is dynamically reduced according to the high correlation strength, so that the digital human is quickly activated and makes a celebration action, and the digital human's action parameters are semantically synchronized with the background elements.

[0077] Through the above solution, this application enables the parameter configuration of the background generation instruction to dynamically adjust the background style and elements according to the real-time scene, and the triggering conditions of the digital human activation instruction can automatically optimize the activation timing as the user's interaction intention evolves. This solves the problem of the instruction set being out of sync with the needs of the live streaming scene, and improves the performance of video enhancement processing and user experience.

[0078] In some of the embodiments described above in this application, a real-time enhancement processing of the original video stream based on visual semantic features and multimodal driving instruction set is proposed to generate an enhanced video stream. However, in its implementation, the background replacement and digital human generation lack a matching mechanism for spatial layout and human posture, as well as a consistency guarantee for screen synchronization and lighting rendering. This results in visual defects in the enhanced video stream, such as harsh edges of background replacement, disconnect between the position of the digital human and the human's movements, and inconsistent lighting, which affect the immersion and smoothness of the live streaming scene.

[0079] To address this, this application further proposes a technical solution for real-time enhancement of the original video stream based on visual semantic features and a multimodal driving instruction set, resulting in an enhanced video stream. (See [link to relevant documentation]). Figure 4 Taking the server as the executing entity as an example, the following steps are included.

[0080] 401. Analyze the multimodal driving instruction set to obtain the target background type indicated by the background generation instruction and the digital human behavior parameters indicated by the digital human activation instruction; 402. Match the spatial layout information in the visual semantic features with the target background type to obtain a spatial mask for background replacement. At the same time, couple the human posture information in the visual semantic features with the digital human behavior parameters to obtain the integrated position and state of the digital human in the video stream. 403. Replace the background region of the original video stream based on the spatial mask to obtain a background-replaced video stream, and render and embed a digital human model in the original video stream based on the integrated position and state to obtain a digital human embedded video stream. 404. Perform image synchronization and lighting rendering consistency processing on the background replacement video stream and the digital human embedded video stream, and output an enhanced video stream.

[0081] In practical applications, spatial mask refers to a pixel-level probability distribution map that defines the background replacement area. It can be implemented using a soft boundary mask based on semantic segmentation or a binary mask based on edge detection. The purpose is to separate the foreground subject from the background area and avoid the abrupt edge problems caused by traditional hard boundary replacement. Digital human behavior parameters refer to a set of parameters that define the sequence of digital human actions, facial expression changes, and interaction intensity. These can include preset action library indexes and dynamic adjustment coefficients, etc., to match the digital human behavior with the real-time interactive needs of the live streaming scene. Specifically, integrated position and state refer to the spatial coordinates and dynamic posture information of the digital human in the video frame. This can be determined based on human motion trajectory prediction or keyframe interpolation technology, to ensure the coordination and spatial consistency of the digital human's actions with those of a real person.

[0082] The proposed solution first parses the multimodal driving instruction set to obtain control parameters for background replacement and digital human generation. Based on the spatial layout information in visual semantic features, it dynamically matches the target background type to generate a spatial mask reflecting the scene structure, thereby solving the problem of inaccurate background region definition. Simultaneously, it couples the human's posture information with the digital human's behavior parameters in real time, constructs a relative motion model based on the human's activity space boundary and motion vectors, and dynamically corrects the integrated position and state of the digital human to eliminate the phenomenon of disjointed movements. On this basis, it performs background replacement and digital human embedding processing separately to form two independent video streams. Through spatial registration and illumination feature extraction in a unified coordinate space, it performs geometrically perceptual feathering processing on the edge region of the digital human, so that the background replacement video stream and the digital human embedding video stream remain synchronized in the spatial, temporal, and illumination dimensions, and finally outputs a visually coherent enhanced video stream. As a specific implementation method, in a game live streaming scenario, when the multimodal driving instruction set instructs a sci-fi style background and an interactive digital human, the system parses the target background type as a dynamic starry sky scene, and the digital human's behavior parameters include waving interaction actions. A spatial mask generated based on spatial layout information distinguishes the streamer from the background area, preserving detailed edges such as hair. The streamer's posture information is coupled with the waving action parameters, and a collision detection algorithm determines the initial integration position of the digital human on the left side of the screen. The digital human's offset is corrected in real time based on the streamer's head motion vector. After completing background replacement and digital human rendering, the lighting direction and intensity of the background area are extracted in a unified coordinate space, and real-time lighting correction is performed on the digital human model, allowing the lighting effects of the digital human and the new background to blend naturally. This technical solution solves the problem of harsh edges in background replacement, avoids the disconnect between the digital human's position and the character's actions, eliminates visual defects caused by inconsistent lighting, and enhances the visual immersion and smoothness of the enhanced video stream in the live streaming scenario.

[0083] Specifically, in some of the embodiments described above in this application, a technical means for background replacement based on visual semantic features and multimodal driving instruction sets is proposed. However, in its implementation, simple spatial mask generation methods are difficult to adapt to the complex and ever-changing background requirements in dynamic live streaming scenarios. Specifically, when the target background type does not match the spatial layout information of the original video, traditional mask processing is prone to edge artifacts, texture breaks, or visual separation between the background and foreground. Especially in live streaming scenarios, background replacement driven by user interaction intent and audio event features needs to meet high real-time requirements, but coarse mask generation cannot adaptively adjust the edge sharpness and texture details according to the specific requirements of the target background, resulting in abrupt transition areas and insufficient compatibility, ultimately affecting the immersiveness and smoothness of the enhanced video stream.

[0084] To address this, this application further proposes the method of matching the spatial layout information in the visual semantic features with the target background type to obtain a spatial mask for background replacement, including: The spatial layout information is analyzed to obtain the foreground main area, background area and transition area, and the boundary features and texture complexity of each area are extracted; Based on the requirements of the target background type, a compatibility score is determined between the background area and the transition area and the target background. The requirements include the target background's requirements for edge sharpness and texture detail. Based on the compatibility score and the boundary features, an initial spatial mask is generated, and the transition region is feathered. Based on the texture complexity and the spatial consistency requirements of the target background type, edge optimization and noise suppression are performed on the initial spatial mask to obtain the spatial mask.

[0085] Scene parsing refers to the process of semantically dividing video frames, which can be achieved using image segmentation techniques such as fully convolutional neural networks or conditional random fields. Its purpose is to identify the boundary structures of foreground subjects, backgrounds, and transition regions. Boundary features are quantitative indicators describing the geometric attributes and texture continuity of region boundaries. These can include features such as edge gradient magnitude and curvature change rate, which can be extracted using Canny edge detection or phase consistency algorithms. Their purpose is to characterize the sharpness and complexity of region boundaries. Compatibility scoring is a quantitative indicator evaluating the degree of matching between the target background and the original region. It can be calculated based on the similarity between the target background's preset requirements and the original region's features, aiming to dynamically adapt to different backgrounds. The requirements for edges and textures based on the type; the required characteristics can be understood as the visual adaptation requirements implicit in the target background type, which are specifically manifested in the definition of the edge sharpness threshold and the degree of texture detail preservation. For example, an abstract style background may require lower edge sharpness but higher texture smoothness, with the aim of guiding the calculation of compatibility scores; transition region feathering refers to the operation of gradually blurring the mask edges, which can be implemented based on Gaussian filtering or bilateral filtering, with the aim of making the transition between the foreground and background more natural; edge optimization and noise suppression refer to the steps of refining the initial mask, which can be implemented using edge enhancement filters or nonlocal mean denoising algorithms, with the aim of improving the edge quality of the mask and reducing visual artifacts.

[0086] Specifically, the proposed solution first analyzes the spatial layout information to obtain the division of the foreground main body region, background region, and transition region, and extracts the boundary features and texture complexity of each region, providing a structured basis for subsequent processing. Next, based on the requirements of the target background type, the system evaluates the compatibility of the background and transition regions with the target background. These requirements clearly define the adaptation requirements for edge sharpness and texture details, enabling the compatibility score to dynamically respond to the specific needs of different background types. Subsequently, the system uses the compatibility score and boundary features to generate an initial spatial mask and performs adaptive feathering on the transition region. The feathering intensity is dynamically adjusted based on the boundary features and compatibility score results to ensure the natural blending of the mask edges with the target background. Finally, the system combines texture complexity and spatial consistency requirements to perform edge optimization and noise suppression on the initial spatial mask. By identifying and specifically optimizing areas prone to noise, the mask retains key edges while suppressing visual interference. This series of steps forms a closed-loop optimization process, ensuring that the generated spatial mask can adapt to the complex and ever-changing background requirements of dynamic live streaming scenarios.

[0087] As a preferred embodiment, the solution of this application is implemented as follows: scene parsing uses a semantic segmentation model based on the U-Net architecture to output pixel-level region classification results; compatibility scoring is achieved by calculating the structural similarity index and edge sharpness index between the target background texture and the original background region; during the initial spatial mask generation, a linear decay function is applied to regions with compatibility scores below a threshold to construct transition masks; feathering processing uses an adaptive Gaussian kernel based on boundary curvature, and regions with larger curvature use smaller standard deviation parameters; in the edge optimization stage, nonlocal mean filtering is applied to suppress noise in regions with low texture complexity, while edge detail features are preserved in regions with high texture complexity.

[0088] Through the above scheme, this application can adaptively adjust the spatial mask generation process according to the specific requirements of the target background type for edge sharpness and texture details, effectively avoiding edge artifacts and texture breaks in background replacement, and enabling natural blending of transition areas, thereby improving the visual coherence and immersion of the enhanced video stream, which is especially suitable for highly interactive real-time live streaming scenarios.

[0089] In some of the embodiments described above in this application, the human posture information in visual semantic features is coupled with the digital human behavior parameters. However, in the implementation process, the existing methods have difficulty capturing the dynamic change trend and real-time motion characteristics of the human posture, resulting in the initial position setting of the digital human lacking spatial rationality, failing to effectively avoid conflicts with the human's activity area, and the digital human behavior is not synchronized with the human's movement trajectory, causing the digital human to appear stiff, positional drift, or disconnected from interaction in the live video, which seriously affects the visual immersion and user experience.

[0090] To this end, this application further proposes a step to couple the human posture information in visual semantic features with the digital human behavior parameters to obtain the integrated position and state of the digital human in the video stream, including: Extract the 3D coordinates and motion vectors of the key points from the character's posture information; scale the 3D coordinates using the resolution of the video stream, and calculate the velocity parameters of the motion vectors based on the frame rate of the video stream to determine the boundaries of the character's activity space; By utilizing the digital human type and preset action sequence in the digital human behavior parameters, the initial integration position of the digital human is determined within the activity space boundary using a collision detection algorithm; By integrating motion vectors and interaction levels in digital human behavior parameters, a relative motion model between the digital human and the real person is constructed. The relative motion model is used to output the dynamic position offset of the digital human relative to the real person. Based on the dynamic position offset output by the relative motion model, the initial integrated position is corrected in real time, and the preset action sequence is driven to keep synchronized with the movement trajectory of the person, generating the integrated position and state of the digital human in the video stream.

[0091] Among them, the 3D coordinates of the joints refer to the position coordinates of key parts of the human body in 3D space. They can be obtained using a monocular pose estimation algorithm based on deep learning or a depth sensor, with the aim of representing the human's posture. The motion vector can be understood as a vector describing the direction and velocity of the human's joint movement. Specifically, it can be obtained by calculating the coordinate difference between consecutive frames, with the aim of capturing the real-time movement trend of the human. The resolution of the video stream is used to scale the 3D coordinates, which means adjusting the coordinate units according to the size of the video screen. It can be dynamically calculated based on the video resolution parameter, with the aim of adapting the coordinates to the size of the video screen. The frame rate of the video stream is used to calculate the velocity parameter of the motion vector, which means converting displacement into velocity according to the frame interval time. It can use the reciprocal of the frame rate as the time unit, with the aim of synchronizing motion analysis with video timing. The activity space boundary can be understood as the geometric boundary of the human's range of movement. It can be determined by the maximum bounding rectangle or convex hull algorithm of the joint coordinates, with the aim of defining a safe area to avoid digital human conflicts. The digital human type refers to the appearance and body characteristics of the digital human, which can include cartoon characters or realistic ones. The categories of models are designed to adapt to different scenario requirements; preset action sequences refer to predefined behavioral animations of the digital human, which can be stored as keyframe sequences or skeletal animation data, aiming to provide an executable action basis; collision detection algorithms can be understood as algorithms that detect overlap between the digital human and the human's regions, which can be implemented using bounding box collision or pixel-level collision detection, aiming to ensure the spatial rationality of the initial position; interaction level refers to the intensity level of interaction between the user and the digital human, which can be dynamically determined based on user input or scene context, aiming to adjust the intensity of interaction; relative motion model refers to the mathematical model describing the digital human's motion relative to the human, which can be built based on kinematic equations or machine learning models, aiming to output dynamic position offsets to maintain synchronization; dynamic position offset refers to the real-time adjustment of the digital human's position relative to the human, which can be represented as a two-dimensional or three-dimensional vector, aiming to correct the initial position to cope with the human's movement; keeping the preset action sequence synchronized with the human's movement trajectory means adjusting the rhythm of the digital human's movements to match the human's movements, which can be achieved through time scaling or phase adjustment, aiming to ensure action coordination. Specifically, the proposed solution first extracts the 3D coordinates and motion vectors of key points from the character's posture information. Then, it scales the coordinates using the video stream's resolution and calculates speed parameters based on the frame rate to determine the character's activity space boundaries, providing a dynamic safe zone for digital human placement. Based on this, using the digital human type and a preset action sequence, a collision detection algorithm is used within the activity space boundaries to determine the initial integration position, ensuring a reasonable position and avoiding conflicts. Furthermore, a relative motion model is constructed by fusing motion vectors and interaction levels. This model outputs a dynamic position offset for real-time correction of the initial position. Finally, the preset action sequence is synchronized with the character's movement trajectory based on the offset to generate the integration position and state.The entire process, through the organic integration of dynamic boundary determination, collision detection, relative motion modeling, and synchronous driving, achieves the integration of digital human position and motion coordination, resolving positional conflicts and synchronization issues. As a specific implementation method, the solution of this application is implemented as follows: In a live streaming scenario, when the host waves, the system extracts the three-dimensional coordinates of the hand joints from the video stream, scales the coordinates to the screen size according to the 1080p video resolution, calculates the velocity parameters of the motion vector based on a 30fps frame rate, and determines the boundary of the character's activity space as the central area of ​​the screen. Subsequently, based on the digital human type being a cartoon assistant and a preset waving motion sequence, the initial integrated position of the digital human is determined within the activity space boundary using a bounding box collision detection algorithm, placing it in the safe area to the right of the host. Furthermore, by fusing the hand motion vector with a high level of interaction, a relative motion model is constructed to output dynamic position offsets, and the initial position is corrected in real time, ensuring the digital human maintains an interactive posture synchronized with the host's waving. Finally, the preset motion sequence is driven to match the character's motion trajectory, generating a natural and smooth integrated digital human effect.

[0092] Through the above solution, this application can capture the dynamic changes in the posture of the person, ensure the spatial rationality of the initial position of the digital human, effectively avoid conflicts with the activity area of ​​the person, and realize the synchronization of the behavior of the digital human with the movement trajectory of the person, thereby eliminating the phenomenon of stiffness, position drift or disconnection of the digital human in the live video, and significantly improving the visual immersion and user experience.

[0093] Specifically, in some of the embodiments described above in this application, a method is proposed to perform screen synchronization and lighting rendering consistency processing on the background replacement video stream and the digital human embedded video stream in order to output an enhanced video stream. However, in its implementation, since the background replacement video stream and the digital human embedded video stream are generated independently, it may lead to inconsistencies in the spatial coordinate system, causing the digital human's position to shift, mismatch between the background lighting conditions and the digital human's rendering, resulting in a false texture effect, and abrupt edge transitions, leading to visual disjointedness. This disrupts the natural integration of virtual elements and real images in the live streaming scene and reduces the user's immersive experience.

[0094] In response, this application further proposes to perform image synchronization and lighting rendering consistency processing on the background replacement video stream and the digital human embedded video stream, and output an enhanced video stream, including: Spatial registration is performed on the background replacement video stream and the digital human embedding video stream, and a unified coordinate space for the two video streams is established based on the spatial layout information in the visual semantic features. Within a unified coordinate space, the lighting features of the background area are extracted, and the digital human model is rendered in real time based on the lighting features to obtain the digital human rendering result. The digital human rendering result is synthesized with the background replacement video stream, and the edge region of the digital human is geometrically feathered based on spatial layout information to obtain an enhanced video stream.

[0095] Spatial registration refers to the process of aligning two video streams in a spatial dimension. This can be achieved using feature point matching algorithms or optical flow estimation techniques, aiming to eliminate the digital human position drift caused by coordinate system differences. Establishing a unified coordinate space based on spatial layout information from visual semantic features can be understood as constructing a common reference framework using scene structure understanding information. This can dynamically define the coordinate origin based on foreground subject distribution maps or depth maps generated by semantic segmentation, aiming to adapt to character movement and perspective changes in live streaming scenarios. Extracting the illumination features of the background region refers to capturing the light field characteristics of the background environment. This can be achieved using HD-based... The lighting estimation model or brightness-color temperature analysis module of the R image is used to obtain key parameters such as the direction and intensity of ambient light. Real-time lighting rendering of the digital human model based on lighting features can be understood as dynamically adjusting the lighting and shadow performance of virtual elements according to the background lighting. This can be achieved using a physically based rendering engine or real-time shadow mapping technology, with the aim of making the lighting and shadow of the digital human consistent with the background environment. Geometrically perceptive feathering refers to implementing adaptive transition processing for edge areas. It can dynamically adjust the feathering intensity based on edge curvature detection or boundary complexity analysis, with the aim of avoiding detail blurring or abrupt transitions caused by uniform feathering. Specifically, the proposed solution first maps the background replacement video stream and the digital human embedded video stream to a unified coordinate space through spatial registration. This process dynamically calibrates the coordinate system based on spatial layout information from visual semantic features, ensuring that the background replacement area and the integrated position of the digital human are aligned in spatial dimensions. Subsequently, the illumination features of the background area are extracted within the unified coordinate space, and the digital human model is rendered with real-time illumination based on these features. This step, through the dynamic capture and application of background illumination features, ensures that the digital human rendering result matches the current background lighting conditions in real time. Finally, when compositing the digital human rendering result with the background replacement video stream, the geometric characteristics of the digital human's edges are identified based on spatial layout information, and adaptive feathering is implemented. This processing dynamically adjusts the transition intensity according to changes in edge curvature and boundary complexity. These steps form a closed-loop processing mechanism, with each stage closely integrated: spatial registration provides an accurate spatial reference for illumination feature extraction; illumination features drive digital human rendering to ensure lighting consistency; and geometrically aware feathering optimizes visual boundaries during the compositing stage. These three elements work synergistically in the image fusion process, achieving systematic alignment of space, lighting, and edges based on the contextual awareness capabilities of visual semantic features.As a specific implementation method, the solution of this application is implemented as follows: In a live streaming scenario, when the background replacement video stream uses a virtual stage background and the digital human embedded in the video stream contains a virtual anchor, the spatial registration module extracts SIFT feature points from the two video streams for matching, and dynamically adjusts the coordinate origin to the center position of the main body of the character by combining the spatial layout information in the visual semantic features; the lighting feature extraction module analyzes the brightness histogram and color temperature distribution of the background area to generate lighting features containing the ambient light direction vector; the real-time lighting rendering module inputs the lighting features into the PBR rendering pipeline of the Unity engine to dynamically adjust the material reflectivity and shadow parameters of the virtual anchor model; in the compositing stage, an adaptive feathering algorithm based on edge curvature is used to perform non-uniform blurring on the outline of the digital human, wherein a weaker feathering intensity is applied to high curvature areas to retain details, and a stronger feathering intensity is applied to low curvature areas to achieve a smooth transition. Through the above technical solutions, this application solves the problem of inconsistent spatial coordinate systems caused by the independent generation of background replacement video stream and digital human embedded video stream, avoiding digital human position offset; by dynamically capturing and applying background lighting features, it eliminates the incongruity of light and shadow between digital human and background, overcoming the limitations of static lighting models in dynamic live streaming scenarios; at the same time, the feathering processing of geometry perception adaptively adjusts the transition intensity according to the edge geometry characteristics, realizing the natural fusion of digital human and background at the visual boundary, improving the realism and coherence of the enhanced video stream, thereby ensuring the seamless integration of virtual elements and real images in live streaming scenarios.

[0096] Specifically, in some of the embodiments described above in this application, a regional differentiated coding process is proposed to allocate coding resources based on visual semantic features and digital human activation instructions. However, in its implementation, there is a lack of a dynamic response mechanism for real-time network status and a priority-refined allocation strategy, which makes it impossible to ensure the coding quality of high-priority areas (such as key areas of digital humans) when the network fluctuates. The traditional uniform coding strategy causes the core visual elements that users care about to degrade in quality due to uneven resource allocation.

[0097] To address this, this application further proposes a method based on visual semantic features, digital human activation instructions, and real-time network status, using an adaptive encoder to perform region-specific coding on the enhanced video stream, outputting the target video stream. (See [link to relevant documentation]). Figure 5 Taking the server as the executing entity as an example, the following steps are included.

[0098] 501. Based on visual semantic features and digital human activation instructions, each frame of the enhanced video stream is divided into multiple visual regions with different encoding priorities. The visual regions include digital human key regions specified by digital human activation instructions and secondary key regions determined by saliency in visual semantic features. 502. Based on the available bandwidth of the real-time network status and the encoding priority of each visual region, dynamically allocate the target bit rate and encoding complexity budget to each visual region. 503. Based on the target bitrate and coding complexity budget allocated to each visual region, the adaptive encoder is controlled to use the corresponding quantization parameters, frame rate and motion estimation range to perform parallel and differentiated video coding for different visual regions. 504. The encoded bitstreams of each visual region are reused and encapsulated to obtain the target video stream.

[0099] Dividing each frame of the enhanced video stream into multiple visual regions with different encoding priorities refers to dividing video frames into regions based on the semantic importance of the content. This can be achieved using semantic segmentation algorithms based on convolutional neural networks or edge detection and region growing methods based on traditional image processing. The goal is to identify high-value regions of interest to the user to optimize resource allocation. Dynamically allocating target bitrate and encoding complexity budget refers to adjusting encoding parameters in real time based on network conditions and region priorities. This can be achieved using dynamic adjustment strategies based on feedback control or static mapping tables based on predefined rules. The goal is to ensure that critical regions maintain high quality even when bandwidth is limited. Controlling the adaptive encoder to use corresponding quantization parameters, frame rate, and motion estimation range refers to configuring differentiated encoding parameters for different regions. This can be achieved using configurable video encoders such as x264's region encoding function or custom parameter interfaces of hardware encoders. The goal is to achieve efficient resource utilization through parameter customization. Specifically, the solution in this application combines visual semantic features, digital human activation commands, and real-time network status. First, it performs semantically driven region segmentation on each frame of the enhanced video stream, identifying key and secondary key regions of the digital human. Then, based on dynamic changes in network status and region priority, it adjusts the bitrate and complexity budget for each region in real time. Next, based on the allocated budget, it configures differentiated encoding parameters for different regions and performs parallel encoding. Finally, it multiplexes and encapsulates the encoded bitstream. This mechanism ensures that key regions receive priority resource guarantees during network fluctuations, thereby maintaining the user experience.

[0100] As a preferred embodiment, the solution of this application is implemented as follows: In a live streaming scenario, when a digital human activation command is detected indicating that the digital human is located in the center of the screen, a pre-trained semantic segmentation model is used to divide the video frame into a key region (central region) and a secondary key region (region surrounding the person); simultaneously, network bandwidth is monitored, and if the bandwidth decreases, the bitrate allocation for the background region is reduced, while the bitrate for the key region of the digital human is increased; during encoding, a smaller quantization parameter and a higher frame rate are used for the key region of the digital human, while a larger quantization parameter is used for the background region; finally, the encoded regional bitstream is encapsulated into H.264 format and output through a standard multiplexer.

[0101] Through the above scheme, this application can intelligently prioritize the allocation of encoding resources to the key areas of the digital human that users are concerned about when network conditions fluctuate, avoiding the problem of quality degradation of core visual elements caused by traditional uniform encoding strategies, thereby providing users with a smoother and higher quality video viewing experience in various network environments.

[0102] Specifically, in some of the embodiments described above in this application, regional differential coding is proposed to intelligently allocate coding resources based on the semantic importance of the image content and network status. However, in its implementation, how to identify and dynamically divide visual regions with different coding priorities, especially for complex live streaming scenarios involving interactions between digital humans and real people, presents significant challenges. Existing methods may rely solely on static semantic segmentation or simple region labeling, failing to fully integrate the real-time interactive intent in the digital human activation command with the dynamic saliency changes in visual semantic features. This results in inaccurate priority settings for core interactive regions (such as the overlapping area between the digital human and the real person), leading to a decline in the quality of key visual content during bandwidth fluctuations and affecting the user's perceptual experience of the live streaming interaction focus.

[0103] In response, this application further proposes a method based on visual semantic features and digital human activation instructions to divide each frame of the enhanced video stream into multiple visual regions with different encoding priorities, including: For each frame of the enhanced video stream, initial semantic segmentation is performed based on spatial layout information and semantic labels in visual semantic features to obtain initial partitions including the main body region, background region, and potential interaction region. Parse the digital human location information included in the digital human activation instruction, and map the digital human location information to the initial partition; By determining spatial overlap, the area that includes the digital human and overlaps with the main human area is identified as the core interaction area, and the core interaction area is marked as the key area of ​​the digital human with the highest coding priority. Based on the dynamic saliency map in visual semantic features and the interaction intensity parameter in the digital human activation instructions, visual attention re-evaluation is performed on the regions outside the core interaction region to extract regions that have motion correlation or semantic relevance with the core interaction region, and the extracted regions are identified as secondary key regions.

[0104] Among them, spatial layout information refers to feature data that characterizes the spatial distribution and structural hierarchy of objects in a video frame. It can be achieved using geometric feature maps extracted by convolutional neural networks or spatial coordinate information output by scene parsing algorithms. The purpose is to provide a spatial structural basis for the scene and ensure the geometric accuracy of region division. Digital human position information can be understood as instruction parameters that indicate the coordinate position of the digital human in the video frame. It can be based on the coordinate values ​​contained in the digital human activation instructions or obtained from the instruction content through image recognition technology. The purpose is to align the instruction intent with the video content space and improve the reliability of position recognition. Spatial overlap determination refers to the method of determining the overlapping area by calculating the spatial intersection between the digital human region and the main human body region. It can be achieved using pixel-level mask intersection operations or bounding box overlap detection algorithms. The purpose is to identify the interactive focus area where the user's attention is concentrated. Dynamic saliency map refers to feature map that reflects the change of visual attention distribution in the video frame over time. It can be generated based on optical flow analysis or temporal saliency detection models. Interaction intensity parameter refers to an indicator that quantifies the degree of interaction between the user and the digital human. It can be derived from instruction configuration or user behavior analysis. The purpose is to dynamically adjust the region priority to adapt to the real-time changes of the live broadcast scene. Specifically, the proposed solution first performs initial semantic segmentation on video frames based on spatial layout information and semantic tags in visual semantic features, establishing a basic partition containing the main character, background, and potential interaction areas. Then, the location information in the digital human activation command is mapped to this partition, aligning the command intent with the spatial coordinates of the video content. Next, spatial overlap determination identifies areas that simultaneously contain the digital human and overlap with the main character, marking them as the digital human key regions with the highest encoding priority. Finally, combining the motion trajectory changes detected by the dynamic saliency map with the intent intensity indicated by the interaction intensity parameter, visual attention re-evaluation is performed on areas outside the core interaction region, extracting secondary regions that have motion or semantic logical connections with the core region as secondary key regions. This mechanism, through the synergistic effect of multi-source information, ensures that the region division reflects both the static semantic structure and captures the instantaneous dynamic characteristics of the live streaming scene, making the encoding priority setting closely match the user's attention distribution.

[0105] As a preferred embodiment, the solution of this application is implemented as follows: In a game live streaming scenario, when the system detects that a user sends a bullet screen containing the keyword "celebration," a digital human activation command is triggered. For the current video frame, initial semantic segmentation is first performed based on spatial layout information to distinguish the main body of the streamer, the background, and potential interaction areas; then, the digital human's position coordinates in the command are parsed and mapped to the segmentation results; the area where the digital human overlaps with the streamer is determined as the core interaction area through spatial overlap determination; finally, the streamer's gesture trajectory is detected using a dynamic saliency map, and combined with interaction intensity parameters, the gesture-related area is identified as a secondary key area. In this embodiment, more encoding resources are allocated to the key areas of the digital human to ensure the clear presentation of the celebration animation.

[0106] Through the above solution, this application can identify the core interactive areas in the live streaming scene, dynamically divide the visual areas with different encoding priorities, effectively avoid the problem of quality degradation of key visual content when bandwidth fluctuates, and improve the user's perception experience of the live streaming interactive focus.

[0107] In some of the embodiments described above in this application, a method is proposed to allocate bitrate and complexity budgets to visual regions based on real-time network status and coding priority. However, in its implementation, the allocation mechanism fails to fully combine the dynamic changes in bandwidth fluctuation trends and the fine-grained control of regional priorities. This results in the inability to intelligently prioritize the allocation of coding resources to high-priority regions when network bandwidth decreases, which may cause the core visual elements that users are concerned about to suffer quality degradation due to insufficient resources.

[0108] To address this, this application further proposes a method for dynamically allocating target bitrate and coding complexity budgets to each visual region based on available bandwidth under real-time network conditions and coding priorities for each visual region, including: Monitor real-time network status to obtain instantaneous values ​​of available bandwidth and bandwidth fluctuation trends; Based on encoding priority, differentiated bitrate allocation weights and encoding complexity levels are configured for the key regions, secondary key regions and other visual regions of the digital human. The base bitrate budget for each visual region is determined based on the instantaneous value of available bandwidth, bandwidth fluctuation trend, and bitrate allocation weight. Based on the coding complexity level, allocate a corresponding coding complexity budget to each visual region; Based on bandwidth fluctuation trends, the base bitrate budget and encoding complexity budget are adjusted in tandem. Specifically, when the bandwidth fluctuation trend indicates a decrease in available bandwidth, priority is given to maintaining the budget allocation for key areas of the digital human, while the budget for other visual areas is reduced accordingly.

[0109] Specifically, bandwidth fluctuation trend refers to the trend characteristics of network bandwidth changes over time. It can be implemented using sliding window statistical analysis or time series prediction models, with the aim of anticipating changes in network status and providing a forward-looking basis for resource allocation. Encoding complexity level refers to the level of computational resource consumption during video encoding. It can be implemented using a preset discrete level division or dynamic evaluation based on regional content complexity, with the aim of matching regional importance with encoding resource consumption. In practical applications, coordinated adjustment refers to the coordinated change mechanism between bitrate budget and encoding complexity budget. It can be implemented using a proportional allocation strategy or a dynamic mapping function based on network status, with the aim of ensuring resource security in core areas during bandwidth fluctuations. Specifically, the solution in this application obtains instantaneous bandwidth values ​​and fluctuation trends by monitoring network status in real time, enabling the system to simultaneously grasp the current bandwidth capacity and future direction of change. Based on the encoding priority of visual regions, the system configures differentiated bitrate allocation weights and encoding complexity levels for key regions, secondary key regions, and other regions of the digital human, ensuring that high-priority regions receive higher weights. Combining instantaneous bandwidth values, fluctuation trends, and weight configurations, the system dynamically calculates the basic bitrate budget for each region, with the fluctuation trend used for predictive adjustments. Simultaneously, corresponding computing resources are allocated according to the encoding complexity level. When the bandwidth fluctuation trend indicates a decrease, the system adjusts the bitrate budget and encoding complexity budget in a coordinated manner, prioritizing resources for key regions of the digital human, thereby maintaining the quality of core visual elements under resource-constrained conditions.

[0110] As a specific embodiment, the solution of this application is implemented as follows: In a live streaming scenario, when the network monitoring module detects a continuous downward trend in bandwidth, the resource allocation engine automatically increases the bitrate allocation weight of key areas of the digital human, while reducing the encoding complexity level of other areas. For example, at critical moments of interaction between the digital human and the audience, the system ensures that the digital human's face and gesture areas maintain high encoding quality, while the background areas appropriately reduce detail representation, thereby prioritizing the core elements of user experience under limited bandwidth.

[0111] Through the above solution, this application can intelligently prioritize the encoding resources of key areas of digital humans when network bandwidth fluctuates, effectively avoiding the quality degradation of core visual elements that users care about due to insufficient resources, and improving the stability of real-time video transmission and user experience.

[0112] Specifically, in some of the embodiments described above in this application, a method is proposed to implement differentiated bitrate allocation weights and encoding complexity levels based on encoding priority to achieve regional differentiated encoding. However, in its implementation, the weight configuration only depends on the predefined priority level and ignores the dynamically changing features within the region (such as the complexity of human action in key areas of a digital human and the visual salience intensity of secondary key areas). This results in insufficiently refined bitrate resource allocation when the network fluctuates in real time, and the encoding quality of key visual areas cannot be adaptively guaranteed.

[0113] In response, this application further proposes a method based on encoding priority to assign differentiated bitrate allocation weights and encoding complexity levels to the key regions, secondary key regions, and other visual regions of the digital human, including: Based on the encoding priority, the key regions, secondary key regions, and other visual regions of the digital human are mapped to predefined high, medium, and low weight intervals, respectively. Within the three weight ranges of high, medium and low, based on the complexity of human action in the key areas of the digital human and the visual salience intensity of the secondary key areas, a bitrate allocation weight value is assigned to each visual area. The three weight ranges of high, medium and low are synchronously mapped to three coding complexity levels, so that the coding complexity level of each visual region corresponds to its weight range, thereby completing the configuration of bitrate allocation weight and coding complexity level.

[0114] Among them, encoding priority refers to a quantitative indicator of the degree of influence of different visual regions in a video frame on the user experience. It can be determined based on the semantic importance of the regional content and the intensity of user interaction, aiming to provide a basic hierarchical basis for resource allocation. The predefined high, medium, and low weight intervals can be understood as a pre-set weight value range framework, which can be implemented using discrete threshold division or continuous mapping functions, aiming to establish a structured weight allocation system. The complexity of human motion specifically refers to the comprehensive representation of dynamic characteristics such as joint movement amplitude and velocity change rate in key areas of the digital human. It can be obtained through temporal posture sequence analysis or motion energy spectrum calculation, aiming to quantify... The sensitivity of the action to the encoding quality; visual saliency intensity can be understood as the prominence of secondary key regions in visual perception, which can be evaluated based on the saliency map or texture edge feature intensity generated by the attention mechanism, with the aim of identifying the user's focus; the bitrate allocation weight value refers to the specific resource allocation ratio dynamically adjusted for each region, which can be generated by linear interpolation or nonlinear function mapping, with the aim of achieving fine-grained resource control; the encoding complexity level is specifically a graded indicator of the fineness of the encoding processing, which can correspond to different quantization parameter ranges, intra-frame prediction modes or motion search depths, with the aim of ensuring that the encoding strategy is strictly matched with resource requirements.

[0115] Specifically, the proposed solution first maps regions to weight intervals based on encoding priority, establishing a correspondence between regional importance and resource allocation framework to ensure that high-priority regions receive basic weight guarantees. Building upon this, it dynamically analyzes the complexity of character actions in key regions of the digital human and the visual salience intensity of secondary key regions, adjusting the bitrate allocation weight values ​​of each region in real time within the weight intervals. This allows resource allocation to respond to dynamic content changes. Finally, the weight intervals are synchronously mapped to encoding complexity levels, enabling high-weight regions to automatically match high-complexity encoding strategies, thus forming a closed-loop linkage mechanism between weight configuration and encoding processing. This mechanism, through dynamic feature-driven weight adjustment, achieves a shift in bitrate resource allocation from static grading to content-adaptive approaches. Especially in network fluctuation scenarios, it prioritizes resource supply to key regions, avoiding resource mismatch issues caused by content dynamism.

[0116] As a preferred embodiment, the solution of this application is implemented as follows: In a live streaming scenario, when a virtual anchor makes a large gesture in a key area of ​​the digital human, the system determines that the character's movement complexity is high through joint motion trajectory analysis. The system then increases the bitrate allocation weight value for that area within a high-weight range. Simultaneously, for the highlighted areas of frequently interacting user comments in the secondary key areas, the system calculates their visual saliency intensity using a saliency graph model. If the intensity exceeds a threshold, the weight value is increased. The encoding complexity level is simultaneously adjusted to a high-complexity mode, employing more refined quantization parameters and an expanded motion search range for processing. The remaining visual areas maintain the basic encoding strategy. In this embodiment, the dynamic adjustment of weight values ​​is completed in real time based on content features, without the need for preset fixed parameters, ensuring a close fit between resource allocation and scenario requirements.

[0117] Through the above solution, this application can adaptively adjust the bitrate resource allocation according to the dynamic characteristics of video content under real-time network fluctuation conditions, effectively ensuring the stability of the encoding quality of key and secondary key areas of digital human, avoiding blurring or stuttering of key visual elements due to insufficient resources, thereby enhancing the user's immersive experience of enhanced video streams.

[0118] In some of the embodiments described above in this application, an initial bitrate budget is proposed to be determined based on the instantaneous value of available bandwidth and the bitrate allocation weight for dynamic allocation of encoding resources. However, in its implementation, when network bandwidth fluctuates, especially when stability decreases, the initial bitrate budget fails to be dynamically adjusted in conjunction with the bandwidth fluctuation trend. This results in the inability to optimize resource allocation in a timely manner to prioritize the bitrate requirements of key and secondary key areas of the digital human. This may cause the core visual elements to degrade in quality due to insufficient bitrate, affecting the user's immersive experience of key interactive content.

[0119] In response, this application further proposes a method for determining the base bitrate budget for each visual region based on the instantaneous value of available bandwidth, bandwidth fluctuation trends, and bitrate allocation weights, including: The initial bitrate budget for each visual region under static bandwidth conditions is determined based on the instantaneous value of available bandwidth and the bitrate allocation weight. Based on bandwidth fluctuation trends, the initial bitrate budget is dynamically adjusted; when the bandwidth fluctuation trend indicates a decrease in bandwidth stability, the initial bitrate budget for the remaining visual areas is reduced according to preset rules, and the bitrate budget saved is allocated to the key and secondary key areas of the digital human according to the bitrate allocation weight. The dynamically adjusted bitrate budget for each visual region is used to determine the base bitrate budget.

[0120] The instantaneous value of available bandwidth refers to a real-time measurement indicator of the current network status, which can be achieved using periodic network probe packets or traffic monitoring probes, aiming to provide an immediate quantitative basis for network capacity. The bandwidth fluctuation trend can be understood as a predictive analysis result of the network bandwidth change pattern, which can be achieved using a sliding window statistical model or time series prediction algorithm, aiming to proactively identify the trend of bandwidth stability changes. The bitrate allocation weight refers to a quantitative parameter characterizing the resource priority of each visual region, which can be achieved based on a regional importance mapping table or a dynamic scoring mechanism, aiming to establish a benchmark framework for differentiated resource allocation. The dynamic correction process refers to the adaptive adjustment operation of the initial bitrate budget based on the bandwidth fluctuation trend, which can be achieved using a rule engine trigger or an adaptive feedback control mechanism, aiming to keep the resource allocation strategy synchronized with the dynamic changes of the network. Specifically, the proposed solution first establishes an initial bitrate budget under static conditions based on the instantaneous value of available bandwidth and bitrate allocation weights, constructing a basic allocation framework for each visual region that reflects the current network capacity and regional importance. Then, it dynamically corrects the initial bitrate budget based on bandwidth fluctuation trends, whereby the bandwidth fluctuation trend serves as a predictive indicator of network state changes, enabling the system to proactively respond to potential bandwidth degradation rather than passively adjusting based solely on instantaneous values. In particular, when the bandwidth fluctuation trend indicates a decrease in stability, the initial bitrate budget for the remaining visual regions is reduced according to preset rules, and the saved bitrate budget is allocated to the key and secondary key regions of the digital human according to the bitrate allocation weights. This mechanism triggers resource reallocation through fluctuation trends, transferring redundant resources from low-priority regions to high-priority regions. Finally, the dynamically corrected bitrate budget is determined as the base bitrate budget, providing a resource basis calibrated by fluctuation trends for subsequent encoding processes, thus forming a complete closed loop from network state perception to dynamic resource optimization. As a specific implementation method, the solution of this application is implemented as follows: In a real-time video live streaming scenario, the adaptive encoder continuously monitors the available network bandwidth. When the bandwidth fluctuation trend analysis module outputs a signal of decreased stability, the system automatically lowers the initial bitrate budget for the background edge region, while proportionally allocating the released bitrate resources to key regions containing the digital human's face and gestures. For example, when a bandwidth fluctuation trend indicates that the network may be about to become congested, the encoder prioritizes the bitrate allocation for key regions of the digital human to ensure the clarity of facial expressions and interactive actions, while appropriately reducing the encoding complexity for non-critical regions such as background transition areas, thereby maintaining the visual coherence of core interactive content under bandwidth-constrained conditions.Through the above technical solution, this application can dynamically adjust the initial bitrate budget according to the bandwidth fluctuation trend, optimize the resource allocation strategy in a timely manner when the network bandwidth stability decreases, prioritize the bitrate requirements of key and secondary key areas of the digital human, effectively avoid blurring or stuttering problems caused by insufficient bitrate of core visual elements, thereby improving the immersive experience quality of users for key interactive content in live streaming scenarios.

[0121] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0122] Figure 6 This is a schematic diagram of the structure of a real-time video processing system based on artificial intelligence, provided in an embodiment of this application. See also... Figure 6 The system includes: The feature extraction module 601 is used to extract features from the original video stream, audio stream and bullet screen text in the live streaming scene, and obtain the visual semantic features of the original video stream, the audio event features of the audio stream and the text interaction features of the bullet screen text. The generation module 602 is used to generate a multimodal driving instruction set based on audio event features and text interaction features. The multimodal driving instruction set includes background generation instructions and digital human activation instructions. Enhancement module 603 is used to perform real-time enhancement processing on the original video stream based on visual semantic features and multimodal driving instruction set to obtain an enhanced video stream. The real-time enhancement processing includes background replacement based on text interaction features and digital human generation based on audio event features. The encoding module 604 is used to perform regional differential encoding processing on the enhanced video stream based on visual semantic features, digital human activation instructions and real-time network status through an adaptive encoder, and output the target video stream. In the regional differential encoding process, more encoding resources are allocated to the visual regions containing the digital human.

[0123] It should be noted that the above embodiments of the AI-based real-time video processing system are only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the AI-based real-time video processing system and the AI-based real-time video processing method embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0124] Figure 7This is a schematic diagram of a server structure provided in an embodiment of this application. The server 700 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 701 and one or more memories 702. The one or more memories 702 store at least one computer program, which is loaded and executed by the one or more processors 701 to implement the methods provided in the various method embodiments described above. Of course, the server 700 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 700 may also include other components for implementing device functions, which will not be elaborated upon here.

[0125] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a computer program that can be executed by a processor to perform the AI-based real-time video processing method described in the above embodiments. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, or optical data storage device, etc.

[0126] In an exemplary embodiment, a computer program product or computer program is also provided, which includes program code stored in a computer-readable storage medium. The processor of a computer device reads the program code from the computer-readable storage medium and executes the program code, causing the computer device to perform the above-described artificial intelligence-based real-time video processing method.

[0127] In some embodiments, the computer program involved in the present application embodiments may be deployed and executed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network. Multiple computer devices distributed in multiple locations and interconnected through a communication network may constitute a blockchain system.

[0128] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0129] The above are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An artificial intelligence-based real-time video processing method, characterized by, The method comprises: feature extraction is performed on the original video stream, audio stream and barrage text in a live scene to obtain visual semantic features of the original video stream, audio event features of the audio stream and text interaction features of the barrage text; based on the audio event features and the text interaction features, a multi-modal driving instruction set is generated, the multi-modal driving instruction set comprising background generation instructions and digital human activation instructions; based on the visual semantic features and the multi-modal driving instruction set, real-time enhancement processing is performed on the original video stream to obtain an enhanced video stream, the real-time enhancement processing comprising background replacement based on the text interaction features and digital human generation based on the audio event features; based on the visual semantic features, the digital human activation instructions and a real-time network state, the enhanced video stream is subjected to region-differentiated encoding processing by an adaptive encoder to output a target video stream, wherein more encoding resources are allocated to a visual region containing a digital human in the region-differentiated encoding processing.

2. The method of claim 1, wherein, The generation of the multi-modal driving instruction set based on the audio event features and the text interaction features comprises: acoustic event detection and speech content analysis are performed based on the audio event features to obtain audio event types and audio semantic content; real-time keyword extraction and intent analysis are performed based on the text interaction features to obtain scene keywords and user interaction intents; initial instruction sequences are generated based on the audio event types and the user interaction intents; the initial instruction sequences are subjected to parameter configuration and trigger condition configuration based on the audio semantic content and the scene keywords to generate the multi-modal driving instruction set.

3. The method of claim 2, wherein, The acoustic event detection and speech content analysis based on the audio event features to obtain audio event types and audio semantic content comprises: multi-scale time series analysis is performed on the audio event features to extract fusion audio representations, the fusion audio representations comprising short-time burst features and long-time context features; the fusion audio representations are input into a multi-head attention encoder, different attention heads of the multi-head attention encoder paying attention to acoustic event modes and speech semantic modes respectively to generate enhanced feature representations; the audio event types and the audio semantic content are synchronously output by parallel event classifiers and semantic decoders based on the enhanced feature representations, wherein input weights of the event classifiers and the semantic decoders are dynamically adjusted based on semantic correlation degrees between the audio event types and the audio semantic content.

4. The method of claim 1, wherein, The real-time enhancement processing of the original video stream based on the visual semantic features and the multi-modal driving instruction set to obtain an enhanced video stream comprises: the multi-modal driving instruction set is analyzed to obtain a target background type indicated by the background generation instructions and digital human behavior parameters indicated by the digital human activation instructions; spatial layout information in the visual semantic features is matched with the target background type to obtain a spatial mask for background replacement, and meanwhile, character pose information in the visual semantic features is coupled with the digital human behavior parameters to obtain integrated positions and states of digital humans in the video stream; replace a background region of the original video stream based on the spatial mask to obtain a background-replaced video stream, and render and embed a digital human model in the original video stream based on the integrated position and state to obtain a digital human-embedded video stream; perform picture synchronization and lighting rendering consistency processing on the background-replaced video stream and the digital human-embedded video stream, and output the enhanced video stream.

5. The method of claim 4, wherein, The matching of the spatial layout information in the visual semantic feature with the target background type to obtain a spatial mask for background replacement comprises: scene analysis of the spatial layout information to obtain a foreground subject region, a background region, and a transition region, and extraction of boundary features and texture complexity of each region; determination of compatibility scores of the background region and the transition region with a target background based on requirement characteristics of the target background type, the requirement characteristics including adaptation requirements of the target background for edge sharpness and texture details; generation of an initial spatial mask based on the compatibility scores and the boundary features, and feathering processing of the transition region; edge optimization and noise suppression of the initial spatial mask based on the texture complexity and spatial consistency requirements of the target background type to obtain the spatial mask.

6. The method of claim 4, wherein, The coupling of the character posture information in the visual semantic feature with the digital human behavior parameters to obtain an integrated position and state of a digital human in a video stream comprises: extraction of joint node three-dimensional coordinates and motion vectors from the character posture information; scaling of the three-dimensional coordinates using the resolution of the video stream, and calculation of a speed parameter of the motion vectors based on the frame rate of the video stream to determine a character activity space boundary; determination of an initial integrated position of a digital human within the activity space boundary using a digital human type and a preset action sequence in the digital human behavior parameters; fusion of the motion vectors and an interaction level in the digital human behavior parameters to construct a relative motion model of a digital human and a real character, the relative motion model being used to output a dynamic position offset of the digital human relative to the character; real-time correction of the initial integrated position based on the dynamic position offset output by the relative motion model, and driving of the preset action sequence and a character motion trajectory to keep synchronization, to generate an integrated position and state of a digital human in a video stream.

7. The method of claim 1, wherein, The region-differentiated encoding processing of the enhanced video stream by the adaptive encoder based on the visual semantic feature, the digital human activation instruction, and the real-time network state to output a target video stream comprises: division of each frame of the enhanced video stream into multiple visual regions with different encoding priorities based on the visual semantic feature and the digital human activation instruction, the visual regions including a digital human key region specified by a digital human activation instruction and a secondary key region determined by saliency in the visual semantic feature; dynamic allocation of a target code rate and an encoding complexity budget for each visual region based on available bandwidth of the real-time network state and the encoding priority of each visual region; and based on the target code rate and the encoding complexity budget allocated for each visual region, control the adaptive encoder to use corresponding quantization parameters, frame rates, and motion estimation ranges to perform parallel and differentiated video encoding on different visual regions; multiplex and package the encoded visual region bitstreams to obtain the target video stream.

8. The method of claim 7, wherein, The method comprises the following steps: based on the spatial layout information and the semantic labels in the visual semantic features, performing initial semantic segmentation on each frame of the augmented video stream to obtain initial partitions including a character subject region, a background region, and a potential interaction region; analyzing the digital human position information included in the digital human activation instruction and mapping the digital human position information to the initial partitions; determining a region including a digital human and overlapping with the character subject region as a core interaction region through spatial overlap determination, and marking the core interaction region as a digital human key region with the highest encoding priority; based on the dynamic saliency map in the visual semantic features and the interaction intensity parameter in the digital human activation instruction, performing visual attention reevaluation on regions outside the core interaction region, extracting regions having motion correlation or semantic relevance with the core interaction region, and determining the extracted regions as the secondary key regions.

9. The method of claim 7, wherein, The method comprises the following steps: monitoring the real-time network state to obtain an instantaneous value of available bandwidth and a bandwidth fluctuation trend; based on the encoding priorities, configuring the digital human key region, the secondary key regions, and the remaining visual regions with differentiated code rate allocation weights and encoding complexity levels, respectively; based on the instantaneous value of the available bandwidth, the bandwidth fluctuation trend, and the code rate allocation weights, determining a basic code rate budget for each visual region; based on the encoding complexity levels, allocating a corresponding encoding complexity budget for each visual region; based on the bandwidth fluctuation trend, performing linkage adjustment on the basic code rate budget and the encoding complexity budget; when the bandwidth fluctuation trend indicates a decrease in available bandwidth, the budget allocation of the digital human key region is preferentially maintained, and the budgets of the remaining visual regions are correspondingly reduced.

10. An artificial intelligence based real-time video processing system, characterized in that, The system comprises: a feature extraction module configured to extract features from an original video stream, an audio stream, and a barrage text in a live scene to obtain visual semantic features of the original video stream, audio event features of the audio stream, and text interaction features of the barrage text; a generation module configured to generate a multi-modal driving instruction set based on the audio event features and the text interaction features, the multi-modal driving instruction set comprising background generation instructions and digital human activation instructions; and a video stream generation module configured to generate an augmented video stream based on the multi-modal driving instruction set. An enhancement module is configured to perform real-time enhancement processing on the original video stream based on the visual semantic features and the multi-modal driving instruction set to obtain an enhanced video stream, wherein the real-time enhancement processing includes background replacement based on the text interaction features and digital human generation based on the audio event features; An encoding module is configured to perform region-differentiated encoding processing on the enhanced video stream by using an adaptive encoder based on the visual semantic features, the digital human activation instruction and real-time network status to output a target video stream, wherein more encoding resources are allocated to a visual region containing a digital human during the region-differentiated encoding processing.

Images