Al-based live broadcast scene automatic identification and adaptation method and system

Abstract

The invention provides an Al-based live broadcast scene automatic identification and adaptation method and system. Belongs to the technical field of artificial intelligence. The method comprises the following steps: carrying out multi-modal data real-time acquisition on a live broadcast stream, and generating a visual frame sequence, an audio waveform, a text semantic stream and a network state parameter; performing dynamic region segmentation based on a visual frame sequence, performing voiceprint feature extraction in combination with an audio waveform, performing keyword clustering through a text semantic stream, and fusing network state parameters to generate a live scene original feature set; through multi-modal data fusion and scene semantic modeling, deep features of visual, audio, text and network states in live broadcast can be accurately captured, scene types and semantic information can be accurately identified even in a complex composite scene, and the accuracy and comprehensiveness of scene identification are greatly improved.

Description

Technical Field

[0001] This invention proposes an AI-based method and system for automatic recognition and adaptation of live streaming scenarios, belonging to the field of artificial intelligence technology. Background Technology

[0002] With the booming development of the live streaming industry, users have increasingly demanding requirements for the live streaming experience, expecting a personalized and high-quality viewing experience in different scenarios. However, current live streaming scene recognition and adaptation technologies have many limitations.

[0003] Existing technologies mostly employ single-modal recognition methods, relying solely on one type of information from vision, audio, or text, leading to biased scene judgments. For example, a visual model might classify a scene as a meeting simply because multiple people are seated in the frame, ignoring promotional messages in the audio or product images shared on the screen, resulting in misjudgment. Furthermore, scene definitions are often rigid, and the limited pre-defined categories struggle to handle complex and varied scenarios, making it difficult to accurately categorize educational live streams that include experimental demonstrations.

[0004] In the adaptation phase, existing technologies also present numerous problems. There is a delay of several seconds in parameter adjustment after scene switching, which seriously affects the continuity of user experience. Moreover, modules such as encoding, beautification, and subtitles run independently, and resource scheduling lacks coordination. It is impossible to dynamically allocate computing power and bandwidth according to scene priority, resulting in resource waste or insufficient processing of critical links.

[0005] In addition, existing solutions mostly focus on improving single functions, such as more accurate classification, filter switching or intelligent beautification, but fail to deeply integrate the deep semantic features of the scene with multimodal perception, dynamic resource scheduling and edge-cloud collaborative architecture to build a complete adaptive live streaming operating system, and cannot achieve end-to-end scene-specific experience generation. Summary of the Invention

[0006] This invention provides an AI-based method and system for automatic identification and adaptation of live streaming scenes, in order to solve the problems mentioned in the background section above:

[0007] The present invention proposes an AI-based method for automatic identification and adaptation of live streaming scenes, the method comprising:

[0008] S1. Real-time multimodal data acquisition of the live stream generates visual frame sequences, audio waveforms, text semantic streams, and network state parameters; dynamic region segmentation is performed based on the visual frame sequences, voiceprint feature extraction is performed in combination with the audio waveforms, keyword clustering is performed through the text semantic stream, and network state parameters are fused to generate the original feature set of the live stream scene.

[0009] S2. Perform multimodal feature fusion processing based on the original feature set of the live streaming scene to construct a structured scene fingerprint. Based on the structured scene fingerprint, perform scene semantic modeling, identify basic scene types, and further analyze deep semantic features to generate composite scene semantic description data.

[0010] S3. Based on the semantic description data of the composite scenario, perform dynamic resource demand prediction, obtain the corresponding results and conduct corresponding analysis to generate a resource collaborative configuration scheme.

[0011] S4. Real-time commands are issued to edge computing nodes through a resource collaborative configuration scheme to drive the collaborative operation of the four modules. After local optimization is completed at the edge nodes, key scene features are uploaded to the cloud for global model updates, forming a closed loop of edge and cloud collaborative feedback. The generation accuracy of structured scene fingerprints is continuously optimized through global model updates, generating a dynamically optimized set of scene adaptation parameters.

[0012] S5. Based on the dynamically optimized scene adaptation parameter set, end-to-end experience generation is performed. Related operations are completed under low latency constraints. The generated results are scored in real time through the experience quality assessment model. When the score is lower than the threshold, the re-identification process is triggered, forming a closed-loop control of the entire process, and finally generating exclusive live streaming experience data.

[0013] The present invention proposes a system for implementing the AI-based automatic live streaming scene recognition and adaptation method described above, the system comprising:

[0014] Data acquisition module: performs real-time multimodal data acquisition of the live stream, generating visual frame sequences, audio waveforms, text semantic streams, and network state parameters; performs dynamic region segmentation based on visual frame sequences, extracts voiceprint features by combining audio waveforms, performs keyword clustering through text semantic streams, and integrates network state parameters to generate the original feature set of the live stream scene;

[0015] Feature fusion module: Performs multimodal feature fusion processing based on the original feature set of the live streaming scene, constructs a structured scene fingerprint, performs scene semantic modeling based on the structured scene fingerprint, identifies basic scene types, and further analyzes deep semantic features to generate composite scene semantic description data;

[0016] Demand forecasting module: Performs dynamic resource demand forecasting based on semantic description data of complex scenarios, obtains corresponding results and performs corresponding analysis, and generates resource collaborative configuration schemes;

[0017] Dynamic optimization module: It issues real-time instructions to edge computing nodes through a resource collaboration configuration scheme, driving the four modules to work together. After the edge nodes complete local optimization, the key scene features are uploaded to the cloud for global model update, forming a closed loop of edge and cloud collaboration feedback. The global model update continuously optimizes the generation accuracy of structured scene fingerprints and generates a dynamically optimized set of scene adaptation parameters.

[0018] Closed-loop control module: Generates end-to-end experience based on dynamically optimized scene adaptation parameter set, completes related operations under low latency constraints, scores the generated results in real time through experience quality assessment model, and triggers re-identification process when the score is lower than the threshold, forming a closed-loop control of the entire process of understanding, decision-making, execution and feedback, and finally generating exclusive live broadcast experience data.

[0019] The beneficial effects of this invention are as follows: Through multimodal data fusion and scene semantic modeling, it can accurately capture the deep features of visual, audio, text, and network status in live streaming. Even in complex and composite scenarios, it can accurately identify scene types and semantic information, greatly improving the accuracy and comprehensiveness of scene recognition. Based on dynamic resource demand prediction and edge-cloud collaborative architecture, it can adjust the parameters and resource allocation of modules such as encoding, beautification, and subtitles in real time according to different scenarios, effectively reducing resource waste, enhancing resource utilization efficiency, and reducing problems such as stuttering and latency caused by unreasonable resource scheduling. The closed-loop control of the entire process of understanding, decision-making, execution, and feedback can quickly complete parameter adjustments when switching scenes, avoiding experience gaps and providing users with a smoother experience. At the same time, it can not only meet the exclusive experience needs of different scenarios such as parties, lectures, and conferences, but also continuously optimize based on real-time feedback, avoiding user churn due to poor scene adaptation, and significantly improving user satisfaction and the competitiveness of the live streaming platform. Attached Figure Description

[0020] Figure 1 This is a diagram illustrating the steps of the method described in this invention;

[0021] Figure 2 This is a system module diagram of the present invention. Detailed Implementation

[0022] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0023] One embodiment of the present invention, such as Figure 1 As shown, an AI-based method for automatic recognition and adaptation of live streaming scenarios includes:

[0024] S1. Real-time multimodal data acquisition of the live stream generates visual frame sequences, audio waveforms, text semantic streams, and network state parameters; dynamic region segmentation is performed based on the visual frame sequences, voiceprint feature extraction is performed in combination with the audio waveforms, keyword clustering is performed through the text semantic stream, and network state parameters are fused to generate the original feature set of the live stream scene.

[0025] S2. Perform multimodal feature fusion processing based on the original feature set of the live streaming scene to construct a structured scene fingerprint, which includes visual, audio, text and network dimensions; perform scene semantic modeling based on the structured scene fingerprint to identify basic scene types, which include parties, lectures and conferences, and further analyze deep semantic features, which include emotional intensity, information density and interaction patterns, to generate composite scene semantic description data.

[0026] S3. Based on the semantic description data of the composite scene, perform dynamic resource demand prediction, obtain corresponding results and perform corresponding analysis. The corresponding results include encoding complexity index, beautification rendering priority, subtitle generation strategy and director switching frequency. The corresponding analysis includes dynamically allocating the bitrate of the encoding module through the encoding complexity index, scheduling the computing power of the graphics processing unit through the beautification rendering priority, adjusting the parameters of the natural language processing model through the subtitle generation strategy, and intelligently mixing the signals of multiple cameras through the director switching frequency to generate a resource collaborative configuration scheme.

[0027] S4. Real-time commands are issued to edge computing nodes through a resource collaborative configuration scheme to drive the collaborative operation of four modules, namely the encoding module, beautification module, subtitle module, and directing module; after local optimization is completed at the edge nodes, key scene features are uploaded to the cloud for global model updates, forming a closed loop of edge and cloud collaborative feedback; the generation accuracy of structured scene fingerprints is continuously optimized through global model updates to generate a dynamically optimized set of scene adaptation parameters;

[0028] S5. Based on the dynamically optimized scene adaptation parameter set, end-to-end experience generation is performed, and related operations are completed under low latency constraints. The related operations include encoding quality adjustment, beauty effect adaptation, dynamic subtitle generation, and broadcast screen switching. The generated results are scored in real time through the experience quality evaluation model. When the score is lower than the threshold, a re-identification process is triggered, forming a closed-loop control of the entire process of understanding, decision-making, execution, and feedback, and finally generating exclusive live broadcast experience data.

[0029] The working principle and effects of the above technical solution are as follows: Through multimodal data acquisition and fusion processing, the accuracy of live streaming scene recognition can be significantly improved, avoiding scene misjudgment caused by single-dimensional feature analysis and reducing the disconnect between adaptation strategies and actual scenarios. Dynamic resource demand prediction and collaborative configuration can improve the utilization efficiency of computing power and bandwidth resources, reduce ineffective resource consumption, and avoid live streaming stuttering or image quality degradation caused by resource allocation imbalance. The feedback loop of edge and cloud collaboration can continuously optimize scene adaptation parameters, enhance the dynamic adjustment capability of adaptation strategies, and reduce subsequent manual intervention costs. Full-process closed-loop control and real-time quality scoring can promptly correct adaptation deviations, avoid poor live streaming experience affecting user viewing, accurately match the differentiated needs of different scenarios such as galas and lectures, and ensure the smooth and stable operation of the entire live streaming process, thereby improving the overall live streaming experience.

[0030] In one embodiment of the present invention, S1 includes:

[0031] S11. Real-time multimodal data acquisition of the live stream to generate visual frame sequences, audio waveforms, text semantic streams, and network status parameters;

[0032] S12. Perform synchronous preprocessing on visual frame sequences, audio waveforms, text semantic streams, and network state parameters to complete the timestamp alignment of multimodal data and generate a time-series synchronized multimodal dataset.

[0033] S13. Based on the temporal synchronous multimodal dataset, extract the inter-frame difference information of the visual frame sequence, perform dynamic region segmentation, and generate a foreground target region set and a background environment region set.

[0034] S14. Retrieve audio waveforms from the time-series synchronous multimodal dataset, combine them with the time-series change features of the foreground target region set, extract voiceprint features, and generate human voice feature vectors and environmental sound feature vectors.

[0035] S15. Extract the text semantic stream from the time-series synchronous multimodal dataset, perform keyword clustering based on word segmentation results and word frequency statistics to generate topic semantic clusters; integrate the foreground target region set, background environment region set, human voice feature vector, ambient sound feature vector, topic semantic clusters and network state parameters to perform feature dimension calibration and generate the original feature set of the live streaming scene.

[0036] The working principle and effects of the above technical solution are as follows: Synchronous preprocessing of multimodal data achieves timestamp alignment, ensuring the temporal consistency of data from different dimensions and avoiding feature association errors caused by data misalignment. Dynamic region segmentation based on inter-frame differences can accurately separate the foreground and background, and extracting voiceprint features by combining foreground temporal changes improves the targeting of feature extraction and reduces interference from irrelevant environmental information. Keyword clustering and multi-dimensional feature fusion calibration enrich the dimensional coverage of the original features, improve the completeness and consistency of the feature set, and avoid the one-sidedness brought by single-dimensional features. The entire process makes the generated original feature set of the live scene more closely match the dynamics of the actual scene, reduces the error of subsequent feature processing, provides reliable data support for subsequent scene recognition, and prevents the overall adaptation effect from being affected by insufficient initial feature quality.

[0037] In one embodiment of the present invention, S13 includes:

[0038] Retrieve the visual frame sequence from the time-synchronized multimodal dataset, extract continuous frame segments to generate a continuous visual frame subset; perform grayscale conversion on the continuous visual frame subset to generate a grayscale frame sequence;

[0039] Calculate the pixel difference between adjacent frames in the grayscale frame sequence to generate an inter-frame difference matrix; perform threshold segmentation on the inter-frame difference matrix to generate an inter-frame difference heatmap.

[0040] Based on the inter-frame difference heatmap, dynamic pixel regions and static pixel regions are divided to generate an initial region division result; connected component analysis is performed on the initial region division result to merge discrete dynamic pixel regions and generate a dynamic region set.

[0041] Isolated noise pixels are removed from the dynamic region set to generate a cleaned dynamic region set; based on the cleaned dynamic region set, the corresponding static region range is divided to generate a foreground target region set and a background environment region set.

[0042] The working principle and effects of the above technical solution are as follows: Extracting continuous visual frame segments and converting them to grayscale simplifies data processing while preserving core inter-frame differences and reducing interference from redundant color image data in subsequent analysis. A difference matrix is ​​generated by calculating the pixel differences between adjacent frames, and combined with threshold segmentation to transform it into a heatmap, making dynamic regions more intuitive and improving the accuracy of dynamic and static pixel region segmentation. Connected component analysis effectively merges discrete dynamic pixel regions, avoiding misjudgment of foreground targets caused by dynamic region fragmentation; removing isolated noisy pixels further purifies dynamic regions, reducing noise interference in region segmentation. The final generated foreground target and background environment region set has clear boundaries and strong integrity, avoiding the impact of blurred region segmentation or noise on subsequent feature extraction, providing a high-quality visual region foundation for subsequent fusion processing combining audio and other features.

[0043] In one embodiment of the present invention, S2 includes:

[0044] S21. Based on the original feature set of the live streaming scene, multimodal feature fusion processing is performed. Information from each dimension is integrated using a weighted concatenation method to generate an initial scene feature matrix. The initial scene feature matrix is ​​then reduced in dimension and redundant data is removed to construct a structured scene fingerprint. The structured scene fingerprint includes visual, audio, text, and network dimensions.

[0045] S22. Construct a semantic association network based on structured scene fingerprints to perform scene semantic modeling;

[0046] S23. Identify basic scene types by matching a preset scene feature library through a semantic association network. Basic scene types include parties, lectures, and conferences. Extract temporal change features and interactive behavior features within different basic scene types.

[0047] S24. Further analyze the deep semantic features by combining temporal change features and interactive behavior features; the deep semantic features include emotional intensity, information density and interaction patterns. Integrate the basic scene types and deep semantic features to generate composite scene semantic description data.

[0048] The working principle and effects of the above technical solution are as follows: Multimodal feature weighted splicing and fusion can fully integrate information from various dimensions of visual, audio, and text networks. Combined with dimensional reduction and redundancy removal, it reduces the interference of invalid features on the processing flow, improves feature processing efficiency, and avoids modeling chaos caused by feature redundancy. Structured scene fingerprints make the correlation between features of various dimensions clearer. The semantic association network built on this basis can strengthen the semantic correlation of scenes, improve the accuracy of basic scene type matching and recognition, and avoid confusion and misjudgment between different scenes. For different basic scenes, temporal change and interaction behavior features are extracted to further analyze deep semantics, enriching the dimensions of scene description. This makes the generated composite scene semantic description data more in line with the dynamics of actual scenes and avoids the one-sidedness of information caused by only recognizing basic types. The overall process can provide accurate scene basis for subsequent dynamic resource demand prediction and prevent resource allocation from becoming disconnected from scene requirements due to insufficient scene semantic description.

[0049] In one embodiment of the present invention, step S21 includes:

[0050] S211. Retrieve the original feature set of the live streaming scene, split the features into visual, audio, text and network dimensions, and generate single-dimensional feature subsets; standardize each single-dimensional feature subset, unify the feature value scale, and generate a standardized single-dimensional feature set.

[0051] S212. Calculate the information contribution of each standardized single-dimensional feature set and generate a feature weight coefficient set; perform weighted processing on the standardized single-dimensional feature set based on the feature weight coefficient set to generate a weighted single-dimensional feature set.

[0052] S213. Orderly concatenate the weighted single-dimensional feature set, integrate the information of each dimension, and generate an initial scene feature matrix; calculate the linear correlation between features based on the initial scene feature matrix to generate a feature correlation matrix.

[0053] S214. Based on the feature correlation matrix, filter highly overlapping features to generate a redundant feature list; remove the features corresponding to the redundant feature list from the initial scene feature matrix to generate a deredundant feature matrix.

[0054] S215. Perform dimensionality compression on the redundant feature matrix, retain the core feature information, and generate a low-dimensional core feature matrix; construct a structured scene fingerprint based on the low-dimensional core feature matrix.

[0055] The working principle and effects of the above technical solution are as follows: Single-dimensional feature splitting and standardization unify the numerical scale of features of different dimensions, avoiding feature fusion imbalance caused by scale differences. By calculating the information contribution and assigning weights, the role of core features can be highlighted, the interference of secondary features on the fusion result can be reduced, and the targeting of feature fusion can be improved. Based on the correlation matrix, redundant features are screened and eliminated, reducing the complexity of data processing and avoiding information redundancy and low modeling efficiency caused by highly overlapping features. Dimensional compression simplifies feature dimensions while retaining core information, reducing the computational consumption of subsequent processing. At the same time, the generated low-dimensional core feature matrix makes the structured scene fingerprint more concise and accurate. The entire process ensures the integrity of feature information and improves the efficiency of feature processing, avoiding the impact of feature quality issues on the effect of subsequent scene semantic modeling, and laying a reliable foundation for accurate scene type identification.

[0056] In one embodiment of the present invention, S215 includes:

[0057] Retrieve the redundant feature matrix, calculate the variance contribution value of each feature, and generate a feature variance contribution sequence; sort the features in descending order of the variance contribution sequence, define the core feature screening interval, and generate a core feature candidate set.

[0058] The core feature candidate set is subjected to a second numerical normalization process to eliminate numerical interference during the dimensionality compression process and generate a normalized candidate feature matrix.

[0059] The feature space mapping algorithm is used to transform the dimension of the normalized candidate feature matrix, reduce the feature dimension size, and generate a low-dimensional transition feature matrix.

[0060] Calculate the information overlap between the low-dimensional transition feature matrix and the redundancy-removed feature matrix to generate an information retention assessment result; adjust the dimension mapping parameters based on the information retention assessment result to fill in the key information gaps and generate a low-dimensional core feature matrix.

[0061] Extract the feature association structure of the low-dimensional core feature matrix, sort out the temporal and logical relationships between the features, and generate a feature association map; integrate the numerical information of the low-dimensional core feature matrix with the structural information of the feature association map to generate a structured scene fingerprint.

[0062] The working principle and effects of the above technical solution are as follows: By calculating the variance contribution value to screen core features, high-value information can be accurately identified, avoiding the dilution of core information caused by the mixing of secondary features. Secondary normalization processing eliminates numerical fluctuation interference during dimensionality compression, ensuring the stability of feature data and reducing deviations in subsequent dimensionality transformations. Feature space mapping effectively reduces the dimensionality, lowers the computational cost of data processing, and improves feature processing efficiency. Information overlap assessment and parameter adjustment can promptly fill in key information gaps, avoiding the loss of core information caused by dimensionality compression and ensuring the integrity of the low-dimensional core feature matrix. The fusion of numerical information and feature association maps to generate structured scene fingerprints ensures that the fingerprints contain both core data support and clear feature logic relationships, improving the accuracy of subsequent scene semantic modeling and preventing scene recognition effects from being affected by incomplete or logically confused fingerprint information.

[0063] In one embodiment of the present invention, S3 includes:

[0064] S31. Construct a resource demand prediction model based on the semantic description data of composite scenes. Input the dynamic change characteristics of the scene and historical resource configuration data to perform dynamic resource demand prediction and generate the coding complexity index, beauty rendering priority, subtitle generation strategy and director switching frequency.

[0065] S32. Based on the coding complexity index, analyze the coding resource requirements of different video regions, dynamically allocate the bitrate of the coding module, and generate a differentiated bitrate configuration scheme; based on the priority of beautification rendering, sort the rendering tasks, schedule the computing power of the graphics processing unit, and generate a computing power allocation sequence.

[0066] S33. Based on the subtitle generation strategy, analyze the temporal rhythm and importance of the text semantic flow, adjust the parameters of the natural language processing model, and generate a subtitle generation parameter set.

[0067] S34. Analyze the temporal correlation of multiple camera signals according to the director's switching frequency, perform intelligent mixing of multiple camera signals, and generate shot switching sequences and mixing rules.

[0068] S35, integrates differentiated bitrate configuration schemes, computing power allocation sequences, subtitle generation parameter sets, and shot switching sequences with mixing rules to generate resource collaborative configuration schemes.

[0069] The working principle and effects of the above technical solution are as follows: Resource demand prediction is performed based on semantic description data of composite scenes. Combined with dynamic scene change characteristics and historical configuration data, this improves the accuracy of resource demand judgment and avoids a disconnect between resource configuration and actual scene needs. Differentiated bitrate allocation adapts to the encoding requirements of different video regions, improving bitrate utilization efficiency and reducing ineffective bandwidth consumption. Computing power is scheduled according to beautification priority, allowing graphics processing unit resources to focus on serving core tasks, avoiding idle or overloaded computing power. Subtitle parameters are adjusted to match the timing and importance of the text, ensuring subtitle generation quality. Intelligent mixing sorts out the timing correlation of camera signals, making shot transitions smoother. The resource collaborative configuration generated by the integrated solutions of various modules ensures coordinated operation of encoding, beautification, subtitles, and directing, avoiding conflicts and interference between modules. The entire process can accurately match the resource needs of different scenarios such as galas and lectures, while improving overall resource utilization efficiency and preventing problems such as live streaming stuttering and poor effects caused by chaotic resource configuration.

[0070] In one embodiment of the present invention, S32 includes:

[0071] Retrieve the coding complexity index and video region segmentation data, establish the correspondence between each region and the complexity index, and generate a region complexity correspondence table; based on the region complexity correspondence table, divide the region into three complexity levels: high, medium, and low, and generate a region complexity rating result.

[0072] Based on different complexity levels, a base bitrate range is set, and combined with the area ratio of the region, a regional base bitrate set is generated; real-time bandwidth data of live broadcast transmission is collected, bandwidth fluctuation amplitude is calculated, and bitrate adjustment coefficient is generated.

[0073] The values ​​in the regional base bitrate set are corrected using a bitrate adjustment factor to obtain the target bitrate for the adapted bandwidth of each region, and a list of regional target bitrates is generated.

[0074] Integrate the regional target bitrate list, clarify the regional bitrate allocation rules of the encoding module, and generate a differentiated bitrate configuration scheme; retrieve the beautification rendering priority data and the list of rendering tasks to be executed, establish the mapping relationship between tasks and priorities, and generate a task priority lookup table;

[0075] The rendering tasks are sorted in descending order of priority, duplicate tasks are removed, and an ordered task queue is generated. The rendering parameters and effect requirements of each task in the ordered task queue are analyzed, the required computing resources are estimated, and the computing power requirement value of each task is generated.

[0076] Monitor the current operating status of the graphics processing unit, count the occupied and remaining computing power, and generate the total available computing power; combine task priority and computing power requirement value, allocate available computing power resources to each task proportionally, and generate task computing power allocation results; based on the order of the ordered task queue, integrate the computing power allocation results of each task to generate a computing power allocation sequence.

[0077] The working principle and effects of the above technical solution are as follows: Establishing a correspondence between regions and encoding complexity and classifying them into levels allows bitrate allocation to better match the actual encoding needs of different regions, avoiding resource waste or insufficient encoding quality caused by blind allocation. Combining real-time bandwidth fluctuations to generate adjustment coefficients to correct the base bitrate ensures the target bitrate accurately adapts to changes in transmission bandwidth, reducing live stream stuttering or bitrate redundancy caused by unstable bandwidth and guaranteeing the stability of encoding transmission. Mapping and sorting tasks with beautification priorities eliminates duplicate tasks, making computing power scheduling more targeted and preventing core rendering tasks from being affected by insufficient computing power allocation, while also reducing the computing resources occupied by duplicate tasks. Real-time monitoring of the graphics processing unit's operating status allocates computing power, balancing the resource needs of each task and avoiding idle or overloaded computing power. The entire process improves the utilization efficiency of bitrate and computing power while ensuring stable output of encoding quality and beautification effects, avoiding a decline in live stream experience due to unbalanced resource allocation.

[0078] In one embodiment of the present invention, step S4 includes:

[0079] S41. Based on the resource collaborative configuration scheme, standardized control instructions are generated and real-time instructions are issued to edge computing nodes. The edge computing nodes parse the standardized control instructions and drive the encoding module, beautification module, subtitle module and directing module to work together to complete the real-time processing of the live stream.

[0080] S42. After completing local optimization at the edge nodes, extract key scene features and resource consumption data during the processing to generate edge node processing feedback data.

[0081] S43. Upload the edge node processing feedback data to the cloud. The cloud combines the feedback data from multiple edge nodes to update the global model and correct the scene recognition and resource prediction parameters.

[0082] S44. Form a closed loop of edge and cloud collaborative feedback, and continuously optimize the generation accuracy of structured scene fingerprints through global model updates; generate dynamically optimized scene adaptation parameter sets based on the optimized structured scene fingerprints.

[0083] The working principle and effects of the above technical solution are as follows: The standardized generation and distribution of control commands ensure the uniformity of execution parameters across modules, avoiding operational conflicts due to differences in command formats, and enabling smoother collaborative operation of the four major modules, including encoding and beautification. Edge computing nodes perform real-time processing of the live stream locally, significantly reducing data transmission latency, minimizing the risk of stuttering caused by remote processing, and ensuring a smooth live streaming experience. Extracting key scene features and resource consumption data to form feedback, combined with data from multiple edge nodes for global cloud model updates, accurately corrects scene recognition and resource prediction parameters, improving the model's flexibility in adapting to different scenarios. The cloud-edge collaborative feedback loop continuously optimizes the accuracy of structured scene fingerprint generation, ensuring that the dynamic adaptation parameter set constantly reflects changes in the actual scene, reducing adaptation deviations caused by fixed model parameters. The entire process not only guarantees the real-time performance and stability of the current live stream processing but also improves the accuracy of subsequent scene adaptation through continuous iterative optimization, avoiding the problem of declining adaptation effects after long-term operation.

[0084] In one embodiment of the present invention, S41 includes:

[0085] Retrieve the resource collaboration configuration scheme, break down the specific configuration requirements of each module such as encoding, beautification, subtitle, and directing, and generate module configuration details; based on the module configuration details, formulate instruction encoding specifications, unify instruction field formats and transmission protocols, and generate standardized instruction templates.

[0086] Fill the module configuration details into the standardized instruction template to generate the initial control instruction set; verify the completeness of the fields in the initial control instruction set and the compatibility between modules, remove conflicting instruction entries, and generate standardized control instructions;

[0087] Establish a secure communication link with the edge computing node, detect the link transmission rate and stability, and generate the link connection status; based on the link connection status, send standardized control commands to the edge computing node in real time, receive command reception confirmation signals from the receiving node, and generate a command delivery completion identifier;

[0088] After receiving standardized control commands, the edge computing node starts the command parsing engine, breaks down the execution parameters of each module, and generates a module execution parameter list. Based on the module execution parameter list, it coordinates the startup sequence of each module to avoid running conflicts and generates a module coordination sequence table.

[0089] The encoding module, beautification module, subtitle module and directing module are started and run in sequence according to the module collaboration timing table; each module processes the live stream in parallel based on the execution parameters, integrates the processing results and generates real-time processed live stream data.

[0090] The system detects the temporal continuity and format standardization of the live stream data after real-time processing, and generates a processing completion verification result. After the verification passes, the system confirms that the live stream processing is complete in real time and generates a processing completion identifier.

[0091] The working principle and effects of the above technical solution are as follows: It breaks down the configuration requirements of each module and establishes standardized instruction templates, unifying the instruction format and transmission protocol, thus avoiding parsing failures or execution deviations at edge nodes due to inconsistent instruction specifications. It verifies the integrity and compatibility of initial control instructions, eliminating conflicting entries and reducing the risk of inter-module operational conflicts, allowing for smoother module collaboration. It establishes secure communication links and monitors transmission status, ensuring the stability and security of instruction issuance and preventing instruction loss or interference during transmission. After parsing the instructions, it coordinates the module startup sequence, allowing modules such as encoding and beautification to start sequentially, avoiding resource contention caused by simultaneous startup and improving operational efficiency. Modules process the live stream in parallel and integrate the results, improving processing speed and reducing live stream latency; it performs timing and format verification on the processed live stream to ensure the output data is qualified, preventing unqualified data from affecting subsequent live stream displays. The entire process not only ensures the accuracy and stability of instruction issuance and execution but also improves the efficiency and quality of real-time live stream processing, preventing live stream interruptions or poor performance due to oversights in intermediate steps.

[0092] In one embodiment of the present invention, step S5 includes:

[0093] S51. Based on the dynamically optimized scene adaptation parameter set, analyze the end-to-end experience generation requirements and start encoding quality adjustment, beauty effect adaptation, dynamic subtitle generation and broadcast screen switching under low latency constraints.

[0094] S52. After completing the end-to-end experience generation, generate initial live broadcast experience data; call the experience quality assessment model to score the initial live broadcast experience data in real time, and quantify the clarity of the live broadcast picture, audio stability, subtitle accuracy and broadcast smoothness.

[0095] S53. Compare the scoring results with the preset threshold. When the score is lower than the threshold, trigger the re-identification process. The re-identification process goes back to the real-time acquisition stage of multimodal data, reacquires the live stream data and completes subsequent processing to correct the adaptation deviation.

[0096] S54. Establish a closed-loop control system for the entire process of understanding, decision-making, execution, and feedback, and continuously iterate and optimize the processing effect of each link; ultimately generating exclusive live streaming experience data.

[0097] The working principle and effects of the above technical solution are as follows: End-to-end experience generation is initiated based on dynamically optimized scene adaptation parameters. Under low-latency constraints, operations such as encoding quality and beautification effects are adjusted, ensuring both the smoothness of the live stream and the precise matching of each stage's effects to scene requirements. Real-time quantification of core indicators such as image clarity and audio stability allows for timely identification of experience shortcomings, preventing substandard live stream experiences from being directly presented to users. When the score falls below a threshold, backtracking and re-identification are triggered, re-collecting and processing data to correct adaptation deviations, reducing the continuous impact of single adaptation errors. Closed-loop control throughout the entire process drives continuous iterative optimization at each stage, enhancing the flexibility and accuracy of the adaptation strategy and preventing experience disconnects caused by fixed parameters. The final generated exclusive live stream experience data fits the personalized needs of different scenarios, improving the user viewing experience while reducing the cost of manual intervention and adjustments, preventing a decline in experience over long-term operation.

[0098] One embodiment of the present invention, such as Figure 2 As shown, a system for implementing the AI-based automatic recognition and adaptation method for live streaming scenes as described above is characterized in that the system comprises:

[0099] Data acquisition module: performs real-time multimodal data acquisition of the live stream, generating visual frame sequences, audio waveforms, text semantic streams, and network state parameters; performs dynamic region segmentation based on visual frame sequences, extracts voiceprint features by combining audio waveforms, performs keyword clustering through text semantic streams, and integrates network state parameters to generate the original feature set of the live stream scene;

[0100] Feature fusion module: Performs multimodal feature fusion processing based on the original feature set of the live streaming scene to construct a structured scene fingerprint, which includes visual, audio, text, and network dimensions; performs scene semantic modeling based on the structured scene fingerprint to identify basic scene types, including parties, lectures, and conferences, and further analyzes deep semantic features, including sentiment intensity, information density, and interaction patterns, to generate composite scene semantic description data;

[0101] Demand Prediction Module: Based on the semantic description data of composite scenes, dynamically predict resource demand, obtain corresponding results, and perform corresponding analysis. The corresponding results include encoding complexity index, beautification rendering priority, subtitle generation strategy, and director switching frequency. The corresponding analysis includes dynamically allocating bitrate to the encoding module through the encoding complexity index, scheduling computing power of the graphics processing unit through beautification rendering priority, adjusting parameters of the natural language processing model through the subtitle generation strategy, and intelligently mixing signals from multiple cameras through the director switching frequency to generate a resource collaborative configuration scheme.

[0102] Dynamic optimization module: Through a resource collaborative configuration scheme, real-time instructions are issued to edge computing nodes to drive the collaborative operation of four modules, namely the encoding module, beautification module, subtitle module, and directing module. After local optimization is completed at the edge nodes, key scene features are uploaded to the cloud for global model updates, forming a closed loop of edge and cloud collaborative feedback. The generation accuracy of structured scene fingerprints is continuously optimized through global model updates, generating a dynamically optimized set of scene adaptation parameters.

[0103] Closed-loop control module: Based on the dynamically optimized scene adaptation parameter set, it generates end-to-end experience and completes related operations under low latency constraints. These related operations include encoding quality adjustment, beauty effect adaptation, dynamic subtitle generation, and broadcast screen switching. The generated results are scored in real time through an experience quality evaluation model. When the score is lower than the threshold, a re-identification process is triggered, forming a closed-loop control of the entire process of understanding, decision-making, execution, and feedback, and finally generating exclusive live broadcast experience data.

[0104] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An AI-based method for automatic recognition and adaptation of live streaming scenarios, characterized in that, The method includes: S1. Real-time multimodal data acquisition of the live stream generates visual frame sequences, audio waveforms, text semantic streams, and network state parameters; dynamic region segmentation is performed based on the visual frame sequences, voiceprint feature extraction is performed in combination with the audio waveforms, keyword clustering is performed through the text semantic stream, and network state parameters are fused to generate the original feature set of the live stream scene. S2. Perform multimodal feature fusion processing based on the original feature set of the live streaming scene, construct a structured scene fingerprint, perform scene semantic modeling, identify basic scene types, and further analyze deep semantic features to generate composite scene semantic description data. S3. Based on the semantic description data of the composite scenario, perform dynamic resource demand prediction, obtain the corresponding results and conduct corresponding analysis to generate a resource collaborative configuration scheme. S4. Real-time commands are issued to edge computing nodes to drive the four modules to work together. After local optimization is completed at the edge nodes, key scene features are uploaded to the cloud for global model updates, forming a closed loop of edge and cloud collaborative feedback. The generation accuracy of structured scene fingerprints is continuously optimized through global model updates, generating a dynamically optimized set of scene adaptation parameters. S5. Based on the dynamically optimized scene adaptation parameter set, end-to-end experience generation is performed. Related operations are completed under low latency constraints. The generated results are scored in real time through the experience quality assessment model. When the score is lower than the threshold, the re-identification process is triggered, forming a closed-loop control of the entire process, and finally generating exclusive live streaming experience data.

2. The AI-based automatic live streaming scene recognition and adaptation method according to claim 1, characterized in that, S1 includes: S11. Real-time multimodal data acquisition of the live stream to generate visual frame sequences, audio waveforms, text semantic streams, and network status parameters; S12. Perform synchronous preprocessing on visual frame sequences, audio waveforms, text semantic streams, and network state parameters to complete the timestamp alignment of multimodal data and generate a time-series synchronized multimodal dataset. S13. Based on the temporal synchronous multimodal dataset, extract the inter-frame difference information of the visual frame sequence, perform dynamic region segmentation, and generate a foreground target region set and a background environment region set. S14. Retrieve audio waveforms from the time-series synchronous multimodal dataset, combine them with the time-series change features of the foreground target region set, extract voiceprint features, and generate human voice feature vectors and environmental sound feature vectors. S15. Extract the text semantic stream from the time-series synchronous multimodal dataset, perform keyword clustering based on word segmentation results and word frequency statistics to generate topic semantic clusters; integrate the foreground target region set, background environment region set, human voice feature vector, ambient sound feature vector, topic semantic clusters and network state parameters to perform feature dimension calibration and generate the original feature set of the live streaming scene.

3. The AI-based automatic live streaming scene recognition and adaptation method according to claim 1, characterized in that, The S2 includes: S21. Based on the original feature set of the live streaming scene, perform multimodal feature fusion processing, integrate the information of each dimension by using the feature dimension weighted concatenation method, and generate an initial scene feature matrix; perform dimension reduction and redundancy removal on the initial scene feature matrix to construct a structured scene fingerprint; S22. Construct a semantic association network based on structured scene fingerprints to perform scene semantic modeling; S23. Identify basic scene types by matching a preset scene feature library through a semantic association network; extract temporal change features and interaction behavior features within different basic scene types. S24. Further analyze the deep semantic features by combining temporal change features and interactive behavior features; integrate basic scene types and deep semantic features to generate composite scene semantic description data.

4. The AI-based automatic live streaming scene recognition and adaptation method according to claim 3, characterized in that, S21 includes: S211. Retrieve the original feature set of the live streaming scene, split the features into visual, audio, text and network dimensions, and generate single-dimensional feature subsets; standardize each single-dimensional feature subset, unify the feature value scale, and generate a standardized single-dimensional feature set. S212. Calculate the information contribution of each standardized single-dimensional feature set and generate a feature weight coefficient set; perform weighted processing on the standardized single-dimensional feature set based on the feature weight coefficient set to generate a weighted single-dimensional feature set. S213. Orderly concatenate the weighted single-dimensional feature set, integrate the information of each dimension, and generate an initial scene feature matrix; calculate the linear correlation between features based on the initial scene feature matrix to generate a feature correlation matrix. S214. Based on the feature correlation matrix, filter highly overlapping features to generate a redundant feature list; remove the features corresponding to the redundant feature list from the initial scene feature matrix to generate a deredundant feature matrix. S215. Perform dimensionality compression on the redundant feature matrix, retain the core feature information, and generate a low-dimensional core feature matrix; construct a structured scene fingerprint based on the low-dimensional core feature matrix.

5. The AI-based automatic live streaming scene recognition and adaptation method according to claim 4, characterized in that, S215 includes: Retrieve the redundant feature matrix, calculate the variance contribution value of each feature, and generate a feature variance contribution sequence; sort the features in descending order of the variance contribution sequence, define the core feature screening interval, and generate a core feature candidate set. The core feature candidate set is subjected to a second numerical normalization process to eliminate numerical interference during the dimensionality compression process and generate a normalized candidate feature matrix. The feature space mapping algorithm is used to transform the dimension of the normalized candidate feature matrix, reduce the feature dimension size, and generate a low-dimensional transition feature matrix. Calculate the information overlap between the low-dimensional transition feature matrix and the redundancy-removed feature matrix to generate an information retention assessment result; adjust the dimension mapping parameters based on the information retention assessment result to fill in the key information gaps and generate a low-dimensional core feature matrix. Extract the feature association structure of the low-dimensional core feature matrix, sort out the temporal and logical relationships between the features, and generate a feature association map; integrate the numerical information of the low-dimensional core feature matrix with the structural information of the feature association map to generate a structured scene fingerprint.

6. The AI-based automatic live streaming scene recognition and adaptation method according to claim 1, characterized in that, The S3 includes: S31. Construct a resource demand prediction model based on the semantic description data of composite scenes. Input the dynamic change characteristics of the scene and historical resource configuration data to perform dynamic resource demand prediction and generate the coding complexity index, beauty rendering priority, subtitle generation strategy and director switching frequency. S32. Based on the coding complexity index, analyze the coding resource requirements of different video regions, dynamically allocate the bitrate of the coding module, and generate a differentiated bitrate configuration scheme; based on the priority of beautification rendering, sort the rendering tasks, schedule the computing power of the graphics processing unit, and generate a computing power allocation sequence. S33. Based on the subtitle generation strategy, analyze the temporal rhythm and importance of the text semantic flow, adjust the parameters of the natural language processing model, and generate a subtitle generation parameter set; S34. Analyze the temporal correlation of multiple camera signals according to the director's switching frequency, perform intelligent mixing of multiple camera signals, and generate shot switching sequences and mixing rules. S35, integrates differentiated bitrate configuration schemes, computing power allocation sequences, subtitle generation parameter sets, and shot switching sequences with mixing rules to generate resource collaborative configuration schemes.

7. The AI-based automatic live streaming scene recognition and adaptation method according to claim 6, characterized in that, S32 includes: Retrieve the coding complexity index and video region segmentation data, establish the correspondence between each region and the complexity index, and generate a region complexity correspondence table; based on the region complexity correspondence table, divide the region into three complexity levels: high, medium, and low, and generate a region complexity rating result. Based on different complexity levels, a base bitrate range is set, and combined with the area ratio of the region, a regional base bitrate set is generated; real-time bandwidth data of live broadcast transmission is collected, bandwidth fluctuation amplitude is calculated, and bitrate adjustment coefficient is generated. The values ​​in the regional base bitrate set are corrected using a bitrate adjustment factor to obtain the target bitrate for the adapted bandwidth of each region, and a list of regional target bitrates is generated. Integrate the regional target bitrate list to generate a differentiated bitrate configuration scheme; retrieve the beautification rendering priority data and the list of rendering tasks to be executed, establish a mapping relationship between tasks and priorities, and generate a task priority comparison table; The rendering tasks are sorted in descending order of priority to generate an ordered task queue; the rendering parameters and effect requirements of each task in the ordered task queue are analyzed to estimate the required computing resources and generate the task computing power requirement value. Monitor the current operating status of the graphics processing unit, count the occupied and remaining computing power, and generate the total available computing power; combine task priority and computing power requirement value, allocate available computing power resources to each task proportionally, and generate task computing power allocation results; based on the order of the ordered task queue, integrate the computing power allocation results of each task to generate a computing power allocation sequence.

8. The AI-based automatic live streaming scene recognition and adaptation method according to claim 1, characterized in that, The S4 includes: S41. Based on the resource collaborative configuration scheme, standardized control instructions are generated and real-time instructions are issued to edge computing nodes. The edge computing nodes parse the standardized control instructions and drive the encoding module, beautification module, subtitle module and directing module to work together to complete the real-time processing of the live stream. S42. After completing local optimization at the edge nodes, extract key scene features and resource consumption data during the processing to generate edge node processing feedback data. S43. Upload the edge node processing feedback data to the cloud. The cloud combines the feedback data from multiple edge nodes to update the global model and correct the scene recognition and resource prediction parameters. S44. Form a closed loop of edge and cloud collaborative feedback, and continuously optimize the generation accuracy of structured scene fingerprints through global model updates; generate dynamically optimized scene adaptation parameter sets based on the optimized structured scene fingerprints.

9. The AI-based automatic live streaming scene recognition and adaptation method according to claim 1, characterized in that, The S5 includes: S51. Based on the dynamically optimized scene adaptation parameter set, analyze the end-to-end experience generation requirements and start encoding quality adjustment, beauty effect adaptation, dynamic subtitle generation and broadcast screen switching under low latency constraints. S52. After completing the end-to-end experience generation, generate initial live broadcast experience data; call the experience quality assessment model to score the initial live broadcast experience data in real time, and quantify the clarity of the live broadcast picture, audio stability, subtitle accuracy and broadcast smoothness. S53. Compare the scoring results with the preset threshold. When the score is lower than the threshold, trigger the re-identification process. The re-identification process goes back to the real-time acquisition stage of multimodal data, reacquires the live stream data and completes subsequent processing to correct the adaptation deviation. S54. Establish a closed-loop control system for the entire process of understanding, decision-making, execution, and feedback, and continuously iterate and optimize the processing effect of each link; ultimately generating exclusive live streaming experience data.

10. A system for implementing the AI-based automatic live streaming scene recognition and adaptation method as described in claim 1, characterized in that, The system includes: Data acquisition module: performs real-time multimodal data acquisition of the live stream, generating visual frame sequences, audio waveforms, text semantic streams, and network state parameters; performs dynamic region segmentation based on visual frame sequences, extracts voiceprint features by combining audio waveforms, performs keyword clustering through text semantic streams, and integrates network state parameters to generate the original feature set of the live stream scene; Feature fusion module: Performs multimodal feature fusion processing based on the original feature set of the live streaming scene, constructs a structured scene fingerprint, performs scene semantic modeling based on the structured scene fingerprint, identifies basic scene types, and further analyzes deep semantic features to generate composite scene semantic description data; Demand forecasting module: Performs dynamic resource demand forecasting based on semantic description data of complex scenarios, obtains corresponding results and performs corresponding analysis, and generates resource collaborative configuration schemes; Dynamic optimization module: It issues real-time instructions to edge computing nodes through a resource collaboration configuration scheme, driving the four modules to work together. After the edge nodes complete local optimization, the key scene features are uploaded to the cloud for global model update, forming a closed loop of edge and cloud collaboration feedback. The global model update continuously optimizes the generation accuracy of structured scene fingerprints and generates a dynamically optimized set of scene adaptation parameters. Closed-loop control module: Generates end-to-end experience based on dynamically optimized scene adaptation parameter set, completes related operations under low latency constraints, scores the generated results in real time through experience quality assessment model, and triggers re-identification process when the score is lower than the threshold, forming a closed-loop control of the entire process of understanding, decision-making, execution and feedback, and finally generating exclusive live broadcast experience data.

Images