Live broadcast content real-time recommendation method based on multi-modal scene perception

Abstract

The invention relates to the technical field of live broadcast content recommendation, and discloses a live broadcast content real-time recommendation method based on multi-modal scene perception. The method comprises the following steps: collecting a visual data stream, an acoustic data stream and an audience interaction text stream of a live broadcast stream in real time, and identifying the type of a live broadcast scene through a scene analyzer; respectively constructing a visual feature extractor, an acoustic feature extractor and a text feature extractor, and generating an object distribution thermodynamic diagram, an emotion spectrum feature and a keyword density matrix; fusing the three types of features to form a dynamic scene feature space, and generating an initial recommendation set based on a training scene adaptive recommendation model; and in combination with real-time audience portrait attributes, adjusting model weight parameters, outputting a final recommended content identifier, and executing switching operation by a content scheduling engine. According to the method, through multi-modal data fusion and dynamic adaptation, the real-time performance and accuracy of live broadcast content recommendation are improved, and the personalized requirements of audiences are better met.

Description

Technical Field

[0001] This invention relates to the field of live streaming content recommendation technology, specifically a real-time live streaming content recommendation method based on multimodal scene perception. Background Technology

[0002] With the rapid development of the online live streaming industry, users' personalized needs for live streaming content are becoming increasingly prominent. Live streaming platforms face the challenge of accurately and in real-time pushing content that matches viewers' interests. Current mainstream live streaming content recommendation methods mostly rely on single-dimensional data analysis, such as making recommendations based solely on viewers' historical viewing records or interactive behaviors. This approach struggles to comprehensively capture the dynamic changes in the live streaming scenario. In visual information processing, traditional methods are often limited to identifying single objects in a live stream, ignoring the spatial relationships between multiple objects. This makes it difficult to accurately determine the core content of the live stream scene. For example, in an outdoor adventure live stream, only the streamer or specific scenery may be identified, but the overall environment and the relationship between the activity and the content may not be grasped, resulting in a low degree of matching between recommended content and the current scene. The application of acoustic data also has shortcomings. Existing technologies mostly focus on basic parameters such as volume and frequency of sound, failing to delve deeper into the emotional information they contain. Voice tone and background sound effects in live streams often convey different emotions such as joy, tension, and calm, and these emotional factors have a significant impact on viewers' preferences. Due to the lack of effective extraction and utilization of emotional spectrum features, recommendation systems struggle to adjust their recommendation strategies based on the emotional atmosphere of the live stream, making it difficult to meet viewers' content needs in specific emotional states. The processing of audience interaction text also has limitations. Currently, most methods rely on simple keyword counting or frequency statistics, failing to construct a matrix model that reflects the importance and distribution patterns of keywords. Audience comments and other interactive text contain a wealth of information about content preferences and real-time feedback, but a single counting method cannot fully represent the inherent relationships between this information. This makes it difficult for recommendation systems to accurately grasp the real-time needs and interest changes of the audience, resulting in recommendation results that are lagging behind or deviate from the audience's expectations. Existing recommendation models lack a dynamic adjustment mechanism when combining live streaming scenario features with audience profiles. Audience profile attributes change with viewing behavior and over time, and models with fixed weights cannot respond to these changes in a timely manner. This results in recommended content failing to adapt to the latest needs of the audience in real time, further reducing the accuracy and effectiveness of recommendations. Summary of the Invention

[0003] The purpose of this invention is to provide a real-time recommendation method for live streaming content based on multimodal scene perception, so as to solve the problems mentioned in the background art.

[0004] To achieve the above objectives, the present invention provides a real-time recommendation method for live streaming content based on multimodal scene perception, the method comprising: The system collects visual data streams, acoustic data streams, and audience interaction text streams from the live stream in real time, and identifies the live scene category through a scene parser. A visual feature extractor is constructed to process visual data streams and generate object distribution heatmaps; an acoustic feature extractor is constructed to process acoustic data streams and generate emotional spectrum features; and a text feature extractor is constructed to process audience interactive text streams and generate keyword density matrices. By integrating object distribution heatmaps, emotional spectrum features, and keyword density matrices, a dynamic scene feature space is formed. A scene-adaptive recommendation model is trained based on the dynamic scene feature space to generate an initial set of live streaming content recommendations. Obtain real-time audience profile attributes, adjust the weight parameters of the scene adaptation recommendation model based on the audience profile attributes, use the scene adaptation recommendation model with adjusted weights to output the final recommended content identifier, and input the final recommended content identifier into the content scheduling engine to execute the live content switching operation.

[0005] Preferably, the scene parser identifies the following live streaming scene categories: A spatiotemporal convolutional network is used to parse the frame sequence features of the visual data stream, a long short-term memory network is applied to capture the temporal dependencies of the acoustic data stream, and an attention mechanism network is used to extract the semantic focus of the audience interactive text stream. The frame sequence features, temporal dependencies and semantic focus are combined to output scene classification labels.

[0006] Preferably, the step of constructing a visual feature extractor to process the visual data stream and generate an object distribution heatmap includes: Load a pre-trained object detection network to segment entity regions in the visual data stream, calculate the spatial distribution density of entity regions per unit time, and map the spatial distribution density to a two-dimensional coordinate plane to form a heat map.

[0007] Preferably, the fused object distribution heatmap, sentiment spectrum features, and keyword density matrix include: The object distribution heatmap is processed by performing region normalization, and the emotional spectrum features are converted into feature vectors that match the dimensions of the region normalization result. The timestamps of the keyword density matrix are aligned with the time axis of the object distribution heatmap, and the region normalization result, feature vectors and timestamp-aligned keyword density matrix are spliced ​​together.

[0008] Preferably, the scene adaptation recommendation model trained based on the dynamic scene feature space includes: A deep belief network is configured as the basic architecture of the recommendation model. The deep belief network is trained by inputting historical dynamic scene feature space samples. During the iterative optimization process, the network parameters are adjusted using a contrastive loss function. The network parameters are frozen when the accuracy of the validation set reaches the preset standard.

[0009] Preferably, the weight parameters of the scene-adaptive recommendation model adjusted according to audience profile attributes include: The set of preference labels in the audience profile attributes is analyzed, the correlation matrix between the set of preference labels and the dynamic scene feature space is calculated, the scene adaptation weight coefficient is generated based on the correlation matrix, and the scene adaptation weight coefficient is applied to the output layer of the deep belief network.

[0010] Preferably, the scenario-adaptive recommendation model using adjusted weights includes: The real-time dynamic scene feature space is input into the weighted deep belief network, and the probability distribution of recommended content is generated through network forward propagation. The set of content identifiers corresponding to the peak of the probability distribution is selected.

[0011] Preferably, after generating the initial live content recommendation set, the method further includes: A real-time feedback channel is established to receive audience dwell time data. When the audience dwell time is lower than a preset threshold, the model optimization module is triggered. The model optimization module uses the artificial bee colony algorithm to update the deep belief network parameters.

[0012] Preferably, the set of content identifiers corresponding to the peak values ​​of the selected probability distribution includes: Set a dynamic probability threshold to filter low-probability content identifiers, sort the remaining content identifiers by probability value to generate a priority queue, and extract the top N identifiers from the priority queue to form the final recommended content identifiers.

[0013] Preferably, the content scheduling engine performs the live content switching operation including: The system matches the final recommended content identifier with the physical address of the content resource library, preloads the target live content stream through the content delivery network, and sends a switching command to the player terminal to perform a seamless transition.

[0014] Compared with the prior art, the beneficial effects of the present invention are: By integrating data from visual, acoustic, and audience interaction text, a more comprehensive and in-depth understanding of the live streaming scene is achieved. Real-time acquisition of these three data streams and subsequent feature extraction ensures that the analysis of the live streaming scene is no longer limited to a single dimension, capturing the dynamic characteristics of the scene from multiple perspectives, including visual content, auditory emotion, and audience feedback. The object distribution heatmap generated by the visual feature extractor clearly presents the spatial distribution and importance of each object in the image, helping to accurately identify the core scenes and activities of the live stream and providing a basis for subsequent recommendations that align with the actual scene. The emotional spectrum features generated by the acoustic feature extractor delve into the emotional information contained in sound, enabling it to keenly capture changes in the emotional atmosphere during a live stream. Whether it's upbeat music, tense commentary, or calm conversation, these emotional elements can be effectively extracted and integrated into scene feature analysis, allowing recommended content to better match the emotional tone of the live stream and enhance the viewer's immersion and viewing experience. The keyword density matrix constructed by the text feature extractor breaks through the limitations of traditional keyword statistics, comprehensively reflecting the distribution patterns and importance levels of keywords in audience interactive text. Key preferences and real-time feedback from audience interactions such as bullet comments and replies are systematically analyzed, enabling the recommendation system to accurately grasp changes in audience interests and needs, and making recommended content more closely aligned with audience real-time expectations. The formation of a dynamic scene feature space organically integrates the three types of features, achieving complementarity and enhancement of multi-dimensional information and constructing a richer and more comprehensive scene description. This feature space can be dynamically updated in real time, accurately reflecting every subtle change in the live streaming scene, providing highly realistic input data for the recommendation model, and improving the real-time performance and accuracy of recommendations. The scene-adaptive recommendation model, trained on a dynamic scene feature space, generates an initial recommendation set that is highly matched to the current scene. By adjusting the model's weight parameters incorporating real-time audience profile attributes, the recommendation process fully considers individual differences, achieving precise adaptation from scene commonality to audience individuality. This dynamic adjustment mechanism ensures that the recommendation model can respond promptly to changes in audience profiles; whether it's a shift in audience interests or the formation of new viewing preferences, the model captures and reflects these changes in the recommendation results, ensuring that the final recommended content remains synchronized with the latest needs of the audience. Finally, the recommended content identifier is input into the content scheduling engine to execute the switching operation, ensuring that the recommended content can be presented to the audience quickly and accurately, reducing recommendation latency and improving the smoothness of live content switching. Overall, this method, through deep fusion and dynamic adaptation of multimodal data, makes live content recommendations more targeted, real-time, and accurate, effectively enhancing the interactivity and stickiness between the audience and the live content, and bringing better content distribution results to the live streaming platform. Attached Figure Description

[0015] Figure 1 This is a schematic diagram illustrating the working principle of the real-time live content recommendation method based on multimodal scene perception described in this invention. Figure 2 A flowchart for the scene parser to identify live scene categories; Figure 3 This is a multimodal feature processing and scene classification diagram; Figure 4A flowchart for generating object distribution heatmaps for a visual feature extractor; Figure 5 For feature extraction and fusion spatial graph; Figure 6 Flowchart for training a recommendation model in a dynamic scene feature space; Figure 7 A flowchart for generating the final recommendation for dynamic threshold filtering. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Please see Figure 1 This invention provides a real-time recommendation method for live streaming content based on multimodal scene perception, the method comprising: By real-time acquisition of visual data streams, acoustic data streams, and audience interaction text streams from the live stream, a scene parser is used to identify the live stream scene category. A visual feature extractor processes the visual data stream to generate an object distribution heatmap, an acoustic feature extractor processes the acoustic data stream to generate sentiment spectrum features, and a text feature extractor processes the audience interaction text stream to generate a keyword density matrix. The object distribution heatmap, sentiment spectrum features, and keyword density matrix are fused to form a dynamic scene feature space. Based on this feature space, a scene-adaptive recommendation model is trained to generate an initial set of recommended live stream content. Audience profile attributes are acquired in real-time, the weight parameters of the scene-adaptive recommendation model are adjusted, and the final recommended content identifier is output. The content scheduling engine then executes the live stream content switching operation.

[0018] Example 1: See Figure 2 The scene parser identifies live streaming scene categories, simultaneously processing three heterogeneous data streams and outputting a unified classification result. After the visual data stream is input, the spatiotemporal convolutional network first samples the frame sequence, extracting keyframes at fixed intervals to form a temporally continuous image sequence. The network architecture employs a multi-layer three-dimensional convolutional kernel structure, with each kernel simultaneously sliding operations in both spatial and temporal dimensions: detecting local feature patterns in a single frame spatially and capturing motion trajectory changes across frames temporally. The output of the convolutional layers is transformed by an activation function and then fed into pooling layers, preserving salient features while reducing dimensionality. After multiple convolutional pooling operations, the generated spatiotemporal feature map reflects the dynamic behavior patterns of objects in the live stream, such as the running trajectories of athletes in sports events or the rotating display of goods in e-commerce live streams.

[0019] The acoustic data stream processing employs a Long Short-Term Memory (LSTM) network architecture, which includes memory units and a gating mechanism. The raw audio signal is pre-emphasized, framed, and windowed before being converted into a Mel-spectrum graph and input into the network. Input gates control the storage weights of current acoustic features, forget gates determine the retention ratio of historical acoustic features, and output gates adjust the output intensity of the feature vector. The network transmits historical information through its internal states, effectively capturing long-term dependencies in the audio signal, such as interpreting the semantic coherence of speech or the rhythmic continuity of background music. The hidden layer states of the network are continuously updated over time, and the final output acoustic feature vector contains the temporal evolution of sound events, enabling the identification of fluctuations in the frequency of cheers at a concert or variations in intonation during a live educational session.

[0020] The interactive text stream is processed through an attention mechanism network. The original bullet screen text is segmented, stop words are removed, and then converted into a sequence of word vectors which are input into the network. The attention layer calculates the relevance weight of each word vector to its context: a trainable parameter matrix maps the word vectors to query vectors, key vectors, and value vectors; the query vector is multiplied by all key vectors to generate an attention score; the score is standardized and used as weights to sum the value vectors. This mechanism allows the network to automatically focus on high-frequency keywords and emotionally charged expressions, such as proper nouns related to breaking news or interactive words with exclamatory tones. The network outputs a semantic focus vector, effectively compressing redundant text information while preserving core semantic features.

[0021] The three feature vectors are integrated in the fusion layer. The video feature map output by the spatiotemporal convolutional network is flattened and converted into feature vectors, which are then aligned with the acoustic feature vectors generated by the long short-term memory network and the semantic focus vectors generated by the attention mechanism network according to a fixed dimension. The feature vectors are concatenated to form a joint feature representation, which is then input into the fully connected layer for nonlinear transformation. The transformed feature vectors enter the multi-class classifier, where the probability distribution of each scene category is calculated using the Softmax function. The output layer sets a scene label threshold; the category with the maximum probability value exceeding the threshold is determined as the current scene category, covering major live streaming types such as sports competitions, stage performances, product promotions, and classroom teaching. The entire parsing process adopts a pipelined architecture to achieve parallel processing of multimodal data, with processing latency controlled within 40 milliseconds to meet real-time requirements. The classification result also serves as a timestamp synchronization benchmark, providing a temporal alignment basis for subsequent feature extraction modules.

[0022] See Figure 3This demonstrates the core processing flow of multimodal scene analysis, encompassing four key analytical dimensions. The spatiotemporal visual feature heatmap in the upper left corner reveals the dynamic changes of objects in the live stream; the vertical axis represents continuous time steps, the horizontal axis displays different visual feature dimensions, and grayscale depth reflects feature activation intensity. The acoustic sentiment line graph in the upper right corner depicts the changing trends of five main emotions over time; the fluctuations of each line correspond to the fluctuations in narration tone or background music, such as the peak emotional intensity during moments of excitement, with different grayscale lines representing different emotion types. The text attention histogram in the lower left corner quantifies the importance weights of different keywords; higher bars indicate words more decisive for scene classification. The scene classification probability in the lower right corner displays the system's judgment criteria for the current live stream scene; the probability differences between different scenes reflect the system's confidence in recognizing scenes such as sports competitions and stage performances.

[0023] Example 2: See Figure 4 This system encompasses the extraction and fusion of visual, acoustic, and textual features, constructing a unified dynamic scene feature space. The visual feature extractor processes the visual data stream based on a pre-trained object detection network architecture. This network uses a region proposal network to generate initial candidate boxes, setting multiple anchor boxes on the convolutional feature map, and adjusting the anchor positions and sizes through bounding box regression. After non-maximum suppression filtering, the candidate boxes are input into the region of interest pooling layer, converting candidate regions of different sizes into fixed-length feature vectors. The feature vectors are processed in two ways: a classification branch outputs the entity category probability distribution, and a regression branch optimizes the candidate box coordinate parameters. The detection results label the category, location, and size information of entity objects in the image, forming an object metadata sequence.

[0024] The spatial distribution of entities within a unit time window is analyzed by dividing the live stream into a uniform grid and calculating the frequency of occurrence of each type of entity within each grid cell. A kernel density estimation algorithm is applied to the location points of entities of the same category, and a probability density surface is formed through Gaussian kernel smoothing. This surface is then color-coded and mapped onto a two-dimensional coordinate plane to generate an object distribution heatmap. In the heatmap, red areas represent high-density distribution areas, such as the center of the field where players often congregate in a football match; blue areas represent low-density areas, such as the edge of the field. In e-commerce live streams, this might manifest as a high-intensity heatmap in the product display area.

[0025] The acoustic feature extractor processes the raw audio stream, pre-emphasizing and compensating for high-frequency components before frame-by-frame windowing to block spectral leakage. Each frame's signal is transformed to the frequency domain via Fourier transform, then processed by a Mel filter bank to simulate human hearing characteristics, outputting Mel spectral coefficients to form a time-frequency two-dimensional matrix. The emotion recognition model uses convolutional layers to extract local spectral patterns, recurrent layers to capture long-term contextual relationships, and fully connected layers to output the probability distribution of emotion labels. The Mel spectral matrix and the emotion probability vector are concatenated along the channel dimension to form a time-aligned emotion spectral feature matrix. Changes in the time dimension of the matrix reflect emotional fluctuations, such as a narrator's enthusiastic tone corresponding to a synchronous surge in sound intensity and high-frequency energy.

[0026] The text feature extractor collects interactive text streams such as bullet comments and messages. Text preprocessing includes word segmentation, stop word removal, and stemming, constructing a vocabulary to record effective keywords. A sliding time window mechanism is used to statistically analyze the word frequency of keywords within a fixed interval. An inverse document frequency factor is introduced to suppress the weight of common words, and the calculation formula assigns higher scores to professional terms. An initial frequency matrix is ​​constructed using time windows as rows and keywords as columns. Row normalization eliminates differences in time window lengths, ultimately generating a keyword density matrix. The row vectors of the matrix reflect the semantic focus at specific moments; for example, in sports events, keywords such as "shoot" and "score" show peak density at key moments.

[0027] Preprocessing operations are performed before the three-way feature fusion. The object distribution heatmap undergoes region normalization: the screen is divided into several equal-area regions, the heatmap mean of each region is calculated, and differences in screen proportions across different live streaming scenarios are eliminated. The emotional spectrum feature matrix is ​​converted into a fixed-dimensional feature vector via a fully connected neural network, maintaining dimensionality compatibility with the normalized heatmap. The time synchronization subsystem uses the video stream timestamp as a reference and aligns the time axes of the acoustic feature matrix and text density matrix using a linear interpolation algorithm, controlling the error to millisecond-level precision.

[0028] In the feature concatenation stage, the normalized heatmap serves as the spatial feature base, the transformed sentiment feature vector serves as the attribute feature supplement, and the time-axis aligned keyword density matrix serves as the semantic feature extension. These three elements are concatenated along the feature channel dimension to form a three-dimensional feature data block, constituting a complete dynamic scene feature space. The spatial dimension of the data block corresponds to the screen grid division scheme, the temporal dimension includes continuous time slices, and the channel dimension integrates three types of information features: visual distribution, acoustic sentiment, and textual semantics. In a sports live broadcast scenario, this feature space might appear as follows: the heatmap shows athletes densely distributed in the penalty area (spatial feature), the sentiment spectrum detects increased excitement (acoustic feature), and the bullet screen matrix shows high-frequency words for "penalty kick" (textual feature), collectively representing the state of key moments in the match. After encapsulation, the data block is fed into the downstream recommendation model, providing a multimodal joint representation basis for content decision-making.

[0029] See Figure 5 This section showcases the key data structures for multimodal feature extraction and fusion. The object distribution heatmap in the upper left corner uses an 8×8 grid to divide the live stream screen. The grayscale value of each cell represents the density of objects in that area, with darker areas corresponding to areas where athletes congregate or product displays are prominent. The sentiment spectrum matrix in the upper right corner vertically displays six sentiment types, while the horizontal axis represents the time series. Changes in grayscale within the matrix reflect the dynamic transition of emotional states, such as the transition from calm to excitement. In the keyword density matrix in the lower left corner, the vertical axis represents time progression, and the horizontal axis corresponds to different keyword indices. The density of points reflects changes in the focus of audience discussion at specific moments. The fused feature space slice in the lower right corner displays the spatial distribution features after multimodal data integration. This data structure unifies the encoding of visual distribution, acoustic sentiment, and textual semantics, providing a joint representation basis for content recommendation. Darker areas in the figure represent spatial locations with strong feature responses, corresponding to the core attention areas of the live stream screen.

[0030] Example 3: See Figure 6 This section describes the construction and personalized adjustment process of a scene-adaptive recommendation model. The Deep Belief Network (DBN), as the foundational architecture of the recommendation model, is a deep structure formed by stacking multiple Restricted Boltzmann Machines (RBMs). Each RBM contains a visible layer and a hidden layer; nodes in the visible and hidden layers are fully connected, while nodes within the same layer are unconnected. The training process employs a layer-by-layer greedy algorithm. After the bottom-level RBMs are trained, their parameters are fixed, and their hidden layer outputs serve as the input to the visible layer of the upper-level RBMs. A Softmax classifier is added to the top layer of the network, forming a complete Deep Belief Network structure. When historical dynamic scene feature space samples are input into the network, the visible layer receives the feature vectors and updates the network parameters using a contrastive divergence algorithm. During the parameter update process, the state of the visible layer nodes is calculated using an activation function.

[0031] in: This represents the state of the i-th node in the visible layer. This represents the state of the j-th node in the hidden layer. For connection weights, For the visible layer bias term, The activation function is Sigmoid. Network training uses a triplet loss function to construct the comparison relationship between samples. Each training sample contains an anchor sample, a positive sample, and a negative sample. Anchor samples are feature vectors from the target scene, positive samples are feature vectors from similar scenes, and negative samples are feature vectors from dissimilar scenes. The loss function drives the network to learn feature representations that reduce the distance between anchor samples and positive samples, and increase the distance between anchor samples and negative samples. An early stopping mechanism is implemented during training; training is terminated when the recommendation accuracy on the validation set no longer improves for several consecutive training epochs to prevent overfitting. Finally, the network parameters are frozen after training, forming the basic recommendation model.

[0032] The audience profile attribute analysis module processes user registration information, historical behavior records, and social relationship data. Registration information includes demographic characteristics, historical behavior records include viewing time, interaction frequency, and content preferences, and social relationship data involves following lists and friend networks. This raw data is cleaned and transformed to generate a structured set of preference tags. Each tag in the tag set is associated with a weight value, reflecting the strength of the user's preference for that feature. When calculating the correlation between the preference tag set and the dynamic scene feature space, a tag-feature matrix M is constructed, where rows correspond to preference tags and columns correspond to scene features. Matrix elements... The association strength between the i-th label and the j-th scene feature is represented by a statistical analysis method. Matrix factorization decomposes the label-feature matrix into a low-rank approximation:

[0033] Where: P is the label latent factor matrix, and Q is the feature latent factor matrix. The main feature vectors are extracted from the decomposition results to form the scene adaptation weight matrix W, whose dimension matches the dimension of the deep belief network output layer. During the weight adjustment phase, the weight matrix W is performed with the network output layer activation values ​​A using the Hadamard product:

[0034] Calculation result The personalized recommendation score enhances the weight of content matching user preferences and suppresses the recommendation score of irrelevant content. The adjustment process preserves the relative ranking of the original recommendation scores, fine-tuning the recommendation strength only within local ranges. The personalized recommendation score is input into the Softmax function to generate the final recommendation probability distribution. This distribution reflects both the characteristics of the live streaming scenario and incorporates user personalized preferences, achieving dual adaptation between the scenario and the user. The model update mechanism periodically collects user feedback data; when user behavior patterns change significantly, the label-feature matrix is ​​recalculated and the weight matrix is ​​updated to maintain the dynamic adaptability of the recommendation system. The entire recommendation process is completed within milliseconds, meeting the real-time requirements of live streaming scenarios. Incremental learning is used for network parameter adjustment to avoid the computational overhead of retraining and ensure stable system operation. Recommendation results are output in the form of content identifiers, providing a basis for subsequent content scheduling decisions.

[0035] Example 4: Focusing on the operation and real-time optimization mechanism of the recommendation model after weight adjustment, covering the entire process from dynamic scene feature processing to content recommendation. In the initial stage, the dynamic scene feature space collected in real time serves as the input source to the weighted deep belief network. The feature space is composed of multimodal features: a normalized object distribution heatmap reflects the distribution pattern of entities in the scene; a transformed sentiment spectrum feature vector carries acoustic sentiment information; and a time-synchronized keyword density matrix contains the semantic focus of the text. The feature space is encoded by the input layer and passed to the hidden layer, where forward propagation computation is performed. The hidden layer consists of multiple restricted Boltzmann mechanisms, with each node calculating its output value through an activation function and passing it to the next layer. The propagation process involves multiple levels of nonlinear transformations, gradually abstracting higher-order feature patterns. For example, in a live sports event scenario, the heatmap shows players highly clustered in the penalty area, the sentiment vector detects a sharp increase in cheering intensity, and the keyword matrix shows the high-frequency word "penalty kick." These features are integrated and encoded into a high-dimensional feature representation. The output layer uses the Softmax function to process the final hidden layer output, calculating the probability distribution value of each candidate content. The probability value reflects the overall suitability of the content with the current scene and audience profile, and its distribution ranges from 0 to 1. The peak value indicates the best matching content. After the probability distribution is calculated, the system selects the set of content identifiers corresponding to the peak value as the recommendation output.

[0036] The real-time feedback channel continuously monitors audience behavior data, collecting three core metrics through user behavior logs: dwell time (the duration a viewer spends watching the current recommended content page); interaction frequency (the number of comments sent or likes); and exit flag (whether a viewer prematurely ends their viewing). This data is recorded with millisecond precision, forming a time-series behavior dataset. After the feedback data is collected, it is input into the decision engine. When the dwell time is detected to be below a preset threshold (e.g., 40 seconds for sports events), the system automatically triggers the model optimization module. The threshold setting depends on the scenario type; for example, a longer threshold (e.g., 90 seconds) is used for education and training to accommodate in-depth content consumption habits. When behavior data triggers a decision, the system generates standardized records including timestamps, dwell time, interaction frequency, and exit flags. Table 1 shows a segment of audience behavior data collected in a football match live stream scenario.

[0037] Table 1: Data snippets of audience behavior collected during live football match broadcasts.

[0038] Timestamp Duration of stay (seconds) Interaction frequency (times / minute) Exit flag 20230807T142030 38 5 yes 20230807T142045 27 2 yes 20230807T142100 52 8 no 20230807T142115 31 3 yes In the table data, records with a dwell time of less than 40 seconds trigger the optimization process (such as the first and third rows of data). The exit flag is "Yes" to indicate that the user has stopped watching, providing a direct signal for model correction.

[0039] The Artificial Bee Colony Algorithm performs model optimization tasks by simulating bee colony foraging behavior and updating deep belief network parameters in stages. In the initial stage, a parameter space is defined, with each bee position corresponding to a weight or bias parameter in the network. In the "hired bees" stage, an initial position neighborhood is randomly selected, and a fitness function value is calculated: the function is defined based on current behavioral feedback, combined with metrics such as dwell time and interaction frequency to construct a loss value. A smaller loss value indicates better network performance. For example, when a football live stream shows a low dwell time, the algorithm calculates the trend of loss value changes within the location's neighborhood. In the "observation bees" stage, optimization directions are selected based on fitness: high-fitness bee positions attract more observation bees to search, exploring new solutions through the position update formula. In the "scout bees" stage, local optima are addressed: if the position loss value fails to improve after multiple consecutive iterations, the bee's position is randomly reset to break the deadlock. The iterative process continuously optimizes network parameters, such as adjusting the hidden layer activation threshold or the output layer scaling factor. After the algorithm finishes, network parameters are updated to enhance the model's responsiveness to dynamic changes in viewer preferences. The entire process has a latency controlled within milliseconds, ensuring the continuity of real-time recommendations. After optimization, the model continues to receive new feature space inputs, generates subsequent recommendation probability distributions, and forms a closed-loop feedback system.

[0040] In a specific example, taking the live broadcast of the European Championship football final as an example: when the match enters the penalty shootout, the system inputs a dynamic scene feature space, including a high-density area for the goalkeeper in the heatmap, an increased acoustic excitement index in the emotional spectrum, and a keyword matrix highlighting the high-frequency word "save". The weight adjustment model outputs a recommendation probability distribution, with the peak content being "slow-motion replay". Viewer behavior data is shown in the table; some users' short dwell time triggers optimization; after adjusting network parameters using the artificial bee colony algorithm, the new recommendation probability distribution favors "tactical analysis" content, improving user retention. The recommendation output is finally converted into content identifiers and handed over to the downstream engine for processing. The entire implementation ensures a balance between real-time performance and adaptability, with the processing rate matching the live data flow rate.

[0041] Example 5: See Figure 7 This system encompasses the post-processing of recommendation results and the content scheduling execution stage. The input is the probability distribution data output by the scene-adaptive recommendation model after weight adjustments. After the probability distribution of recommended content is generated, a dynamic probability threshold filtering operation is performed. The system automatically sets a probability threshold value based on the real-time identified live streaming scene category. This threshold is scene-adaptive. A higher threshold is set for sports events to filter low-probability results, as events in this type of scenario change rapidly and audience attention is focused, requiring accurate recommendations of key content. A lower threshold is used for education and training to retain more alternatives, as knowledge transfer requires supplementary materials to expand understanding. A medium threshold is set for concert live streaming scenarios to balance the need for continuity of works and performance diversity. The threshold value dynamically fluctuates with the progress of the live stream; for example, in a football match, the threshold is raised to a peak level during crucial moments and appropriately lowered during halftime. The remaining content identifiers after filtering form a temporary result set, with each identifier associated with a probability value and metadata tags.

[0042] The temporary result set is processed by sorting the probability values ​​in descending order. The comparison algorithm uses a two-pointer quicksort mechanism, where the pointers are swapped during movement to maintain the order. The algorithm's time complexity is optimized to linear logarithmic level, processing a queue of thousands of identifiers in less than milliseconds. A priority queue structure is generated, with the highest probability identifier stored at the head, decreasing sequentially to the lowest valid probability identifier at the tail. The queue data structure is implemented using a max-heap, supporting efficient access and deletion of the head element. The first N elements of the queue are extracted to form the final recommendation set, with N dynamically calculated. The calculation factor includes the real-time load status of the content resource library, the bandwidth margin of edge nodes, and the performance parameters of the playback terminal device. The load balancer periodically pushes resource library status codes; high-order status codes trigger a strategy to reduce the N value; when bandwidth monitoring shows available margin, the range of N values ​​is appropriately expanded. The final recommendation set is encapsulated as a structured data packet, containing a list of identifiers, corresponding probability weights, and recommendation validity period.

[0043] After receiving the recommendation data packet, the content scheduling engine performs address resolution. It retrieves the distributed hash table of the content resource library and queries the physical storage location of the target live content using an identifier. The hash table uses a consistent hashing algorithm to organize data nodes, mitigating mapping disturbances caused by node changes. The physical address includes the edge node IP, storage path, and content fragment index. The query result triggers a preloading instruction from the content distribution network, selecting the optimal edge node group to construct a transmission path. The transmission path design considers the network topology, prioritizing jumps between nodes within the same city, followed by cross-regional backbone nodes. The preloading process employs a tiered buffering strategy: the first stage loads keyframe data blocks, and the second stage supplements the full data. The cache management module monitors the memory usage of edge nodes and evicts low-value cached content according to a priority queue.

[0044] The player terminal receives switching control commands and executes transition operations. These commands include the target stream address, switching timestamp, and audio transition strategy. The switching time window selects the keyframe position of the live stream to avoid image degradation caused by intra-frame prediction residuals. The player maintains a dual-buffer mechanism: the original buffer continues to output the current frame sequence, while the background buffer asynchronously loads the target live stream data. When the buffer progress reaches the switching threshold, the frame sequence is replaced at the video keyframe boundary. Audio track processing employs cross-fading technology, maintaining a stable level on the original audio track while the target track gradually enters, with the transition period controlled within a range imperceptible to human hearing. Visually, a hard switch is performed to ensure instantaneous switching of image content, avoiding blurry transitions that could interfere with the transmission of key information. A switching completion event triggers a feedback signal to the recommendation system, forming an operational closed loop.

[0045] A specific implementation example occurred in a basketball live stream scenario: during the final two minutes of the game when the score was tied, the probability distribution of the recommended model output showed that the identifier for "Star Player Statistics" had a probability of 0.89, and the identifier for "Slow Motion Replay" had a probability of 0.78. The dynamic threshold was set to 0.75, and the identifier for "Game Preview" (0.70) was filtered out. The queue was generated with statistics at the head, and while the calculated N value was 3, only two items were actually extracted due to resource limitations. The scheduling engine located the statistics in edge node group A-03 and initiated a preload of 200ms keyframe data blocks. The switching command was executed at the timer's pause point, and the player switched the frame on the scoreboard screen while the background audio commentary remained uninterrupted. The entire data processing latency was controlled within 80 milliseconds, and the operation precisely matched the characteristics of the live stream.

[0046] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0047] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A real-time recommendation method for live streaming content based on multimodal scene perception, characterized in that, include: The system collects visual data streams, acoustic data streams, and audience interaction text streams from the live stream in real time, and identifies the live scene category through a scene parser. A visual feature extractor is constructed to process visual data streams and generate object distribution heatmaps; an acoustic feature extractor is constructed to process acoustic data streams and generate emotional spectrum features; and a text feature extractor is constructed to process audience interactive text streams and generate keyword density matrices. By integrating object distribution heatmaps, emotional spectrum features, and keyword density matrices, a dynamic scene feature space is formed. A scene-adaptive recommendation model is trained based on the dynamic scene feature space to generate an initial set of live streaming content recommendations. Obtain real-time audience profile attributes, adjust the weight parameters of the scene adaptation recommendation model based on the audience profile attributes, use the scene adaptation recommendation model with adjusted weights to output the final recommended content identifier, and input the final recommended content identifier into the content scheduling engine to execute the live content switching operation.

2. The real-time recommendation method for live streaming content based on multimodal scene perception according to claim 1, characterized in that, The scene parser identifies the following live streaming scene categories: A spatiotemporal convolutional network is used to parse the frame sequence features of the visual data stream, a long short-term memory network is applied to capture the temporal dependencies of the acoustic data stream, and an attention mechanism network is used to extract the semantic focus of the audience interactive text stream. The frame sequence features, temporal dependencies and semantic focus are combined to output scene classification labels.

3. The real-time recommendation method for live streaming content based on multimodal scene perception according to claim 2, characterized in that, The process of constructing a visual feature extractor to process visual data streams and generate object distribution heatmaps includes: Load a pre-trained object detection network to segment entity regions in the visual data stream, calculate the spatial distribution density of entity regions per unit time, and map the spatial distribution density to a two-dimensional coordinate plane to form a heat map.

4. The real-time recommendation method for live streaming content based on multimodal scene perception according to claim 3, characterized in that, The fusion object distribution heatmap, sentiment spectrum features, and keyword density matrix include: The object distribution heatmap is processed by performing region normalization, and the emotional spectrum features are converted into feature vectors that match the dimensions of the region normalization result. The timestamps of the keyword density matrix are aligned with the time axis of the object distribution heatmap, and the region normalization result, feature vectors and timestamp-aligned keyword density matrix are spliced ​​together.

5. The real-time recommendation method for live streaming content based on multimodal scene perception according to claim 4, characterized in that, The scene adaptation recommendation model trained based on dynamic scene feature space includes: A deep belief network is configured as the basic architecture of the recommendation model. The deep belief network is trained by inputting historical dynamic scene feature space samples. During the iterative optimization process, the network parameters are adjusted using a contrastive loss function. The network parameters are frozen when the accuracy of the validation set reaches the preset standard.

6. The real-time recommendation method for live streaming content based on multimodal scene perception according to claim 5, characterized in that, The weight parameters of the scene-adaptive recommendation model adjusted according to audience profile attributes include: The set of preference labels in the audience profile attributes is analyzed, the correlation matrix between the set of preference labels and the dynamic scene feature space is calculated, the scene adaptation weight coefficient is generated based on the correlation matrix, and the scene adaptation weight coefficient is applied to the output layer of the deep belief network.

7. The real-time recommendation method for live streaming content based on multimodal scene perception according to claim 6, characterized in that, The scenario-adaptive recommendation model using adjusted weights includes: The real-time dynamic scene feature space is input into the weighted deep belief network, and the probability distribution of recommended content is generated through network forward propagation. The set of content identifiers corresponding to the peak of the probability distribution is selected.

8. The real-time recommendation method for live streaming content based on multimodal scene perception according to claim 7, characterized in that, After generating the initial live content recommendation set, the process also includes: A real-time feedback channel is established to receive audience dwell time data. When the audience dwell time is lower than a preset threshold, the model optimization module is triggered. The model optimization module uses the artificial bee colony algorithm to update the deep belief network parameters.

9. The real-time recommendation method based on multimodal scenarios according to claim 8, characterized in that, The set of content identifiers corresponding to the peak values ​​of the selected probability distribution includes: Set a dynamic probability threshold to filter low-probability content identifiers, sort the remaining content identifiers by probability value to generate a priority queue, and extract the top N identifiers from the priority queue to form the final recommended content identifiers.

10. The real-time recommendation method for live streaming content based on multimodal scene perception according to claim 9, characterized in that, The content scheduling engine performs live content switching operations including: The system matches the final recommended content identifier with the physical address of the content resource library, preloads the target live content stream through the content delivery network, and sends a switching command to the player terminal to perform a seamless transition.

Images