Live broadcast content real-time recommendation method based on multi-modal scene perception

Abstract

The application relates to the technical field of live content recommendation and discloses a live content real-time recommendation method based on multi-modal scene perception. The method collects visual data flow, acoustic data flow and audience interactive text flow of a live stream in real time, identifies a live scene category through a scene parser, constructs visual, acoustic and text feature extractors respectively, generates an object distribution heat map, emotion spectrum features and a keyword density matrix, fuses the three types of features to form a dynamic scene feature space, trains a scene adaptation recommendation model based on the dynamic scene feature space to generate an initial recommendation set, adjusts model weight parameters in combination with real-time audience portrait attributes, outputs a final recommendation content identifier and executes a switching operation through a content scheduling engine. Through multi-modal data fusion and dynamic adaptation, the method improves the real-time performance and accuracy of live content recommendation and better meets the personalized needs of audiences.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of live content recommendation, in particular to a live content real-time recommendation method based on multi-modal scene perception. BACKGROUND

[0002] With the rapid development of the network live industry, the user's personalized demand for live content is increasingly prominent, and the live platform is facing the challenge of how to accurately and timely push the content that matches the interests of the audience. The current mainstream live content recommendation method relies on single-dimensional data analysis, such as only recommending according to the audience's historical viewing records or interactive behavior, which is difficult to fully capture the dynamic changes of the live scene.

[0003] In terms of visual information processing, traditional methods are often limited to identifying single objects in the live picture, ignoring the spatial distribution relationship between multiple objects, resulting in an inability to accurately determine the core content of the live scene. For example, in an outdoor adventure live broadcast, only the anchor or specific scenery is identified, but the overall environment and activity association cannot be grasped, resulting in a low matching degree of recommended content to the current scene.

[0004] The application of acoustic data also has deficiencies, and the existing technology focuses on basic parameters such as sound volume and frequency, failing to deeply mine the emotional information contained therein. The voice tone, background sound effects and other in the live broadcast often convey different emotions such as joy, tension and calmness, and these emotional factors have an important influence on the audience's preferences. Due to the lack of effective extraction and utilization of emotional spectrum features, the recommendation system is difficult to adjust the recommendation strategy according to the emotional atmosphere of the live broadcast, and difficult to meet the content needs of the audience in a specific emotional state.

[0005] The processing of audience interactive text also has limitations, and currently simple keyword counting or frequency statistics are used, without building a matrix model reflecting the importance and distribution of keywords. The audience's bullet screen, comments and other interactive texts contain a lot of information about content preferences and real-time feedback, and a single counting method cannot fully reveal the internal relationship of these information, making it difficult for the recommendation system to accurately grasp the real-time needs and interest changes of the audience, and the recommended results lag or deviate from the audience's expectations.

[0006] The existing recommendation model lacks a dynamic adjustment mechanism when combining live scene features and audience portraits. The audience's portrait attributes will change with viewing behavior and time, and a fixed weight model cannot respond to these changes in time, resulting in recommended content that cannot adapt to the latest needs of the audience in real time, further reducing the accuracy and effectiveness of the recommendation. SUMMARY

[0007] The present application aims to provide a live content real-time recommendation method based on multi-modal scene perception to solve the problems raised in the background art.

[0008] To achieve the above-mentioned purpose, the present application provides a live content real-time recommendation method based on multi-modal scene perception, which comprises:

[0009] Real-time collection of visual data stream, acoustic data stream and audience interactive text stream of live stream, identification of live scene category through scene parser;

[0010] Construction of visual feature extractor to process visual data stream to generate object distribution heat map, construction of acoustic feature extractor to process acoustic data stream to generate emotional spectrum feature, and construction of text feature extractor to process audience interactive text stream to generate keyword density matrix;

[0011] Fusion of object distribution heat map, emotional spectrum feature and keyword density matrix to form dynamic scene feature space;

[0012] Training of scene adaptation recommendation model based on dynamic scene feature space to generate initial live content recommendation set;

[0013] Acquisition of real-time audience portrait attributes, adjustment of weight parameters of scene adaptation recommendation model according to audience portrait attributes, output of final recommendation content identifier using scene adaptation recommendation model with adjusted weight, and input of final recommendation content identifier into content scheduling engine to perform live content switching operation.

[0014] Preferably, the scene parser identifies the live scene category, which comprises:

[0015] Temporal and spatial convolution network is used to analyze the frame sequence feature of visual data stream, long short-term memory network is used to capture the time sequence dependence relationship of acoustic data stream, attention mechanism network is used to extract the semantic focus of audience interactive text stream, and frame sequence feature, time sequence dependence relationship and semantic focus are combined to output scene classification label.

[0016] Preferably, the construction of visual feature extractor to process visual data stream to generate object distribution heat map comprises:

[0017] A pre-trained object detection network is loaded to segment the entity region in visual data stream, the spatial distribution density of entity region in unit time is calculated, and the spatial distribution density is mapped to two-dimensional coordinate plane to form heat distribution map.

[0018] Preferably, the fusion of object distribution heat map, emotional spectrum feature and keyword density matrix comprises:

[0019] The region normalization processing is performed on the object distribution heat map, the emotional spectrum feature is converted into a feature vector matched with a dimension of the region normalization result, and the timestamp of the keyword density matrix is aligned with the time axis of the object distribution heat map.

[0020] Preferably, the training of the scene-adaptive recommendation model based on the dynamic scene feature space comprises:

[0021] The deep belief network is configured as the recommendation model infrastructure, the deep belief network is trained by inputting the historical dynamic scene feature space sample, the network parameter is adjusted by using the contrast loss function in the iterative optimization process, and the network parameter is frozen when the verification set accuracy rate reaches the preset standard.

[0022] Preferably, the adjusting of the weight parameter of the scene-adaptive recommendation model according to the audience portrait attribute comprises:

[0023] The preference label set in the audience portrait attribute is analyzed, the correlation matrix of the preference label set and the dynamic scene feature space is calculated, the scene-adaptive weight coefficient is generated according to the correlation matrix, and the scene-adaptive weight coefficient is applied to the output layer of the deep belief network.

[0024] Preferably, the use of the scene-adaptive recommendation model with the adjusted weight comprises:

[0025] The real-time dynamic scene feature space is input into the deep belief network with the adjusted weight, the recommendation content probability distribution is generated by the network forward propagation calculation, and the content identifier set corresponding to the probability distribution peak value is selected.

[0026] Preferably, the generating of the initial live content recommendation set further comprises:

[0027] The real-time feedback channel is established to receive the audience stay duration data, the model optimization module is triggered when the audience stay duration is lower than the preset threshold, and the model optimization module updates the deep belief network parameter by using the artificial bee colony algorithm.

[0028] Preferably, the selection of the content identifier set corresponding to the probability distribution peak value comprises:

[0029] The dynamic probability threshold is set to filter the low-probability content identifier, the remaining content identifier is sorted according to the probability value to generate a priority queue, and the final recommendation content identifier is formed by cutting the first N positions of the priority queue.

[0030] Preferably, the execution of the live content switching operation by the content scheduling engine comprises:

[0031] The final recommendation content identifier is matched with the physical address of the content resource library, the target live content stream is preloaded through the content distribution network, and the switching instruction is sent to the player terminal to perform seamless transition.

[0032] Compared with the prior art, the present application has the beneficial effects that:

[0033] By integrating visual, acoustic and audience interactive text data, a more comprehensive and in-depth perception of the live scene is achieved. Real-time collection of three types of data streams and feature extraction enables the analysis of the live scene to go beyond a single dimension, capturing dynamic features from multiple angles such as picture content, sound emotion and audience feedback. The object distribution heat map generated by the visual feature extractor clearly presents the spatial distribution and importance of each object in the picture, helping to accurately identify the core scene and activity content of the live broadcast, and providing a basis for subsequent recommendations that fit the actual picture.

[0034] The acoustic feature extractor generates emotional spectrum features, which deeply excavate the emotional information contained in the sound, and can sensitively capture the emotional atmosphere changes in the live broadcast process. Whether it is cheerful music, tense commentary or calm communication, these emotional elements can be effectively extracted and integrated into scene feature analysis, making the recommended content better fit the emotional tone of the live broadcast at the moment, enhancing the audience's sense of immersion and viewing experience.

[0035] The keyword density matrix constructed by the text feature extractor breaks through the limitations of traditional keyword statistics, comprehensively reflecting the distribution rules and importance levels of keywords in audience interactive text. The key preferences and real-time feedback in audience interactive information such as bullet screen and comments are systematically sorted out, enabling the recommendation system to accurately grasp the audience's immediate interests and demand changes, making the recommended content more in line with the audience's real-time expectations.

[0036] The formation of the dynamic scene feature space organically integrates the three types of features, achieving the complementation and enhancement of multi-dimensional information, and constructing a richer and more three-dimensional scene description. This feature space can be updated in real time and accurately reflect every subtle change in the live scene, providing highly realistic input data for the recommendation model and improving the real-time and accuracy of the recommendation.

[0037] The scene-adaptive recommendation model trained based on the dynamic scene feature space generates an initial recommendation set that is highly matched with the current scene. Combined with real-time audience portrait attributes, the model weight parameters are adjusted, enabling the recommendation process to fully consider individual differences and achieve precise adaptation from scene commonality to audience individuality. This dynamic adjustment mechanism ensures that the recommendation model can respond to changes in the audience portrait in a timely manner, whether it is a shift in audience interest or the formation of new viewing preferences, which can be captured and reflected in the recommended results, ensuring that the final recommended content always keeps pace with the latest audience needs.

[0038] The final recommended content identifier inputs the content scheduling engine to perform a switching operation, so that the recommended content can be quickly and accurately presented to the audience, the recommendation delay is reduced, and the fluency of live content switching is improved. Overall, the method realizes deep fusion and dynamic adaptation of multi-modal data, so that the live content recommendation is more targeted, real-time and accurate, effectively enhances the interactivity and stickiness of the audience and the live content, and brings better content distribution effect to the live platform. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 A working principle diagram of the live content real-time recommendation method based on multi-modal scene perception is provided.

[0040] Figure 2 A flowchart for the scene analyzer to identify the live scene category is provided.

[0041] Figure 3 A multi-modal feature processing and scene classification diagram is provided.

[0042] Figure 4 A flowchart for the visual feature extractor to generate an object distribution heat map is provided.

[0043] Figure 5 A feature extraction and fusion space diagram is provided.

[0044] Figure 6 A flowchart for training a recommended model in a dynamic scene feature space is provided.

[0045] Figure 7 A flowchart for generating a final recommendation by dynamic threshold filtering is provided. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0047] Please refer to Figure 1 The present application provides a live content real-time recommendation method based on multi-modal scene perception, which comprises:

[0048] The visual data stream, acoustic data stream and audience interactive text stream of the live broadcast stream are collected in real time, and the live scene category is identified by using a scene parser. A visual feature extractor processes the visual data stream to generate an object distribution heat map, an acoustic feature extractor processes the acoustic data stream to generate a sentiment spectrum feature, and a text feature extractor processes the audience interactive text stream to generate a keyword density matrix. The object distribution heat map, the sentiment spectrum feature and the keyword density matrix are fused to form a dynamic scene feature space, and a scene adaptive recommendation model is trained based on the feature space to generate an initial live broadcast content recommendation set. The audience portrait attributes are obtained in real time, the weight parameters of the scene adaptive recommendation model are adjusted, and a final recommendation content identifier is output. The live broadcast content switching operation is performed through a content scheduling engine.

[0049] Embodiment 1: refer to Figure 2 The scene parser identifies the live scene category, and the process synchronously processes three kinds of heterogeneous data streams and outputs a unified classification result. After the visual data stream is input, the spatio-temporal convolution network first performs frame sequence sampling to extract key frames at a fixed interval to form a time-continuous image sequence. The network architecture adopts a multi-layer three-dimensional convolution kernel structure, and a single convolution kernel simultaneously slides and operates in the spatial dimension and the time dimension: local feature patterns in a single frame image are detected in the spatial dimension, and motion trajectory changes across frames are captured in the time dimension. The convolution layer output is converted by an activation function and sent to a pooling layer, which retains significant features while reducing the dimension. After multi-layer convolution and pooling operations, the generated spatio-temporal feature map reflects the dynamic behavior patterns of objects in the live picture, such as the running trajectory of athletes in a sports event or the rotation display of goods in an e-commerce live broadcast.

[0050] The acoustic data stream processing adopts a long short-term memory network architecture, which includes a memory unit and a gating mechanism. The original audio signal is converted into a mel spectrum graph after pre-emphasis, framing and windowing processing, and input into the network. The input gate controls the storage weight of the current acoustic feature, the forgetting gate determines the retention proportion of the historical acoustic feature, and the output gate adjusts the output intensity of the feature vector. The network transfers historical information through internal state to effectively capture the long-term dependence of the audio signal, such as the semantic coherence of the speech sound or the rhythm continuity of the background music. The hidden layer state of the network is updated continuously at time steps, and the final output acoustic feature vector contains the time evolution law of the sound event, which can identify the fluctuation frequency of the cheers in a concert or the rise and fall changes of the explanation tone in a teaching live broadcast.

[0051] The interactive text stream is processed through an attention mechanism network. The original bullet screen text is segmented, stop words are removed, and then converted into a sequence of word vectors which are input into the network. The attention layer calculates the relevance weight of each word vector to its context: a trainable parameter matrix maps the word vectors to query vectors, key vectors, and value vectors; the query vector is multiplied by all key vectors to generate an attention score; the score is standardized and used as weights to sum the value vectors. This mechanism allows the network to automatically focus on high-frequency keywords and emotionally charged expressions, such as proper nouns related to breaking news or interactive words with exclamatory tones. The network outputs a semantic focus vector, effectively compressing redundant text information while preserving core semantic features.

[0052] The three feature vectors are integrated in the fusion layer. The video feature map output by the spatiotemporal convolutional network is flattened and converted into feature vectors, which are then aligned with the acoustic feature vectors generated by the long short-term memory network and the semantic focus vectors generated by the attention mechanism network according to a fixed dimension. The feature vectors are concatenated to form a joint feature representation, which is then input into the fully connected layer for nonlinear transformation. The transformed feature vectors enter the multi-class classifier, where the probability distribution of each scene category is calculated using the Softmax function. The output layer sets a scene label threshold; the category with the maximum probability value exceeding the threshold is determined as the current scene category, covering major live streaming types such as sports competitions, stage performances, product promotions, and classroom teaching. The entire parsing process adopts a pipelined architecture to achieve parallel processing of multimodal data, with processing latency controlled within 40 milliseconds to meet real-time requirements. The classification result also serves as a timestamp synchronization benchmark, providing a temporal alignment basis for subsequent feature extraction modules.

[0053] See Figure 3 This demonstrates the core processing flow of multimodal scene analysis, encompassing four key analytical dimensions. The spatiotemporal visual feature heatmap in the upper left corner reveals the dynamic changes of objects in the live stream; the vertical axis represents continuous time steps, the horizontal axis displays different visual feature dimensions, and grayscale depth reflects feature activation intensity. The acoustic sentiment line graph in the upper right corner depicts the changing trends of five main emotions over time; the fluctuations of each line correspond to the fluctuations in narration tone or background music, such as the peak emotional intensity during moments of excitement, with different grayscale lines representing different emotion types. The text attention histogram in the lower left corner quantifies the importance weights of different keywords; higher bars indicate words more decisive for scene classification. The scene classification probability in the lower right corner displays the system's judgment criteria for the current live stream scene; the probability differences between different scenes reflect the system's confidence in recognizing scenes such as sports competitions and stage performances.

[0054] Example 2: See Figure 4, covering the extraction and fusion process of visual, acoustic and text features, and constructing a unified dynamic scene feature space. The visual feature extractor processes the visual data stream based on a pre-trained target detection network architecture. The network uses a region proposal network to generate initial candidate boxes, sets multiple anchor boxes on the convolutional feature map, and adjusts the anchor position and size through boundary box regression. After screening by non-maximum suppression, the candidate boxes are input into the region of interest pooling layer, which converts candidate regions of different sizes into fixed-length feature vectors. The feature vectors are processed in two ways: the classification branch outputs the entity class probability distribution, and the regression branch optimizes the candidate box coordinate parameters. The detection results mark the class, position and size information of the entity objects in the picture, forming a sequence of object metadata.

[0055] Statistical spatial distribution of entities appearing in a unit time window: divide the live picture into uniform grids and calculate the frequency of each type of entity appearing in each grid cell. For the same type of entity position points, use the kernel density estimation algorithm to smooth the processing through the Gaussian kernel function to form a probability density surface. The surface is mapped to a two-dimensional coordinate plane by color coding to generate an object distribution heat map. The red area in the heat map represents a high-density distribution area, such as the central part of the football field where players often gather in a live football match; the blue area represents a low-density area, such as the edge area of the football field. In e-commerce live scenes, high-intensity heat areas may appear in the product display area.

[0056] The acoustic feature extractor processes the original audio stream, and the front end compensates for the high-frequency components with pre-emphasis, followed by frame and window processing to block spectral leakage. Each frame of signal is converted to the frequency domain by Fourier transform, and then the Mel filter bank is used to simulate the human auditory characteristics, outputting the Mel spectrum coefficients to form a time-frequency two-dimensional matrix. The emotion recognition model uses convolutional layers to extract local patterns of the spectrum, recurrent layers to capture long-term context associations, and fully connected layers to output the probability distribution of emotional labels. The Mel spectrum matrix and the emotion probability vector are connected along the channel dimension to form a time-aligned emotion spectrum feature matrix. The changes in the time dimension of the matrix reflect the emotional fluctuations, such as the simultaneous sudden increase in sound intensity and high-frequency energy corresponding to the passionate tone of the commentator.

[0057] The text feature extractor collects interactive text streams such as bullet screens and comments. Text preprocessing includes word segmentation, removal of stop words, and stem extraction, and a vocabulary table is constructed to record valid keywords. A sliding time window mechanism is used to count the keyword frequency in the window at fixed intervals. The inverse document frequency factor is introduced to suppress the weight of common words, and the formula gives higher scores to professional terms. The time window is used as the row and the keyword as the column to construct the initial frequency matrix, and the row normalization is used to eliminate the length difference of the time window, and finally the keyword density matrix is generated. The row vectors of the matrix reflect the semantic focus at a specific time, such as the keywords "shot" and "score" in sports events, which show peak density at critical moments.

[0058] The preprocessing operation is performed before the three-way feature fusion. The object distribution heat map is regionally normalized: the screen is divided into several equal-area regions, the average heat of each region is calculated, and the difference in screen proportion of different live scenes is eliminated. The emotion spectrum feature matrix is converted into a fixed-dimension feature vector by a fully connected neural network, maintaining compatibility with the normalized heat map dimension. The time synchronization subsystem takes the video stream timestamp as the reference, aligns the time axis of the acoustic feature matrix and the text density matrix by linear interpolation algorithm, and the error is controlled within millisecond level precision.

[0059] In the feature splicing stage, the normalized heat map is used as the spatial feature base, the converted emotion feature vector is used as the attribute feature supplement, and the time axis aligned keyword density matrix is used as the semantic feature expansion. The three are spliced along the feature channel dimension to form a three-dimensional feature data block, which constitutes a complete dynamic scene feature space. The spatial dimension of the data block corresponds to the screen grid division scheme, the time dimension contains continuous time slices, and the channel dimension integrates three types of information features: visual distribution, acoustic emotion, and text semantics. This feature space may present as follows in a sports live scene: the heat map shows that the players are densely distributed in the forbidden zone (spatial feature), the emotion spectrum detects an increase in excitement emotion (acoustic feature), and the barrage matrix appears the high-frequency word “penalty” (text feature), all of which represent the state of the key game moment. The data block is encapsulated and sent to the downstream recommendation model to provide a multi-modal joint representation basis for content decision-making.

[0060] Referring to Figure 5 , the key data structures of multi-modal feature extraction and fusion are shown. The object distribution heat map in the upper left corner divides the live screen into an 8x8 grid, and the gray value of each cell represents the density of object appearance in that area. The darker area corresponds to the player gathering area or the commodity display hotspot. The emotion spectrum matrix in the upper right corner shows six types of emotions vertically and time series horizontally. The gray scale change in the matrix reflects the dynamic transition of emotional state, such as the transition from calm to excitement. In the keyword density matrix in the lower left corner, the vertical axis represents time development, and the horizontal axis corresponds to different keyword indexes. The point density reflects the change of the audience's discussion focus at a specific time. The fused feature space slice in the lower right corner shows the spatial distribution characteristics of the integrated multi-modal data. This data structure encodes visual distribution, acoustic emotion, and text semantics, providing a joint representation basis for content recommendation. The darker area in the figure represents the spatial position with strong feature response, corresponding to the core attention area of the live screen.

[0061] Example 3: Referring to Figure 6, the construction and personalized adjustment process of the scene-adaptive recommendation model. The deep belief network is used as the basic framework of the recommendation model, and a deep structure is formed by stacking multiple restricted Boltzmann machines. Each restricted Boltzmann machine includes a visible layer and a hidden layer, and the nodes in the visible layer are fully connected with the nodes in the hidden layer, and the nodes in the layer are not connected. The training process adopts a layer-by-layer greedy algorithm, and the parameters of the bottom layer restricted Boltzmann machine are fixed after the training is completed, and the hidden layer output is used as the visible layer input of the upper layer restricted Boltzmann machine. A Softmax classifier is added to the uppermost layer of the network to form a complete deep belief network structure. When the historical dynamic scene feature space sample is input into the network, the visible layer receives the feature vector, and the network parameters are updated through the contrastive divergence algorithm. During the parameter updating process, the state of the visible layer node is calculated through the activation function:

[0062]

[0063] wherein: represents the state of the i-th node in the visible layer, represents the state of the j-th node in the hidden layer, is a connection weight, is a visible layer bias term, is a Sigmoid activation function. The network training adopts a ternary loss function to construct a contrast relationship between samples, and each training sample includes an anchor sample, a positive sample and a negative sample. The anchor sample is from the feature vector of the target scene, the positive sample is the feature vector of the same scene, and the negative sample is the feature vector of the different scene. The loss function drives the network to learn to reduce the distance between the anchor sample and the positive sample and to increase the distance between the anchor sample and the negative sample. In the training process, an early stopping mechanism is set, and when the recommendation accuracy on the validation set no longer improves for multiple training cycles, the training is terminated to prevent overfitting. The finally trained network parameters are frozen to form a basic recommendation model.

[0064] The audience portrait attribute analysis module processes user registration information, historical behavior records and social relationship data. The registration information includes demographic characteristics, the historical behavior records include watching time, interaction frequency and content preference, and the social relationship data involves a following list and a friend network. These original data are cleaned and converted to generate a structured preference label set. Each label in the label set is associated with a weight value, reflecting the user's preference intensity for the feature. When calculating the correlation between the preference label set and the dynamic scene feature space, a label-feature matrix M is constructed, the rows of the matrix correspond to the preference labels, and the columns correspond to the scene features. The element represents the correlation intensity of the i-th label and the j-th scene feature, which is calculated by statistical analysis method. The label-feature matrix is decomposed into a low-rank approximation by matrix decomposition technology:

[0065]

[0066] where P is the tag latent factor matrix and Q is the feature latent factor matrix. The main eigenvector is extracted from the decomposition result to form a scene adaptation weight matrix W, whose dimension matches the output layer of the deep belief network. In the weight adjustment stage, the Hadamard product operation is performed between the weight matrix W and the network output layer activation value A:

[0067]

[0068] The operation result As the personalized adjusted recommendation score, the content weight matching the user preference is enhanced, and the recommendation score of irrelevant content is suppressed. The adjustment process preserves the relative ordering relationship of the original recommendation score and only fine-tunes the recommendation strength in a local range. The personalized adjusted recommendation score is input into the Softmax function to generate the final recommendation probability distribution. This distribution reflects both the live scene characteristics and the user's personalized preferences, achieving dual adaptation of the scene and the user. The model updating mechanism periodically collects user feedback data, and when the user behavior pattern changes significantly, the tag-feature matrix is recalculated and the weight matrix is updated to maintain the dynamic adaptability of the recommendation system. The entire recommendation process is completed within milliseconds, meeting the real-time requirements of the live scene. The network parameter adjustment adopts an incremental learning strategy to avoid the computational overhead of retraining, ensuring stable system operation. The recommendation results are output in the form of content identifiers, providing decision-making basis for subsequent content scheduling.

[0069] Example 4: Focus on the recommendation model running after weight adjustment and real-time optimization mechanism, throughout the complete process from dynamic scene feature processing to content recommendation. In the initial stage, the real-time collected dynamic scene feature space is input into the adjusted weight deep belief network as the input source. The feature space is composed of multiple modalities: the normalized object distribution heat map reflects the entity distribution pattern of the picture; the converted emotion spectrum feature vector carries acoustic emotion information; the time-synchronized keyword density matrix contains text semantic focus. The feature space is encoded by the input layer and transmitted to the hidden layer, where forward propagation calculation is performed in the network. The hidden layer consists of multiple restricted Boltzmann machines, and each node calculates the output value through the activation function and transmits it to the next layer. The propagation process includes multiple nonlinear transformations, which gradually abstract high-order feature patterns. For example, in the live scene of a sports event, the heat map shows that the players are highly concentrated in the penalty area, the emotion vector detects a sharp increase in the intensity of cheers, and the keyword matrix appears the high-frequency word "penalty". These features are integrated and encoded into a high-dimensional feature representation. The output layer uses the Softmax function to process the final hidden layer output to calculate the probability distribution value of each candidate content. The probability value reflects the comprehensive adaptation degree of the content to the current scene and the audience portrait, and the distribution range is between 0 and 1, with the peak value indicating the best matching content. After the probability distribution calculation is completed, the system selects the content identifier set corresponding to the peak value as the recommendation output.

[0070] The real-time feedback channel continuously monitors audience behavior data, collecting three core indicators through user behavior logs: dwell time, which refers to the duration of the audience's viewing of the current recommended content page; interaction frequency, which counts the number of bullet screen sends or likes; and exit flag, which records whether the audience terminated the viewing prematurely. These data are recorded with millisecond precision to form a time series behavior dataset. After collecting feedback data, it is input into the decision engine. When the dwell time is detected to be lower than the preset threshold (such as 40 seconds in the sports event scenario), the system automatically triggers the model optimization module. The threshold setting depends on the scenario type, for example, education and training use a longer threshold (such as 90 seconds) to adapt to the habit of deep content consumption. When behavior data triggers a decision, the system generates a standardized record including timestamp, dwell time, interaction frequency, and exit flag. Table 1 shows a segment of audience behavior data collected in a live football game scenario.

[0071] Table 1: Segment of audience behavior data collected in a live football game scenario.

[0072] Time stamp Duration of stay (seconds) Interaction frequency (times / minute) Exit flag 20230807T142030 38 5 Yes 20230807T142045 27 2 Yes 20230807T142100 52 8 No 20230807T142115 31 3 Yes

[0073] In the table data, records with dwell time lower than the threshold of 40 seconds trigger the optimization process (such as the first and third rows of data), and the exit flag of "yes" indicates that the user interrupted the viewing, providing a direct signal for model correction.

[0074] The artificial bee colony algorithm performs the model optimization task, which simulates the foraging behavior of a bee colony to update the parameters of the deep belief network in stages. In the initial stage, the parameter space is defined, and each bee position corresponds to a weight or bias parameter in the network. In the employed bee stage, a random initial position neighborhood is selected, and the fitness function value is calculated: the function is defined based on the current behavior feedback, combining indicators such as dwell time and interaction frequency to construct a loss value. The smaller the loss value, the better the network performance. For example, when low dwell time occurs in a live football game, the algorithm calculates the loss value change trend in the position neighborhood. In the observer bee stage, the optimization direction is selected according to the fitness: high fitness bee positions attract more observer bees to search, and new solutions are explored through position update formulas. In the scout bee stage, local optima are handled: when the position loss value does not improve for several consecutive times, the bee position is randomly reset to break the deadlock. The iterative process continuously optimizes network parameters, such as adjusting the hidden layer activation threshold or the output layer scaling factor. After the algorithm ends, the network parameters are updated to enhance the model's response to dynamic changes in audience preferences. The entire process is delayed to the millisecond level, without affecting the continuity of real-time recommendations. After optimization is complete, the model continues to receive new feature space inputs, generating subsequent recommendation probability distributions, forming a closed-loop feedback system.

[0075] In a specific example, taking the live broadcast of the European Cup football final as an example: when the game enters the penalty shootout moment, the system inputs the dynamic scene feature space, including the high-density area of the goalkeeper area in the heat map, the increase of the acoustic excitement index in the emotional spectrum, and the highlight of the "save" high-frequency word in the keyword matrix. The weight adjustment model outputs the recommended probability distribution, and the peak content is "slow motion playback". The audience behavior data is shown in the table, and part of the user stays for a short time to trigger optimization; after adjusting the network parameters by the artificial bee colony algorithm, the new recommended probability distribution prefers "tactical analysis" content, which improves the user retention rate. The recommended output is finally converted into a content identifier, which is processed by the downstream engine. The whole implementation ensures the balance between real-time and adaptability, and the processing rate matches the live data stream speed.

[0076] Embodiment 5: refer to Figure 7 , which covers the recommended result post-processing and content scheduling execution phase, processing input is the probability distribution data of the scene adaptation recommendation model output after adjusting the weights. After generating the probability distribution of the recommended content, perform a dynamic probability threshold filtering operation. The system automatically sets the probability threshold value according to the real-time recognized live scene category, and the threshold has scene adaptability. A higher threshold is set to filter low-probability results in the sports event category, because the scene event conversion is fast and the audience's attention is focused, so key content needs to be accurately recommended; a lower threshold is used in the education and training category to retain more options, because knowledge transfer needs auxiliary materials to expand cognition. The threshold value in the concert live scene setting is moderate, balancing the need for coherence and diversity of performances. The threshold value floats dynamically with the live progress, such as increasing the threshold to the peak level during the key moments of a football match, and moderately reducing it during the halftime break. The remaining content identifier set after filtering forms a temporary result set, and each identifier is associated with a probability value and a metadata label.

[0077] Perform a descending order arrangement process on the temporary result set. The comparison algorithm uses a double-pointer quicksort mechanism, and the order is maintained by swapping positions during pointer movement. The algorithm's time complexity is optimized to linear logarithmic level, and processing a queue of thousands of identifiers takes less than a millisecond. The sorting generates a priority queue structure, with the highest probability value content identifier stored at the head of the queue, and the lowest effective probability identifier at the tail of the queue. The queue data structure is implemented using a max heap, supporting efficient head element access and deletion operations. The final recommendation set is formed by intercepting the first N elements of the queue, and the value of N is dynamically calculated. The calculation factor includes the real-time load status of the content resource library, the bandwidth margin of the edge node, and the performance parameters of the playback terminal device. The load balancer periodically pushes the resource library status code, and a high status code triggers a strategy to reduce the value of N; when the bandwidth monitor shows the available margin, the value of N is appropriately expanded. The final recommendation set is packaged as a structured data packet, including the identifier list, the corresponding probability weight, and the recommended validity period.

[0078] The content scheduling engine performs address resolution after receiving the recommendation data packet. The distributed hash table of the content repository is retrieved to query the physical storage location of the target live content through the identifier. The hash table organizes data nodes using a consistent hashing algorithm to alleviate mapping disturbance caused by node changes. The physical address includes the edge node IP, storage path, and content shard index. The query result triggers the content distribution network preload instruction to select the optimal edge node group to build the transmission path. The transmission path design considers the network topology, prioritizing intracity node hops and secondarily selecting cross-regional backbone nodes. The preload process uses a hierarchical buffering strategy, loading key frame data blocks in the first stage and supplementing full data in the second stage. The cache management module monitors the memory usage of the edge node and eliminates low-value cache content based on the priority queue.

[0079] The player terminal receives the switch control instruction to perform the transition operation. The instruction includes the target stream address, switch timestamp, and audio transition strategy. The switch time window selects the key frame position of the live stream to avoid image damage caused by intra-frame prediction residuals. The player maintains a double-buffering mechanism, with the original buffer continuing to output the current frame sequence and the background buffer asynchronously loading target live stream data. When the buffer progress reaches the switch threshold, the frame sequence is replaced at the key frame boundary point. The audio track processing uses cross-fade technology, maintaining the level stability of the original audio track while gradually fading in the target track, with the transition period controlled within the human auditory imperceptible range. A hard switch is performed on the visual level to ensure immediate switching of picture content and avoid blurred transitions that interfere with key information transmission. The switch completion event triggers a feedback signal to the recommendation system, forming an operation closed loop.

[0080] A specific implementation example occurs in a live basketball game scenario: during the last two minutes of the game, the score is tied, and the recommendation model outputs a probability distribution with a "star player technical statistics" content identifier probability of 0.89 and a "slow-motion replay" identifier probability of 0.78. The dynamic threshold is set to 0.75, and the "event preview" identifier (0.70) is filtered. The sorting generates a queue with the technical statistics content at the top, and the N value is calculated as 3 but is actually truncated to two items due to resource limitations. The scheduling engine queries that the technical statistics content is located in the edge node group A-03 and starts preloading 200ms of key frame data blocks. The switch instruction specifies execution at the timer pause point, and the player completes the switch at the stable frame of the scoreboard screen, with the background commentary audio seamlessly connecting without interruption. The implementation of the entire data processing delay is controlled within 80 milliseconds, and the operation accurately matches the live time point characteristics.

[0081] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting; it is not intended to exclude myriad other embodiments of the present application that other inventors can develop based on the same general inventive concepts embodied by the described embodiments. That is, although the present application is described in terms of particular embodiments and illustrative figures, it should be apparent that the scope of the present application is not limited to these specific embodiments.

[0082] While the embodiments of the application have been shown and described herein, it will be understood by those skilled in the art that many changes, modifications, substitutions and alterations to these embodiments can be made without departing from the principles and spirits of the application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for real-time recommendation of live content based on multi-modal scene perception, characterized in that, The method comprises the following steps: Real-time acquisition of visual data stream, acoustic data stream and audience interactive text stream of live streaming, identification of live scene category through scene parser; Construction of visual feature extractor to process visual data stream to generate object distribution heat map, construction of acoustic feature extractor to process acoustic data stream to generate emotion spectrum feature, and construction of text feature extractor to process audience interactive text stream to generate keyword density matrix; Fusion of object distribution heat map, emotion spectrum feature and keyword density matrix to form dynamic scene feature space; Training of scene adaptive recommendation model based on dynamic scene feature space to generate initial live content recommendation set; Obtaining real-time audience portrait attribute, adjusting weight parameters of scene adaptive recommendation model according to audience portrait attribute, outputting final recommendation content identifier by using scene adaptive recommendation model with adjusted weight, and inputting final recommendation content identifier into content scheduling engine to perform live content switching operation.

2. The method for live content real-time recommendation based on multi-modal scene perception according to claim 1, characterized in that, The scene parser identifies the live scene category, which comprises the following steps: Using spatio-temporal convolution network to analyze frame sequence features of visual data stream, using long short-term memory network to capture time sequence dependence of acoustic data stream, using attention mechanism network to extract semantic focus of audience interactive text stream, and combining frame sequence features, time sequence dependence and semantic focus to output scene classification label.

3. The live content real-time recommendation method based on multi-modal scene perception according to claim 2, characterized in that, The construction of visual feature extractor to process visual data stream to generate object distribution heat map comprises the following steps: Loading pre-trained object detection network to segment entity regions in visual data stream, calculating spatial distribution density of entity regions in unit time, and mapping spatial distribution density to two-dimensional coordinate plane to form heat distribution map.

4. The live content real-time recommendation method based on multi-modal scene perception according to claim 3, characterized in that, The fusion of object distribution heat map, emotion spectrum feature and keyword density matrix comprises the following steps: Performing region normalization processing on object distribution heat map, converting emotion spectrum feature into feature vector matching the dimension of region normalization result, aligning time stamp of keyword density matrix with time axis of object distribution heat map, and splicing region normalization result, feature vector and keyword density matrix after time stamp alignment.

5. The live content real-time recommendation method based on multi-modal scene perception according to claim 4, characterized in that, The training of scene adaptive recommendation model based on dynamic scene feature space comprises the following steps: Configuring deep belief network as the basic architecture of recommendation model, training deep belief network by inputting historical dynamic scene feature space samples, adjusting network parameters by using contrastive loss function in iterative optimization process, and freezing network parameters when the accuracy of verification set reaches the preset standard.

6. The live content real-time recommendation method based on multi-modal scene perception according to claim 5, characterized in that, The adjusting of weight parameters of scene adaptive recommendation model according to audience portrait attribute comprises the following steps: Analyzing preference label set in audience portrait attribute, calculating correlation matrix of preference label set and dynamic scene feature space, generating scene adaptive weight coefficient according to correlation matrix, and applying scene adaptive weight coefficient to output layer of deep belief network.

7. The live content real-time recommendation method based on multi-modal scene perception according to claim 6, characterized in that, The use of scene adaptive recommendation model with adjusted weight comprises the following steps: Inputting real-time dynamic scene feature space into deep belief network with adjusted weight, calculating and generating recommendation content probability distribution through network forward propagation, and selecting content identifier set corresponding to probability distribution peak value.

8. The live content real-time recommendation method based on multi-modal scene perception of claim 7, characterized in that, After generating the initial live content recommendation set, the method further comprises the following steps: A real-time feedback channel is established to receive audience dwell time data, and a model optimization module is triggered when the audience dwell time is lower than a preset threshold, and the model optimization module updates the deep belief network parameters using an artificial bee colony algorithm.

9. The live content real-time recommendation method based on multi-modal scene perception of claim 8, characterized in that, The content identifier set corresponding to the peak of the selected probability distribution includes: A dynamic probability threshold is set to filter low-probability content identifiers, the remaining content identifiers are sorted according to probability values to generate a priority queue, and the first N positions of the priority queue are intercepted to form final recommended content identifiers.

10. The live content real-time recommendation method based on multi-modal scene perception of claim 9, characterized in that, The content scheduling engine performs a live content switching operation, including: The final recommended content identifier is matched with the physical address of the content resource library, the target live content stream is preloaded through the content distribution network, and a switching instruction is sent to the player terminal to perform seamless transition.

Images