An e-commerce live multi-camera intelligent director and real-time data linkage analysis system

Abstract

The application relates to the technical field of e-commerce data processing, in particular to an e-commerce live broadcast multi-camera intelligent director and real-time data linkage analysis system. The system comprises a data acquisition module, which is used for acquiring multi-camera video streams, extracting picture feature sequences and picture quality scores, synchronously acquiring audience behavior data, commodity interaction data and sales data to generate time series data streams; a live broadcast content analysis module, which is used for identifying live broadcast scene types and commodity display areas based on the picture feature sequences, combining the time series data streams to divide current live broadcast stages into explanation periods, conversion periods and interaction periods, and generating stage semantic vectors; and an audience behavior analysis module, which is used for extracting multi-dimensional behavior features from the time series data streams to establish implicit interest state sequences, modeling audience interest migration tracks through a state transition probability matrix, and outputting interest state vectors.

Description

Technical Field

[0001] This invention relates to the field of e-commerce data processing technology, specifically to an e-commerce live streaming multi-camera intelligent directing and real-time data linkage analysis system. Background Technology

[0002] In e-commerce live streaming events, a multi-camera deployment scheme is usually adopted to cover the full view of the anchor, close-up of the products, and audience interaction. The directing system needs to make real-time decisions on camera switching and image combination based on changes in the live streaming content in order to maintain audience attention and promote product conversion. As the scale of live streaming expands and the density of sessions increases, the manual switching method relying on operator experience is no longer able to meet the high-frequency, real-time image scheduling needs. Automated directing technology has gradually become a key link supporting live streaming business.

[0003] Existing automated broadcast control solutions mainly select camera positions based on fixed rules or single image quality indicators. Rule-based broadcast control systems trigger switching actions based on preset time intervals or scene tags, and cannot dynamically adjust strategies according to the actual evolution of the live content and real-time feedback from the audience. Some systems introduce quality scores such as image clarity and exposure as the basis for camera position selection, but only consider the usability indicators of the image itself, without incorporating behavioral information such as the current business stage of the live broadcast and changes in audience interest into the decision input, resulting in a disconnect between camera position switching and the live broadcast rhythm.

[0004] The limitations of the above approach are that the director's decisions rely only on partial information, making it impossible to prioritize the integrity of product information delivery during the explanation period, enhance the visual presentation of products to promote purchasing behavior during the conversion period, or respond to the audience's participation needs in a timely manner during the interaction period. This makes it impossible to adaptively execute the optimal visual presentation strategy at different stages, thereby affecting audience retention and product conversion.

[0005] Therefore, the present invention provides an intelligent multi-camera directing and real-time data linkage analysis system for e-commerce live streaming. Summary of the Invention

[0006] In order to overcome the shortcomings of the prior art, at least one technical problem raised in the background art is solved.

[0007] The technical solution adopted by this invention to solve its technical problem is: a multi-camera intelligent broadcasting and real-time data linkage analysis system for e-commerce live streaming, comprising:

[0008] The data acquisition module is used to acquire multi-camera video streams and extract image feature sequences and image quality scores, while simultaneously acquiring audience behavior data, product interaction data and sales data to generate time-series data streams.

[0009] The live streaming content analysis module identifies the live streaming scene type and product display area based on the image feature sequence, and divides the current live streaming stage into explanation period, conversion period and interaction period by combining the time-series data stream, and generates stage semantic vectors.

[0010] The audience behavior analysis module is used to extract multi-dimensional behavioral features from the time-series data stream to establish a hidden interest state sequence, model the audience interest migration trajectory through the state transition probability matrix, and output the interest state vector. The state transition probability matrix is ​​used to predict the audience interest state distribution in the next time window.

[0011] The data fusion module is used to fuse the stage semantic vector, interest state vector, and the image quality score of each camera position into a joint state vector;

[0012] The phased broadcast decision module is used to determine the range of executable broadcast actions from a predefined subset of actions based on the current live broadcast phase. The subset of actions is configured with different multi-camera shot combination rules and transition methods for the explanation phase, conversion phase, and interaction phase, and cannot be mixed across phases. The joint state vector is input into the reinforcement learning network to output the broadcast switching action sequence. The reward function of the reinforcement learning network is configured with differentiated weights according to the current phase.

[0013] The linkage feedback module is used to collect the actual changes in audience retention rate and product click conversion rate after the execution of the director switching action sequence, calculate the advantage value, and generate a strategy gradient backhaul signal based on the advantage value to update the network parameters of the reinforcement learning network.

[0014] Preferably, the data acquisition module extracts key frames at a preset frame rate and extracts color, texture and motion features to form the image feature sequence, and calculates the image quality score based on the clarity and exposure of the key frames from each camera position.

[0015] The audience behavior data, product interaction data, and sales data are collected according to a preset sampling period. The audience behavior data includes the number of real-time viewers and the frequency of bullet comments. The product interaction data includes the number of product clicks and the number of times products are added to the cart. The sales data includes the number of orders placed and the real-time transaction amount. The time-series data stream is generated by aligning the data with timestamps.

[0016] Preferably, the steps for generating the stage semantic vector are as follows:

[0017] S301. Calculate the similarity between the color features and texture features in the image feature sequence of each camera position and the preset scene template, and select the scene type with the highest similarity as the current live broadcast scene type.

[0018] S302. Based on the spatial distribution of motion features in the image feature sequence, the area of ​​concentrated motion is determined as the product display area, and the color histogram and contour features of the area are extracted as product display features.

[0019] S303. Based on the direction of increase or decrease of the number of real-time viewers, the frequency of bullet comments, and the number of orders in the time-series data stream at adjacent sampling times, and combined with whether the product display area exists in the current keyframe and the number of consecutive frames, determine the current live broadcast stage.

[0020] S304. Encode the stage type and the live streaming scenario type respectively, and concatenate them with the product display features to generate the stage semantic vector.

[0021] Preferably, the establishment of the implicit interest state sequence includes:

[0022] The audience behavior analysis module slides and divides the time-series data stream according to a preset time window, extracts viewing behavior features, interaction behavior features and conversion behavior features in each window, and generates a multi-dimensional behavior feature sequence by arranging them in time sequence.

[0023] K implicit interest state categories are predefined, and each category corresponds to a state representation vector. The state representation vector is determined by the cluster center obtained by clustering the feature vectors of historical live data windows. K is an integer greater than 1.

[0024] For each time window, the multi-dimensional behavioral features are calculated, and their cosine similarity with each state representation vector is calculated. The window is then mapped to the latent interest state category with the largest cosine similarity value, and the latent interest state sequence is generated by arranging the windows in time sequence.

[0025] Preferably, the process of modeling the audience interest migration trajectory using the state transition probability matrix and outputting an interest state vector specifically involves:

[0026] The frequency of state transitions between adjacent time windows in the hidden interest state sequence is statistically analyzed, and the transition frequency of each state is normalized by row to construct the state transition probability matrix.

[0027] Based on the state transition probability matrix, calculate the predicted probability of the next time window being in each latent interest state category, and take the latent interest state category with the highest predicted probability as the predicted interest state.

[0028] The implicit interest state corresponding to the current time window and the predicted interest state are encoded respectively, and then concatenated to generate the interest state vector.

[0029] Preferably, the method for fusing the joint state vector is as follows:

[0030] The stage semantic vector and the interest state vector are concatenated to obtain a concatenated vector. The image quality scores of each camera position are arranged in order of camera position to form a quality vector.

[0031] The concatenated vector is mapped to the same dimension as the quality vector through a fully connected network, and the mapping result is concatenated with the quality vector to generate the joint state vector.

[0032] Preferably, the predefined subset of actions includes a subset of actions during the explanation period, a subset of actions during the transformation period, and a subset of actions during the interaction period;

[0033] The action subset during the explanation period includes: outputting a panoramic view of the anchor, outputting a close-up view of the product, and a picture-in-picture composite image that overlays the close-up view of the product onto a designated area of ​​the panoramic view of the anchor; the transition method is either a hard cut transition or a fade-in / fade-out transition;

[0034] The conversion period action subset includes: multi-angle close-up shots of each product are rotated and output at preset time intervals, and a guide combination screen that overlays the product close-up shots with the purchase link information layer; the transition methods include hard cut transitions and zoom transitions that move from panoramic shots to close-up shots;

[0035] The interactive action subset includes: a bullet screen overlaying the bullet screen text layer onto the current camera view, a camera view outputting the anchor's reaction, and a combined view of connecting viewers' video windows overlaid on a designated area of ​​the anchor's screen; the transition method is a dissolve transition;

[0036] When the phased broadcast decision module switches between the current live broadcast phases, the broadcast actions of the last N windows of the previous phase are only allowed to transition to the first action of the next phase using the transition method configured for the subset of actions in that phase, where N is a preset positive integer.

[0037] Preferably, the reinforcement learning network is a proximal policy optimization network, comprising a policy network and a value network;

[0038] The policy network takes the joint state vector as input and outputs the selection probability value of each director action in the action subset corresponding to the current stage.

[0039] The value network takes the joint state vector as input and outputs the predicted cumulative reward value that can be obtained from the current state until the end of the live broadcast.

[0040] Preferably, the reward function of the reinforcement learning network is configured with differentiated weights according to the current stage, specifically as follows:

[0041] Explanation period: The reward value is obtained by weighted summation of picture quality score, change in audience retention rate, and product click-through rate; Conversion period: The reward value is obtained by weighted summation of product click-through conversion rate increment, change in audience retention rate, and picture quality score; Interaction period: The reward value is obtained by weighted summation of bullet screen interaction rate increment, change in audience retention rate, and product click-through rate.

[0042] Preferably, when the predicted interest state is a purchase intention state and the current live broadcast stage is the explanation period, the phased broadcast decision module switches the range of executable broadcast actions from the subset of actions in the explanation period to the subset of actions in the conversion period one time window in advance.

[0043] This invention provides a multi-camera intelligent directing and real-time data linkage analysis system for e-commerce live streaming. It has the following beneficial effects:

[0044] 1. This invention uses a state transition probability matrix to discretize and model the trajectory of audience interest migration, enabling the system to predict the evolution trend of the audience from browsing to purchasing, and adjust the screen presentation strategy in advance before the interest inflection point occurs.

[0045] 2. The phase-differentiated reward function configuration enables the same reinforcement learning network to automatically switch optimization objectives in different live streaming phases. In the explanation phase, priority is given to ensuring the integrity of the information conveyed on the screen. In the conversion phase, the focus is on the visual impact of the product presentation. In the interaction phase, the focus is on the visual feedback of the audience's participation atmosphere.

[0046] 3. The linkage feedback module incorporates changes in audience retention rate and increases in product click conversion rate as actual business results into the strategy gradient update, so that the optimization direction of the broadcast strategy is directly aligned with the actual operational indicators of the live broadcast room, rather than relying solely on intermediate indicators such as picture quality. Attached Figure Description

[0047] Figure 1 This is a system flowchart of the present invention;

[0048] Figure 2 This is a schematic diagram of the phased broadcasting decision-making process of the present invention;

[0049] Figure 3 This is a schematic diagram of the linkage feedback closed loop of the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] like Figures 1 to 3 As shown, this invention proposes a multi-camera intelligent broadcasting and real-time data linkage analysis system for e-commerce live streaming, comprising:

[0052] The data acquisition module is used to acquire multi-camera video streams and extract image feature sequences and image quality scores, while simultaneously acquiring audience behavior data, product interaction data and sales data to generate time-series data streams.

[0053] The live streaming content analysis module identifies the live streaming scene type and product display area based on the image feature sequence. It combines the time-series data stream to divide the current live streaming stage into the explanation period, conversion period, and interaction period, and generates stage semantic vectors.

[0054] The audience behavior analysis module is used to extract multi-dimensional behavioral features from time-series data streams to establish a sequence of implicit interest states, model the audience interest migration trajectory through a state transition probability matrix, and output an interest state vector.

[0055] The data fusion module is used to fuse the stage semantic vector, interest state vector, and the image quality scores of each camera position into a joint state vector;

[0056] The phased broadcast decision module is used to determine the range of executable broadcast actions from a predefined subset of actions based on the current live broadcast phase. It inputs the joint state vector into the reinforcement learning network and outputs a broadcast switching action sequence. The action subset is configured with different multi-camera shot combination rules and transition methods for the explanation phase, conversion phase, and interaction phase. The reward function of the reinforcement learning network is configured with differentiated weights according to the current phase.

[0057] The linkage feedback module is used to collect the actual changes in audience retention rate and product click conversion rate after the execution of the director's switching action sequence, and generate strategy gradient feedback signals to update the network parameters of the near-end strategy optimization network.

[0058] The data acquisition module extracts key frames at a preset frame rate and extracts color, texture and motion features to form a sequence of image features. It then calculates the image quality score based on the sharpness and exposure of the key frames from each camera position.

[0059] Audience behavior data, product interaction data, and sales data are collected according to a preset sampling period. Audience behavior data includes the number of real-time viewers and the frequency of bullet comments, product interaction data includes the number of product clicks and the number of times products are added to the cart, and sales data includes the number of orders placed and the real-time transaction amount. A time-series data stream is generated by aligning the data with timestamps.

[0060] Specifically, in order to reduce the computational load of video processing and provide a data foundation for subsequent phased broadcasting, it is necessary to perform keyframe-level feature extraction and quality assessment on multi-camera video streams; at the same time, in order to eliminate the temporal deviation of multi-source heterogeneous data and support the temporal modeling of audience interest migration, it is necessary to align audience behavior, product interaction and sales data according to a unified time benchmark.

[0061] The data acquisition module extracts key frames from the video stream of each camera position according to a preset frame rate. For the i-th key frame of the j-th camera position, it extracts color features, texture features, and motion features respectively, and concatenates the three into a picture feature vector in columns. The vector is then arranged in time sequence to form the picture feature sequence of that camera position.

[0062] Image quality is scored based on the sharpness and exposure of keyframes from each camera position. The average Laplacian gradient magnitude is used as the sharpness index, and the average pixel brightness is used as the exposure index. The average sharpness of keyframes within the current window is taken. With average exposure Then the image quality score of the j-th camera position for:

[0063] ,in To preset the ideal exposure value (unit: pixel brightness value, range (0, 255]); and These are the weighting coefficients, and their sum is 1;

[0064] The average gradient magnitude of the Laplacian operator is calculated by summing the second-order grayscale change rates in the horizontal and vertical directions for each pixel in the keyframe image, taking the absolute values, and then calculating the arithmetic mean of the entire image. A larger value indicates higher edge sharpness and detail retention, resulting in a clearer image. This calculation method is a well-known image sharpness evaluation technique in the art, and its implementation is existing technology; therefore, the specific convolution kernel parameters and discretization derivation process will not be elaborated here.

[0065] Simultaneously, multi-source data is collected according to a preset sampling period (unit: seconds). Among them, audience behavior data includes the number of real-time viewers and the frequency of bullet comments (unit: comments / minute), product interaction data includes the number of product clicks and the number of times added to cart, and sales data includes the number of orders placed and the real-time transaction amount (unit: yuan). The above multi-source data are aligned into a joint observation vector using timestamps as keys, and time-series data streams are generated by arranging them in time sequence.

[0066] Therefore, by using a keyframe extraction mechanism, the computational load of video analysis is reduced from the full frame sequence to the sampled frame sequence, thereby reducing the computational load of subsequent modules. By quantifying the image quality using both clarity and exposure indicators, the director can prioritize the use of camera positions that meet the image quality standards, thus avoiding the impact of single-channel video quality degradation on the live broadcast presentation.

[0067] Existing e-commerce live streaming control systems lack the ability to understand the semantics of the live stream content. Camera switching relies on manual experience or fixed rules, and cannot adaptively adjust to different stages of the live stream. Therefore, the live stream content analysis module generates stage-specific semantic vectors through the following steps:

[0068] The steps for generating stage semantic vectors are as follows:

[0069] S301. Calculate the similarity between the color and texture features in the feature sequence of each camera position and the preset explanation scene template, display scene template and interactive scene template, and select the scene type with the highest similarity as the current live broadcast scene type.

[0070] The color and texture feature vectors from the feature sequences of each camera position are concatenated to form the image content feature. The color feature vector's dimensions represent the pixel count percentage of each bin in the color histogram (dimensionless), and the texture feature vector's dimensions represent the gray-level co-occurrence matrix statistics (dimensionless). The cosine similarity between the image content feature u and the preset scene template is calculated.

[0071] ;in Features of the image content Let be the k-th preset scene template vector, where k=1 corresponds to the explanation scene template, k=2 corresponds to the display scene template, and k=3 corresponds to the interactive scene template. The dimensions of each template vector are the same as u. Let be the L2 norm of vector u, used to normalize the modulus (dimensionless) of image content features. For vectors The L2 norm is used to normalize the modulus (dimensionless) of the k-th scene template.

[0072] Values Dimensionless, selected The scenario type corresponding to the maximum value is used as the current live streaming scenario type;

[0073] There is a temporal correlation between adjacent frames of a video, and moving objects in the picture will shift their positions between frames. To describe this shift, the video frame is divided into several pixel blocks (such as 8×8 or 16×16 pixels). Each pixel block is matched between adjacent frames to obtain the motion vector m of the pixel block. Its direction represents the direction of movement of the pixel block, and its magnitude represents the distance of movement of the pixel block (in pixels).

[0074] S302. Based on the spatial distribution of motion vector amplitude in the motion feature sequence, the area of ​​concentrated motion is determined as the product display area, and the color histogram and contour features of the area are extracted as product display features; specifically, the color histogram (pixel percentage of each color bin, dimensionless) and contour feature vector (coordinate sequence of contour key points, unit is pixel) of the area are extracted, and the two are concatenated as product display features.

[0075] Therefore, by using the spatial distribution of motion vector amplitude, the image is divided into regions according to the level of motion activity. When the anchor displays a product, the area where hand movements are concentrated is automatically locked, eliminating the need for manual labeling of the product's location. At the same time, the color and contour features of this area are extracted to provide a visual representation of the product for the generation of semantic vectors in subsequent stages, enabling the system to perceive when the anchor is displaying what kind of product.

[0076] S303. Based on the direction of increase or decrease of the number of viewers, the frequency of bullet comments, and the number of orders in the time-series data stream at adjacent sampling times, combined with whether the product display area exists in the current keyframe and the number of consecutive frames, determine whether the current live broadcast stage is the explanation period, the conversion period, or the interaction period.

[0077] Specifically, based on the time-series data stream, the increase or decrease direction of the number of real-time viewers, the frequency of bullet comments, and the number of orders at adjacent sampling times is calculated, and denoted as follows: , , (Positive sign indicates increase, negative sign indicates decrease, zero indicates no change, all are dimensionless direction indicators). Detect whether the product display area exists in the current keyframe and count the number of consecutive frames it has existed since its first detection (dimensionless count).

[0078] Therefore, the criteria for determining the stage are as follows:

[0079] Conversion period determination logic: When the product display area exists in the current keyframe, it indicates that the host is displaying or listing the product; if the sign is positive, it indicates that viewer ordering behavior has increased in adjacent sampling times, and product sales are active. When both conditions are met, it means that the live stream is currently in a critical period for promoting purchases, and is determined to be in the conversion period.

[0080] Interaction Period Determination Logic: When the product display area exists in the current keyframe, and the host is also displaying product-related information; if the keyframe is positive but not negative, it indicates that viewer interaction via chat has increased, but order placement has not increased accordingly, suggesting that viewer attention is focused on asking questions and commenting rather than making purchases. This situation is determined to be an interaction period.

[0081] Explanation Period Judgment Logic: When the above-mentioned conditions for the conversion period and interaction period are not met, including the absence of a product display area (the host has not performed product display operations), or although it exists, the frequency of bullet comments and the number of orders do not show an increasing trend, it indicates that the live broadcast is currently in the regular product introduction or host transition narration stage, and is judged as the explanation period.

[0082] S304. Encode the stage type and live streaming scenario type separately, and concatenate them with the product display features to generate a stage semantic vector.

[0083] The stage types and live streaming scene types are encoded using one-hot vectors: Assuming a total of 3 stage types (explanation period, conversion period, interaction period), the current stage type is encoded as a 3-dimensional one-hot vector; assuming a total of 3 live streaming scene types (explanation scene, display scene, interaction scene), the current live streaming scene type is encoded as a 3-dimensional one-hot vector.

[0084] Specifically, due to the high noise, nonlinearity, and time-delay characteristics of viewer behavior data (viewing, interaction, and conversion) in live streams, discrete clicks or bullet comments cannot directly reflect the evolution of their implicit interest states. To achieve temporal modeling of viewer interest transfer, it is necessary to transform the raw time-series data into a quantifiable sequence of implicit interest states. Therefore, the viewer behavior analysis module adopts a method combining sliding window feature extraction and state matching, as detailed below:

[0085] The establishment of the implicit interest state sequence includes:

[0086] The audience behavior analysis module slides and divides the time-series data stream according to a preset time window, extracts the viewing behavior features, interaction behavior features and conversion behavior features in each window, and generates a multi-dimensional behavior feature sequence by arranging them in time sequence.

[0087] K implicit interest state categories are predefined, and each category corresponds to a state representation vector. The state representation vector is determined by the cluster center obtained by clustering the feature vectors of the historical live data window. K is an integer greater than 1.

[0088] For the multi-dimensional behavioral features of each time window, calculate the cosine similarity between them and the representation vectors of each state. Map the window to the latent interest state category with the largest cosine similarity value and arrange them in the window time order to generate a latent interest state sequence.

[0089] As an optional embodiment, the time-series data stream is subjected to sliding segmentation according to a preset time window. Let the length of the time window be W seconds, and the sliding step be S seconds (S<W), so that a total of T consecutive time windows are obtained. For the t-th time window, viewing behavior features (the mean value of the number of real-time viewers in the window and the slope of the change trend, and the mean value of bullet comment frequency), interaction behavior features (the total number of commodity clicks, the total number of add-to-cart operations, and the click-to-view ratio), and conversion behavior features (the total number of orders placed, the mean value of real-time transaction volume, and the add-to-cart-to-order conversion rate) are extracted. After the above features are normalized, they are spliced into a window feature vector, which is arranged in time sequence to generate a multi-dimensional behavior feature sequence.

[0090] K implicit interest state categories are predefined (such as "casual browsing", "interest triggering", "deep attention", "purchase intention", and "hesitation and comparison"), and the state representation vectors are determined by the cluster centers obtained by performing K-Means clustering on the window feature vectors of historical live streaming data. For the feature vector of each time window, the cosine similarity between the feature vector and each state representation vector is calculated, the implicit interest state category corresponding to the maximum similarity value is selected as the implicit interest state of the window, and the implicit interest states are arranged in the time sequence of the windows to generate an implicit interest state sequence.

[0091] Thus, the discrete audience behavior data is mapped to an implicit interest state chain with clear semantics, which intuitively depicts the interest evolution trajectory of the audience in different stages from "casual browsing" to "purchase intention". The implicit interest state sequence not only provides a discrete state chain basis for the subsequent construction of the state transition probability matrix, but also provides an encodable interest state vector for the data fusion module, so that the director system can perceive the current psychological stage of the audience, thereby making more accurate picture scheduling decisions.

[0092] The audience interest migration trajectory is modeled through the state transition probability matrix, and the interest state vector is output, which is specifically as follows:

[0093] Count the frequency of state transitions between adjacent time windows in the implicit interest state sequence, and perform row normalization on the outgoing frequency of each state to construct a state transition probability matrix;

[0094] Based on the state transition probability matrix, the prediction probability of being in each implicit interest state category in the next time window is calculated, and the implicit interest state category with the maximum prediction probability is taken as the predicted interest state;

[0095] The implicit interest state corresponding to the current time window and the predicted interest state are respectively encoded and spliced to generate an interest state vector.

[0096] The encoding adopts a one-hot encoding method: let the total number of implicit interest state categories be C, and the serial number of the implicit interest state category corresponding to the current time window be z (z∈{1,2,…,C}), then this state generates a C-dimensional vector after one-hot encoding The z-th dimension is 1, and the other dimensions are 0; similarly, the predicted interest state is encoded as a C-dimensional vector. ;Will and By concatenating the columns, a 2C-dimensional interest state vector is generated.

[0097] As an optional embodiment, the audience behavior analysis module constructs a state transition probability matrix based on the generated implicit interest state sequence to model the audience interest migration trajectory and generates the interest state vector at the current moment.

[0098] The construction process consists of the following three steps:

[0099] First, count the frequency of state transitions between adjacent windows. Scan adjacent window states in the hidden interest state sequence one by one. If the previous state is i and the next state is j, increment the count value of the transition from state i to state j by 1. After traversing all adjacent window pairs, obtain the transition frequency between each state pair.

[0100] Second, normalize the state transition probability matrix. Let the total number of latent interest state categories be L. Normalize the transition frequency of each state by row to construct the state transition probability matrix. , of which elements The calculation method is as follows: ,in Let be the frequency of transitions from state i to state j. This represents the total number of implicit interest state categories. The total frequency of transitions from state i to all states (i.e., row-normalized). This represents the probability that the next window will transition to state j when the current window is in state i.

[0101] Third, calculate the predicted probability of the next time window being in each latent interest state category based on the state transition probability matrix. Let the latent interest state of the current time window be i, then the predicted probability of the next window being in state j is... The category of the latent interest state with the highest predicted probability is used as the predicted interest state;

[0102] Fourth, perform one-hot encoding on the implicit interest state and the predicted interest state corresponding to the current time window, and concatenate them to generate an interest state vector.

[0103] As a result, the system not only understands the current state of interest of the audience, but also predicts the state of interest that the next window is most likely to move to. This prediction result changes the director's decision-making from "following after the audience has moved" to "preparing in advance before the move occurs", eliminating the inherent delay in the detection-decision-execution link.

[0104] The method for fusing the joint state vector is as follows:

[0105] The stage semantic vector and the interest state vector are concatenated to obtain the concatenated vector; the image quality scores of each camera position are arranged in order of camera position to form a quality vector.

[0106] A fully connected network is used to map the concatenated vector to the same dimension as the quality vector. The mapping result is then added element-wise to the quality vector to generate a joint state vector.

[0107] The fully connected network consists of an input layer, a hidden layer, and an output layer. The number of neurons in the input layer is equal to the dimension of the concatenated vector, the number of neurons in the hidden layer is 64, and the number of neurons in the output layer is R (where R is the total number of positions). The activation function of the hidden layer is the ReLU function, and no activation function is used in the output layer. The network parameters are randomly initialized and updated synchronously with the policy network and the value network during the training of the reinforcement learning network.

[0108] Specifically, the stage semantic vector output by the live content analysis module only reflects the state of the live content (what stage it is currently in, what products are being displayed), the interest state vector output by the audience behavior analysis module only reflects the state of the audience (what stage the audience's interest is currently in), and the image quality scores of each camera position output by the data acquisition module only reflect the image availability of the device.

[0109] If the three types of information are directly input into the subsequent broadcast decision-making module, the decision network will find it difficult to simultaneously weigh the three factors: "what kind of picture is needed at the current stage", "what the audience wants to see" and "which camera position has the required picture quality", which may lead to biased decisions. Therefore, they need to be integrated.

[0110] As an optional implementation, the stage semantic vector is output by the live stream content analysis module, carrying information on the current live stream stage type and product display characteristics; the interest state vector is output by the audience behavior analysis module, carrying information on the current audience's implicit interest state. Both are semantic-level feature representations, differing in dimensions but both describing high-level states during the live stream. Concatenating the two columns yields a concatenated vector, achieving a unified representation of semantic information on both the live stream content and audience behavior sides.

[0111] The image quality score of each camera position is calculated and output in real time by the data acquisition module. The image quality scores of each camera position are arranged in a preset fixed camera position order to form a quality vector. This vector preserves the image quality differences of each camera position at the current moment, providing a quality basis for the director's decision on camera position selection.

[0112] Since the dimensions of the concatenated vector and the quality vector are usually different, a fully connected network is needed to map the concatenated vector to the same R-dimensional space as the quality vector, resulting in a mapped vector. The role of the fully connected network is to linearly or non-linearly transform semantic features from the original dimensional space to the same dimension as the quality space, enabling the fusion of corresponding positions of the two types of features in the same dimension. The mapped vector and the quality vector are added element-wise to generate a joint state vector.

[0113] The element-wise addition fusion method ensures that each dimension of the joint state vector contains both the image quality information of the corresponding camera position and the semantic information aligned with the dimensions. After the subsequent phased broadcast decision module reads the joint state vector, it can simultaneously perceive three types of information: "what stage of the live broadcast is currently in", "what is the current interest status of the audience", and "which camera position has the best image quality". This allows it to make the optimal camera switching decision that takes into account the content rhythm, audience psychology and image quality.

[0114] The predefined subset of actions includes the action subset for the explanation phase, the action subset for the conversion phase, and the action subset for the interaction phase;

[0115] The action subset during the explanation period includes: outputting a panoramic view of the anchor, outputting a close-up view of the product, and a picture-in-picture composite image that overlays the close-up view of the product onto a designated area of ​​the panoramic view of the anchor; the transition method is either a hard cut transition or a fade-in / fade-out transition;

[0116] The conversion period action subset includes: multi-angle close-up shots of each product are rotated and output at preset time intervals, and a guide combination screen that overlays the product close-up shots with the purchase link information layer; the transition methods include hard cut transitions and zoom transitions that move from panoramic shots to close-up shots;

[0117] The interactive action subset includes: a bullet screen overlaying the bullet screen text layer onto the current camera view, a camera view outputting the anchor's reaction, and a combined view of connecting viewers' video windows overlaid on a designated area of ​​the anchor's screen; the transition method is a dissolve transition;

[0118] When the phased broadcast decision module switches between the current live broadcast phases, the broadcast actions of the last N windows of the previous phase are only allowed to transition to the first action of the next phase using a specified transition method, where N is a preset positive integer.

[0119] Specifically, the primary goal during the explanation phase is clarity of information delivery. During this time, the host introduces the product, and the audience is in a state of information reception. The action subset includes a wide-angle camera angle for the host to preserve their facial expressions and body language, a close-up camera angle for the product to showcase its details, and a picture-in-picture composite to balance the dual information needs of the host's explanation and product demonstration. Transitions are limited to hard cuts or fade-in / fade-out transitions to avoid complex transitions that might distract the audience and ensure the continuity of information delivery.

[0120] The conversion period prioritizes driving purchase conversion. During this time, the host guides viewers through product promotions, putting them in a purchase decision-making state. The action subset utilizes multi-angle close-up rotations, rapidly changing camera angles to showcase the product's comprehensive details and eliminate viewer doubts about its appearance. A guide sequence combines product close-ups with purchase links, shortening the path from viewing to clicking to buy. Transitions employ rapid, hard cuts to create a sense of urgency, and zoom transitions from wide shots to close-ups to simulate a focusing effect, enhancing the product's visual impact.

[0121] The interactive period prioritizes enhancing audience engagement. During this time, the host responds to audience comments or engages in live interactions, placing the audience in a state of social participation. A subset of actions is configured with overlaid visuals of comments to visualize audience feedback, camera angles to capture the host's facial expressions in response to audience comments, and combined live interaction shots to present a real-time dialogue between the audience and the host. A dissolve transition technique is used, its slow, fading effect creating a relaxed social atmosphere that matches the emotional tone of the interactive period.

[0122] When the live broadcast transitions from the explanation phase to the conversion phase, the director's actions in the last N windows of the explanation phase are only allowed to transition to the first action of the conversion phase (multi-angle close-up rotation or guide shot combination). Transitions across phases, such as dissolve transitions, are prohibited. Similarly, when transitioning from the conversion phase to the interaction phase, the last N windows are only allowed to transition with hard cuts; when transitioning from the interaction phase to the explanation phase, the last N windows are only allowed to transition with fade-in / fade-out transitions.

[0123] This interlocking mechanism encodes operational rules into uncrossable action boundaries, ensuring the exploration space of reinforcement learning while excluding unreasonable cross-stage action combinations from the selection range. "Uncrossable actions" means that actions from two stages are not allowed to be called simultaneously within the same decision window.

[0124] Reinforcement learning networks are proximal policy optimization networks, including policy networks and value networks;

[0125] The policy network takes the joint state vector as input and outputs the selection probability value of each director action in the action subset corresponding to the current stage. The sum of the selection probability values ​​of all executable director actions is 1.

[0126] The value network takes the joint state vector as input and outputs the predicted value of the cumulative reward that can be obtained from the current state until the end of the live stream.

[0127] Specifically, the policy network is used to generate the selection criteria for the director's actions. The last layer of the policy network uses the Softmax activation function to map the original score of the network output to the [0,1] interval, and the sum of the selection probability values ​​of all executable director actions is 1, forming a standardized probability distribution. The director system samples actions according to this probability distribution. The higher the probability value of an action, the greater the possibility of it being selected. At the same time, it retains the exploration space for other actions and avoids getting stuck in a fixed switching mode with local optima.

[0128] The value network is used to evaluate the quality of the current broadcast setup. The predicted value reflects the expected long-term revenue under the current state—if the current camera layout, shot composition, and live broadcast stage are highly compatible with audience interests, the cumulative reward for subsequent steps will be higher, and the value network will output a higher predicted value; conversely, it will output a lower predicted value. During training, the error between the value network's predicted value and the actual cumulative reward will be fed back as an optimization signal to update the network parameters.

[0129] The policy network and value network work together: the policy network is responsible for "making decisions," that is, selecting specific broadcast actions; the value network is responsible for "making judgments," that is, evaluating the quality of the current state. They are updated alternately within the framework of the proximal policy optimization algorithm, achieving a balance between stability and convergence speed in the broadcast policy.

[0130] The reward function of a reinforcement learning network is configured with differentiated weights according to the current stage, specifically as follows:

[0131] Explanation period: The reward value is obtained by weighted summation of picture quality score, change in audience retention rate, and product click-through rate; Conversion period: The reward value is obtained by weighted summation of product click-through conversion rate increment, change in audience retention rate, and picture quality score; Interaction period: The reward value is obtained by weighted summation of bullet screen interaction rate increment, change in audience retention rate, and product click-through rate.

[0132] The bullet screen interaction rate is the ratio of the number of new bullet screens during the statistical period to the number of real-time viewers at the beginning of the statistical period, and is dimensionless; the bullet screen interaction rate increment is the difference between the bullet screen interaction rate of the current statistical period and the bullet screen interaction rate of the previous statistical period.

[0133] As an optional implementation, the clarity of information delivery is the primary goal during the explanation phase. At this stage, viewers are in the product understanding phase; if conversion metrics are used as the driving force, the website will prematurely pursue sales while neglecting content presentation. Therefore, image quality rating is used as the primary reward metric, guiding the website to prioritize the clarity of the output images, supplemented by changes in viewer retention rate and product click-through rate.

[0134] The primary goal during the conversion period is to drive purchase conversion. At this stage, viewers already have a purchase intention, and the directing strategy should focus on enhancing the visual impact of the products. Therefore, the increase in product click-through rate should be used as the primary reward indicator, guiding the network to choose the combination of screens that can encourage viewers to click and place an order, supplemented by changes in viewer retention rate and screen quality ratings.

[0135] The primary goal during the interactive period is to enhance audience engagement. At this stage, the level of emotional engagement among viewers influences the atmosphere and stickiness of the live stream. Therefore, the increase in bullet screen interaction rate is used as the primary reward indicator, guiding the network to select interactive visual combinations that stimulate bullet screen participation, supplemented by changes in audience retention rate and product click-through rate.

[0136] Each reward indicator was subjected to max-min normalization before weighted summation to eliminate dimensional differences and ensure that the weight coefficients accurately reflect the relative importance of each indicator at the current stage.

[0137] The linkage feedback module collects the actual changes in audience retention rate and product click conversion rate after the execution of the director's switching action sequence, and calculates the actual reward value according to the reward weight corresponding to the current live broadcast stage; the difference between the actual reward value and the cumulative reward prediction value output by the value network is used to obtain the advantage value; the strategy gradient is calculated based on the advantage value, and a strategy gradient feedback signal is generated to update the network parameters of the near-end strategy optimization network.

[0138] The product click-through rate is the ratio of the number of product orders to the total number of product clicks within the statistical period, and is dimensionless; the increment of the product click-through rate is the difference between the product click-through rate of the current statistical period and the product click-through rate of the previous statistical period.

[0139] Specifically, the linkage feedback module is used to transform the actual business effects after the director's actions are executed into updated network parameters, forming a closed-loop mechanism of "decision-execution-feedback-optimization". The specific process is as follows:

[0140] 1. Collect actual business feedback: After the director's switching action sequence is executed, collect the actual change in audience retention rate within the preset statistical period. Increase in product click-through rate .in That is, the difference between the audience retention rate at the end of the statistical period and the beginning of the period; That is, the difference between the product click-through rate at the end of the statistical period and the beginning of the period;

[0141] The audience retention rate is the proportion of viewers whose continuous viewing time exceeds a preset threshold within the statistical period to the total number of viewers; the change in audience retention rate is the difference between the audience retention rate at the end of the current statistical period and the audience retention rate at the beginning.

[0142] II. Calculate the actual reward value: Read the reward weight configuration corresponding to the current live broadcast stage, and... , The actual reward value for this broadcast action sequence is obtained by weighting and summing various indicators such as picture quality score. .

[0143] If the current period is the explanation period Based on picture quality rating, The result is obtained by weighted summation of product click-through rates; if it is the conversion period, The primary indicator used in the calculation is the bullet screen interaction rate; if it is an interactive period, the increase in bullet screen interaction rate is used as the primary indicator in the calculation. All indicators are subjected to maximum-minimum normalization before weighting.

[0144] III. Calculate the advantage value: Read the cumulative reward prediction value of the value network outputting the joint state vector of the previous time step. The actual reward value and By subtracting, we obtain the dominance value. :

[0145] ;in for The cumulative reward prediction value output by the joint state vector at time step; Let be the cumulative reward prediction value output by the joint state vector at time t; This is a discount factor that controls the rate of decay of long-term rewards, with a value ranging from 0 to 1.

[0146] If the advantage value A value >0 indicates that the actual effect of the action sequence is better than the value network's expectation, and the action should be encouraged; if the dominance value is... <0 indicates that the actual effect is lower than expected, and the action should be suppressed;

[0147] IV. Generate policy gradient backpropagation signals and update network parameters. Based on the advantage value. Calculate the policy gradient: when When, increase the probability that the policy network will select the current directing action given the joint state vector; when When the probability decreases, the value network updates its parameters to minimize the error between the actual reward and the predicted value. Both the policy network and value network parameters are iteratively optimized using stochastic gradient descent, ensuring the guiding strategy adjusts towards a better outcome after each feedback.

[0148] In summary, this system integrates and represents three types of heterogeneous information: live broadcast semantics, audience interest evolution, and camera image quality. This allows the director to simultaneously perceive the content requirements of the current live broadcast stage, the audience's implicit interest status, and the available image quality of each camera position, thus avoiding switching deviations caused by a single information source.

[0149] By modeling the trajectory of audience interest migration using a state transition probability matrix, the system can predict trends and adjust presentation strategies in advance before interest inflection points occur. It also limits the range of executable director actions and transition methods in stages, encoding business operation logic into network action constraints to avoid inappropriate stage outputs and scene changes that contradict the live broadcast rhythm.

[0150] The phased reward configuration enables the network to automatically switch optimization targets at different live broadcast stages, while the linkage feedback module introduces the actual business results into the strategy gradient update, so that the optimization direction of the broadcast strategy is directly aligned with the actual operation indicators of the live broadcast room, forming a complete closed loop of decision-making, execution, feedback and optimization.

[0151] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-camera intelligent directing and real-time data linkage analysis system for e-commerce live streaming, characterized in that: include: The data acquisition module is used to acquire multi-camera video streams and extract image feature sequences and image quality scores, while simultaneously acquiring audience behavior data, product interaction data and sales data to generate time-series data streams. The live streaming content analysis module identifies the live streaming scene type and product display area based on the image feature sequence, and divides the current live streaming stage into explanation period, conversion period and interaction period by combining the time-series data stream, and generates stage semantic vectors. The audience behavior analysis module is used to extract multi-dimensional behavioral features from the time-series data stream to establish a hidden interest state sequence, model the audience interest migration trajectory through the state transition probability matrix, and output the interest state vector. The state transition probability matrix is ​​used to predict the audience interest state distribution in the next time window. The data fusion module is used to fuse the stage semantic vector, interest state vector, and the image quality score of each camera position into a joint state vector; The phased broadcast decision module is used to determine the range of executable broadcast actions from a predefined subset of actions based on the current live broadcast phase. The subset of actions is configured with different multi-camera shot combination rules and transition methods for the explanation phase, conversion phase, and interaction phase, and cannot be mixed across phases. The joint state vector is input into the reinforcement learning network to output the broadcast switching action sequence. The reward function of the reinforcement learning network is configured with differentiated weights according to the current phase. The linkage feedback module is used to collect the actual changes in audience retention rate and product click conversion rate after the execution of the director switching action sequence, calculate the advantage value, and generate a strategy gradient backhaul signal based on the advantage value to update the network parameters of the reinforcement learning network.

2. The e-commerce live streaming multi-camera intelligent directing and real-time data linkage analysis system according to claim 1, characterized in that: The data acquisition module extracts key frames at a preset frame rate and extracts color, texture and motion features to form the image feature sequence, and calculates the image quality score based on the clarity and exposure of the key frames from each camera position. The audience behavior data, product interaction data, and sales data are collected according to a preset sampling period. The audience behavior data includes the number of real-time viewers and the frequency of bullet comments. The product interaction data includes the number of product clicks and the number of times products are added to the cart. The sales data includes the number of orders placed and the real-time transaction amount. The time-series data stream is generated by aligning the data with timestamps.

3. The e-commerce live streaming multi-camera intelligent directing and real-time data linkage analysis system according to claim 1, characterized in that: The steps for generating the semantic vector for that stage are as follows: S301. Calculate the similarity between the color features and texture features in the image feature sequence of each camera position and the preset scene template, and select the scene type with the highest similarity as the current live broadcast scene type. S302. Based on the spatial distribution of motion features in the image feature sequence, the area of ​​concentrated motion is determined as the product display area, and the color histogram and contour features of the area are extracted as product display features. S303. Based on the direction of increase or decrease of the number of real-time viewers, the frequency of bullet comments, and the number of orders in the time-series data stream at adjacent sampling times, and combined with whether the product display area exists in the current keyframe and the number of consecutive frames, determine the current live broadcast stage. S304. Encode the stage type and the live streaming scenario type respectively, and concatenate them with the product display features to generate the stage semantic vector.

4. The e-commerce live streaming multi-camera intelligent directing and real-time data linkage analysis system according to claim 1, characterized in that: The establishment of the implicit interest state sequence includes: The audience behavior analysis module slides and divides the time-series data stream according to a preset time window, extracts viewing behavior features, interaction behavior features and conversion behavior features in each window, and generates a multi-dimensional behavior feature sequence by arranging them in time sequence. K implicit interest state categories are predefined, and each category corresponds to a state representation vector. The state representation vector is determined by the cluster center obtained by clustering the feature vectors of historical live data windows. K is an integer greater than 1. For each time window, the multi-dimensional behavioral features are calculated, and their cosine similarity with each state representation vector is calculated. The window is then mapped to the latent interest state category with the largest cosine similarity value, and the latent interest state sequence is generated by arranging the windows in time sequence.

5. The e-commerce live streaming multi-camera intelligent directing and real-time data linkage analysis system according to claim 4, characterized in that: The audience interest migration trajectory is modeled using the state transition probability matrix, and the interest state vector is output as follows: The frequency of state transitions between adjacent time windows in the implicit interest state sequence is statistically analyzed, and the transition frequency of each state is normalized by row to construct the state transition probability matrix. Based on the state transition probability matrix, calculate the predicted probability of the next time window being in each latent interest state category, and take the latent interest state category with the highest predicted probability as the predicted interest state. The implicit interest state corresponding to the current time window and the predicted interest state are encoded respectively, and then concatenated to generate the interest state vector.

6. The e-commerce live streaming multi-camera intelligent directing and real-time data linkage analysis system according to claim 5, characterized in that: The method for fusing the joint state vector is as follows: The stage semantic vector and the interest state vector are concatenated to obtain a concatenated vector. The image quality scores of each camera position are arranged in order of camera position to form a quality vector. The concatenated vector is mapped to the same dimension as the quality vector through a fully connected network, and the mapping result is concatenated with the quality vector to generate the joint state vector.

7. The e-commerce live streaming multi-camera intelligent directing and real-time data linkage analysis system according to claim 6, characterized in that: The predefined subset of actions includes the action subset for the explanation period, the action subset for the transformation period, and the action subset for the interaction period; The action subset during the explanation period includes: outputting a panoramic view of the anchor, outputting a close-up view of the product, and a picture-in-picture composite image that overlays the close-up view of the product onto a designated area of ​​the panoramic view of the anchor; the transition method is either a hard cut transition or a fade-in / fade-out transition; The conversion period action subset includes: multi-angle close-up shots of each product are rotated and output at preset time intervals, and a guide combination screen that overlays the product close-up shots with the purchase link information layer; the transition methods include hard cut transitions and zoom transitions that move from panoramic shots to close-up shots; The interactive action subset includes: a bullet screen overlaying the bullet screen text layer onto the current camera view, a camera view outputting the anchor's reaction, and a combined view of connecting viewers' video windows overlaid on a designated area of ​​the anchor's screen; the transition method is a dissolve transition; When the phased broadcast decision module switches between the current live broadcast phases, the broadcast actions of the last N windows of the previous phase are only allowed to transition to the first action of the next phase using the transition method configured for the subset of actions in that phase, where N is a preset positive integer.

8. The e-commerce live streaming multi-camera intelligent directing and real-time data linkage analysis system according to claim 7, characterized in that: The reinforcement learning network is a proximal policy optimization network, which includes a policy network and a value network; The policy network takes the joint state vector as input and outputs the selection probability value of each director action in the action subset corresponding to the current stage. The value network takes the joint state vector as input and outputs the predicted cumulative reward value that can be obtained from the current state until the end of the live broadcast.

9. The e-commerce live streaming multi-camera intelligent directing and real-time data linkage analysis system according to claim 8, characterized in that: The reward function of the reinforcement learning network is configured with differentiated weights according to the current stage, specifically as follows: During the explanation period: the reward value is obtained by weighted summation of the image quality score, the change in audience retention rate, and the product click-through rate; during the conversion period: the reward value is obtained by weighted summation of the increase in product click-through conversion rate, the change in audience retention rate, and the image quality score. Interaction period: The reward value is obtained by weighted summation of the increase in bullet screen interaction rate, the change in audience retention rate, and the product click rate.

10. The e-commerce live streaming multi-camera intelligent directing and real-time data linkage analysis system according to claim 9, characterized in that: When the predicted interest state is a purchase intention state and the current live broadcast stage is the explanation period, the phased broadcast decision module switches the range of executable broadcast actions from the subset of actions in the explanation period to the subset of actions in the conversion period one time window in advance.

Images