A live material recommendation method and system based on big data analysis

Abstract

The application discloses a live material recommendation method and system based on big data analysis, relates to the technical field of streaming media and multimedia content retrieval, and comprises the following steps: S1, collecting live stream video data, performing space-time feature extraction through an improved TimeSformer model, and introducing a space-time Doppler frequency shift modulation mechanism to modulate original time attention weight; S2, outputting 3D scene features and corresponding camera projection parameters; S3, determining a 2D visual blind area; S4, calculating the initial spatial position of the material to be recommended; S5, outputting 3D superposition parameters; S6, generating 3D rendering materials; S7, outputting a live stream; and S8, analyzing the live stream, and performing real-time rendering and interactive display. The application overcomes the limitations of the traditional method, such as plane mapping violation, neglecting 3D space constraints and being driven by behavior data, and provides an efficient solution for live intelligent recommendation.

Description

Technical Field

[0001] This invention relates to the field of streaming media and multimedia content retrieval technology, and in particular to a method and system for recommending live streaming materials based on big data analysis. Background Technology

[0002] In the fields of live stream data processing and augmented reality, with the rapid evolution of immersive live streaming and real-time interaction demands, classic systems face severe computational and architectural challenges in intelligent recommendation and spatial fusion of live streaming content in complex dynamic scenes. Existing 3D content recommendation methods typically rely on saliency detection of 2D image planes and the overlay of content in manually labeled fixed regions. While this can achieve content insertion to some extent, it mainly depends on 2D pixel-level feature matching and preset planar coordinate mapping. This method, based solely on 2D planar perception, ignores the implicit 3D spatial geometry (such as scene depth, surface normals, etc.) and the dynamic spatiotemporal semantic relationships that evolve over time in the live video stream. This leads to spatial dimension reduction distortion during content localization, thus limiting the depth occlusion and perspective consistency between virtual content and real scenes in 3D space. Furthermore, classic recommendation methods often struggle to fully utilize the spatiotemporal statistical features in the live streaming behavior big data pool to dynamically constrain the content delivery space. They often rely solely on static rules or local image features for blind spot judgments, making them prone to visual occlusion and incongruity when handling high-frequency interactions and complex dynamic live streaming scenes. This severely degrades the visual quality of live streaming recommendations and the user interaction experience.

[0003] Therefore, how to provide a method and system for recommending live streaming content based on big data analysis is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] This invention proposes a live streaming content recommendation method and system based on big data analysis. By introducing an improved TimeSformer model with a spatiotemporal Doppler frequency shift modulation mechanism and a closed-loop feedback optimization process based on a live streaming behavior big data pool, it extracts a relative velocity scalar based on feature change rate mapping and derives a frequency shift scaling factor. It then performs spatiotemporal joint calibration and modulation on the original time attention weights, outputting accurate visual saliency maps and depth maps. Furthermore, by combining 3D scene reconstruction and back projection mapping of 2D visual blind spots, it maps the 2D blind spots to 3D space for spatial availability assessment to calculate the initial spatial position. Subsequently, it determines the 3D overlay parameters through local geometric deformation compensation and secondary correction of composition rules. Finally, it performs physically interactive rendering and shadow fusion on the content to be recommended by combining local lighting parameters. This process establishes a closed-loop pathway from "spatiotemporal dynamic features and behavioral perception" to "3D spatial blind spot mapping and lighting rendering," effectively eliminating spatial dimension reduction distortion and lighting fragmentation in the traditional 2D planar material overlay process. It ensures that the generated live-stream recommendation frames dynamically maintain a high degree of consistency between virtual materials and real scenes in 3D geometric perspective and lighting physics, achieving the technical effect of improving the spatial fit and visual immersion of material recommendations in complex dynamic live-streaming scenarios. This invention overcomes the limitations of traditional methods, such as planar mapping inconsistencies, neglect of 3D spatial constraints, and detachment from behavioral data-driven approaches, providing an efficient solution for intelligent live-stream recommendation.

[0005] A live streaming content recommendation method based on big data analysis according to an embodiment of the present invention includes the following steps: S1. Collect live video data, perform spatiotemporal feature extraction by improving the TimeSformer model, and obtain the original time attention weights; introduce a spatiotemporal Doppler frequency shift modulation mechanism, extract the relative velocity scalar based on the feature change rate mapping and derive the frequency shift scaling factor, modulate the original time attention weights after spatiotemporal joint calibration, and output visual saliency map, depth map and video key frame sequence after significant spatiotemporal perception. S2. Perform 3D scene analysis on the video keyframe sequence and corresponding depth map data, reconstruct the 3D spatial information of the live scene, output 3D scene features, and estimate the corresponding camera projection parameters. S3. Load the live streaming behavior big data pool and extract multi-source heterogeneous behavior data. Combine 3D scene features and visual saliency map to calculate the visual attention area and user attention distribution information. Then, determine the 2D visual blind spot by excluding the visual attention area. S4. Based on 3D scene features, 2D visual blind spots and camera projection parameters, map the 2D visual blind spots to three-dimensional space and perform spatial availability assessment to calculate the initial spatial position of the material to be recommended. S5. Fine-tune and optimize the initial spatial position by combining 3D scene features to determine the 3D overlay parameters of the recommended material that conform to the composition rules in three-dimensional space. S6. Based on the 3D scene features and corresponding ambient lighting information, combine the 3D overlay parameters to perform lighting rendering on the recommended material, blend the shadows and generate 3D rendered material. S7. Based on the 3D overlay parameters, the 3D rendered materials are blended into the three-dimensional space to generate live recommendation frames, which are then transmitted to the client via the network to output the live stream; S8: The client receives and parses the live stream, performs real-time rendering and interactive display, responds to interactive operations to generate real-time feedback data, and sends it back to the server to update the live stream behavior big data pool.

[0006] Optionally, the improved TimeSformer model includes a spatiotemporal embedding coding layer, a separate attention computation layer, a spatiotemporal Doppler frequency shift layer, and a saliency keyframe decoding output layer: The spatiotemporal embedding coding layer is used to perform temporal sampling on the input live stream video data to generate a video frame sequence, and to perform spatial segmentation and feature mapping on the video frames, injecting spatiotemporal location information to generate a video token sequence containing spatiotemporal information. The separate attention calculation layer is used to receive the video token sequence, calculate the correlation between each token along the spatial dimension to extract the global context features of the screen, and then calculate the correlation between each token along the temporal dimension to extract the dynamic change features between frames. After feature fusion, it outputs the spatiotemporal fusion feature sequence and the corresponding original temporal attention weight map. The spatiotemporal Doppler frequency shift layer is used to introduce an acoustic spatiotemporal Doppler frequency shift modulation mechanism, and the specific execution process includes: The feature change rate of the token at the same spatial location in adjacent time steps is calculated based on the spatiotemporal fusion feature sequence. The feature change rate is mapped to a signed relative velocity scalar according to the feature evolution direction, and the baseline propagation velocity scalar is accumulated along the temporal dimension. Calculate the algebraic difference between the reference propagation speed scalar and the relative speed scalar, obtain the ratio between the reference propagation speed scalar and the algebraic difference, and output the frequency shift scaling factor; Along the time dimension, a dynamic attenuation coefficient is generated based on the relative velocity scalar of the current time step. An exponential moving average is performed on the frequency shift scaling factor between the previous time step and the current time step, and the time-series cumulative scaling factor is output. Along the spatial dimension, propagation weights are constructed based on spatial similarity of feature change rates. Spatial neighborhood feature aggregation is performed to output a spatial correction velocity scalar. The spatial correction scaling factor is then recalculated based on the spatial correction velocity scalar. The target frequency shift scaling factor is obtained by fusing the temporal cumulative scaling factor and the spatial correction scaling factor. The target frequency shift scaling factor is then multiplied element-wise with the original temporal attention weight map to output the modulation temporal attention weight map. The saliency keyframe decoding output layer is used to reconstruct and focus spatial features on the spatiotemporal fusion feature sequence using the modulation temporal attention weight map as a dynamic prior to generate a semantic-level visual saliency map, and apply dynamic geometric constraints to output depth map data; then, based on the dynamic response abrupt change of the modulation temporal attention weight map along the time dimension, it locates the jump moment to output the video keyframe sequence.

[0007] Optionally, S2 specifically includes: S21. Based on the video keyframe sequence and the corresponding depth map data, perform multi-view feature matching and feature fusion to reconstruct the three-dimensional spatial information of the live scene and output 3D scene features. S22. Solve the initial projection matrix based on the initial spatial correspondence between 3D scene features and video keyframe sequences, and extract the spatial distribution principal axis based on the global geometric topology distribution statistical features of 3D scene features to construct a scene local coordinate system to eliminate absolute scale ambiguity. S23. Based on the initial projection matrix, backproject the 3D scene features to the video keyframe sequence to calculate the reprojection error distribution. Based on the reprojection error distribution, perform gradient backpropagation update on the initial projection matrix and 3D scene features, and output the optimized camera projection parameters and 3D scene features.

[0008] Optionally, S3 specifically includes: S31. Load the live streaming behavior big data pool and extract multi-source heterogeneous behavior data. Perform spatiotemporal alignment and normalization mapping on the multi-source heterogeneous behavior data and output a structured behavior feature sequence. S32. Jointly correlate the structured behavioral feature sequence with the visual saliency map in terms of spatial dimension, calculate the pixel-level attention response intensity of the visual saliency map and generate the spatial attention distribution, filter high-response connected components based on the numerical statistical features of the spatial attention distribution, and output the visual attention region. S33. Perform reverse masking on the visual saliency map based on the visual attention region, extract the spatial coordinates of the low response region, and output the 2D visual blind spot.

[0009] Optionally, S4 specifically includes: S41. Based on the 2D visual blind zone and camera projection parameters, perform back projection ray calculation to solve the ray clusters of the two-dimensional pixels corresponding to the 2D visual blind zone in the three-dimensional space, perform intersection operation between the ray clusters and the 3D scene features, and output the three-dimensional visual blind zone point cloud. S42. Calculate the local spatial distribution statistical features of the point cloud in the 3D visual blind zone, extract the bounding boundary and surface normal vector distribution of the point cloud cluster based on the local spatial distribution statistical features, evaluate the spatial availability based on the dispersion of the surface normal vector distribution, and output the spatial availability metric. S43. Based on the distribution statistical quantiles of spatial availability metrics, select high-availability 3D visual blind spot point cloud clusters, and combine the enclosing boundary and surface normal vector distribution of the high-availability 3D visual blind spot point cloud clusters to calculate the centroid coordinates and orientation anchor points, generating the initial spatial position of the material to be recommended.

[0010] Optionally, S5 specifically includes: S51. Based on 3D scene features and initial spatial position, calculate the local neighborhood geometric features of the initial spatial position in three-dimensional space, extract the curvature distribution and occlusion depth gradient of the neighborhood scene surface, and output the spatial constraint features. S52. Based on the curvature distribution and occlusion depth gradient of the spatial constraint features, calculate the local offset vector and rotation correction quaternion of the initial spatial position, perform spatial geometric deformation compensation on the initial spatial position, and output the fine-tuned spatial pose. S53. Combine the camera projection parameters to map the fine-tuned spatial pose onto the image plane to extract the projection contour. Construct a composition grid reference line based on the image boundary and the visual interest area. Calculate the offset of the projection contour relative to the composition grid reference line. Generate a scaling factor and translation compensation based on the offset. Perform secondary correction on the fine-tuned spatial pose and output 3D overlay parameters.

[0011] Optionally, S6 specifically includes: S61. Based on 2D pixel brightness gradient and 3D scene feature data, the 2D gradient is mapped to 3D space by combining camera imaging geometry, and the incident light direction vector is estimated by using the geometric constraints of 3D space gradient and surface normal vector. S62. Based on the surface normal vector sampling of 3D scene feature data, the observed light flux is obtained, and under the assumption of diffuse reflection, the reflectivity value is obtained by inversion based on the attenuation relationship between the observed light flux and the incident light direction, and the ambient lighting information is generated by aggregation. S63. Based on 3D scene features and ambient lighting information, extract the target point set in the spatial neighborhood corresponding to the 3D overlay parameters, extract the main direction of the incident light direction vector of the target point set to obtain the statistical main axis of the incident light direction, perform statistical averaging on the reflectivity values ​​to obtain the ambient reflectivity distribution, and stitch together to output the local lighting parameters; S64. Perform physical interaction calculations between the local lighting parameters and the material properties of the material to be recommended, solve the incident light attenuation coefficient and specular reflection distribution on the surface of the material to be recommended, perform photometric mapping and texture recoloring on the material to be recommended, and output lighting-adapted material. S65. Based on the incident light direction statistical principal axis in the local lighting parameters and the orientation anchor point in the 3D overlay parameters, calculate the occlusion projection area of ​​the material to be recommended in three-dimensional space, perform shadow gradient rendering on the occlusion projection area and overlay it onto the lighting adaptation material, and output the 3D rendered material.

[0012] Optionally, S7 specifically includes: S71. Based on the 3D overlay parameters, the 3D rendered material is mapped to the corresponding pixel coordinates of the original video frame. According to the 3D scene features and depth map data, the 3D rendered material and the original video frame are subjected to joint depth occlusion test and edge feathering fusion to generate a spatial fusion image frame. S72. Perform image quality statistical distribution evaluation on the spatially fused image frame, adaptively calculate the coding quantization parameters and bit rate control coefficients based on the image quality statistical distribution characteristics, and output the video frame to be transmitted. S73. Perform network-state-aware bitrate allocation and protocol encapsulation on the video frames to be transmitted, and send them down through the transmission channel and output a live stream suitable for terminal decoding and rendering.

[0013] Optionally, S8 specifically includes: S81: Receive the live stream and perform protocol decryption and decapsulation operations, extract the video frame to be transmitted and the encoded metadata, perform hardware decoding and pixel format conversion on the video frame to be transmitted, and output the decoded image frame; S82. Input the decoded image frame into the graphics rendering pipeline, extract the display timestamp statistical distribution characteristics of the decoded image frame, adaptively adjust the vertical synchronization beat and frame rendering timing based on the display timestamp statistical distribution characteristics, and execute screen drawing to output the interactive display interface. S83. Capture the touch event stream on the interactive display interface, analyze the spatial coordinate distribution and time interval statistical characteristics of the touch event stream, structure the spatial coordinate distribution and time interval statistical characteristics into interactive behavior vectors, and send the interactive behavior vectors back to the server to update the live broadcast behavior big data pool.

[0014] A live streaming content recommendation system based on big data analysis according to an embodiment of the present invention includes the following modules: The spatiotemporal feature extraction and modulation perception module is used to collect live video data. It performs spatiotemporal feature extraction by improving the TimeSformer model to obtain the original temporal attention weights. It introduces a spatiotemporal Doppler frequency shift modulation mechanism, extracts the relative velocity scalar based on the feature change rate mapping and derives the frequency shift scaling factor. After spatiotemporal joint calibration, it modulates the original temporal attention weights and outputs a visual saliency map, depth map and video key frame sequence through significant spatiotemporal perception. The 3D scene reconstruction and parameter estimation module is used to perform 3D scene analysis on video keyframe sequences and corresponding depth map data, reconstruct the 3D spatial information of the live scene, output 3D scene features, and estimate the corresponding camera projection parameters. The behavioral data fusion and blind spot determination module is used to load the live broadcast behavior big data pool and extract multi-source heterogeneous behavioral data. It combines 3D scene features and visual saliency maps to calculate the visual attention area and user attention distribution information, and determines the 2D visual blind spot by excluding the visual attention area. The blind spot spatial mapping and position calculation module is used to map the 2D visual blind spot to a three-dimensional space and perform spatial availability assessment based on 3D scene features, 2D visual blind spots and camera projection parameters, so as to calculate the initial spatial position of the material to be recommended. The spatial position fine-tuning and parameter determination module is used to fine-tune and optimize the initial spatial position by combining 3D scene features, and determine the 3D overlay parameters of the material to be recommended in three-dimensional space to conform to the composition rules. The lighting rendering and shadow blending module is used to perform lighting rendering on the recommended material based on the 3D scene characteristics and corresponding ambient lighting information, combined with 3D overlay parameters, to blend shadows and generate 3D rendered material. The spatial fusion and live stream output module is used to fuse 3D rendered materials into a three-dimensional space based on 3D overlay parameters to generate live recommendation frames, and transmit them to the client via the network to output the live stream; The client-side interaction and data feedback module is used to parse the live stream, perform real-time rendering and interactive display, respond to interactive operations to generate real-time feedback data, and send it back to the server to update the live stream behavior big data pool.

[0015] The beneficial effects of this invention are: This invention significantly improves the spatial fit and visual immersion of material recommendations in complex dynamic live streaming scenarios by introducing a spatiotemporal Doppler frequency shift modulation mechanism and a closed-loop feedback optimization process driven by big data on live streaming behavior. By innovatively constructing a relative velocity scalar and frequency shift scaling factor based on feature change rate mapping, spatiotemporal joint calibration and modulation are performed on the original temporal attention weights, effectively eliminating dynamic response lag and high-order structure distortion in the traditional spatiotemporal feature extraction process, achieving accurate capture of the visual saliency and depth perception of the live stream. Simultaneously, by employing back-projection ray mapping and spatial availability assessment mechanisms, 2D visual blind spots are accurately projected into 3D space. Combined with local geometric deformation compensation and secondary correction of composition rules to determine 3D overlay parameters, the constraint relationship between scene 3D geometry and user visual attention is deeply explored, effectively overcoming the spatial occlusion and perspective distortion problems caused by traditional 2D planar overlay. In addition, by combining local lighting parameters to perform physically interactive rendering and shadow gradient fusion on the recommended materials, and generating real-time feedback data in response to client-side interactive operations, the system updates the big data pool of behavior. During the material overlay stage, ambient lighting and shadows are integrated to enhance the physical realism of virtual entities. Furthermore, the user attention perception model is continuously optimized through data loop, thereby significantly improving the consistency of lighting and shadows, the robustness of interaction, and the accuracy of recommendations in live streaming recommendation screens. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of a live streaming content recommendation method based on big data analysis proposed in this invention; Figure 2 This is a flowchart illustrating the working principle of the improved TimeSformer model for a live streaming content recommendation method based on big data analysis proposed in this invention. Figure 3 This is a schematic diagram of the structure of a live streaming content recommendation system based on big data analysis proposed in this invention. Detailed Implementation

[0017] The invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0018] refer to Figure 1 and Figure 2 A live streaming content recommendation method based on big data analysis includes the following steps: S1. Collect live video data, perform spatiotemporal feature extraction by improving the TimeSformer model, and obtain the original time attention weights; introduce a spatiotemporal Doppler frequency shift modulation mechanism, extract the relative velocity scalar based on the feature change rate mapping and derive the frequency shift scaling factor, modulate the original time attention weights after spatiotemporal joint calibration, and output visual saliency map, depth map and video key frame sequence after significant spatiotemporal perception. S2. Perform 3D scene analysis on the video keyframe sequence and corresponding depth map data, reconstruct the 3D spatial information of the live scene, output 3D scene features, and estimate the corresponding camera projection parameters. S3. Load the live streaming behavior big data pool and extract multi-source heterogeneous behavior data. Combine 3D scene features and visual saliency map to calculate the visual attention area and user attention distribution information. Then, determine the 2D visual blind spot by excluding the visual attention area. S4. Based on 3D scene features, 2D visual blind spots and camera projection parameters, map the 2D visual blind spots to three-dimensional space and perform spatial availability assessment to calculate the initial spatial position of the material to be recommended. S5. Fine-tune and optimize the initial spatial position by combining 3D scene features to determine the 3D overlay parameters of the recommended material that conform to the composition rules in three-dimensional space. S6. Based on the 3D scene features and corresponding ambient lighting information, combine the 3D overlay parameters to perform lighting rendering on the recommended material, blend the shadows and generate 3D rendered material. S7. Based on the 3D overlay parameters, the 3D rendered materials are blended into the three-dimensional space to generate live recommendation frames, which are then transmitted to the client via the network to output the live stream; S8: The client receives and parses the live stream, performs real-time rendering and interactive display, responds to interactive operations to generate real-time feedback data, and sends it back to the server to update the live stream behavior big data pool.

[0019] In this embodiment, the improved TimeSformer model includes a spatiotemporal embedding coding layer, a separate attention computation layer, a spatiotemporal Doppler frequency shift layer, and a saliency keyframe decoding output layer: The spatiotemporal embedding coding layer is used to perform temporal sampling on the input live video data at a sampling rate of 25 frames per second to generate a video frame sequence. Each video frame is divided into 16-pixel by 16-pixel image blocks. A two-dimensional convolution operation with a kernel size of 16-16 and a stride of 16 is used to perform feature mapping on the image blocks. The mapped feature vectors are then added with a position offset of 0.01 times the temporal step index and a position offset of 0.01 millimeters the spatial horizontal and vertical coordinate indices to inject spatiotemporal position information and generate a video token sequence containing spatiotemporal information. The separate attention computation layer receives the video token sequence, calculates the correlation between tokens along the spatial dimension using scaled dot product to extract global contextual features of the screen, and then calculates the correlation between tokens along the temporal dimension using scaled dot product. At this point, the original temporal attention weight map is explicitly output. The spatiotemporal Doppler frequency shift layer is used to introduce an acoustic spatiotemporal Doppler frequency shift modulation mechanism. The specific execution process includes: Spatial and temporal features are concatenated along the channel dimension and fused through a linear layer to output a spatiotemporal fusion feature sequence. Based on the spatiotemporal fusion feature sequence, the absolute value of the feature difference of the token at the same spatial location in adjacent time steps is calculated and divided by the time step interval of 40 milliseconds to obtain the feature change rate. If the feature evolution direction is enhancement, the feature change rate is mapped to a positive relative velocity scalar; if the feature evolution direction is decay, the feature change rate is mapped to a negative relative velocity scalar. For the zero time step, the baseline propagation velocity scalar is set to 1.0 mm / s. For time steps greater than zero, the relative velocity scalars of all time steps before the current time step are accumulated and divided by the number of time steps to obtain the baseline propagation velocity scalar. Calculate the reference propagation speed scalar plus a zero-prevention offset of 0.001 mm / s, extract the maximum value of the sum of the reference propagation speed scalar minus the absolute value of the relative speed scalar plus the zero-prevention offset, and use it as the safety denominator. Calculate the ratio of the reference propagation speed scalar plus the zero-prevention offset divided by the safety denominator, and output the frequency shift scaling factor. Along the time dimension, the absolute value of the relative velocity scalar is divided by the reference propagation velocity scalar to obtain the dimensionless rate ratio. The minimum value between the dimensionless rate ratio and the value 10 is extracted as the cutoff rate ratio. The cutoff rate ratio is multiplied by the sign of the relative velocity scalar to obtain the directed cutoff ratio. The negative of the directed cutoff ratio of the natural constant 2.71828 is multiplied by the power of 0.1 and the value 1 is added to obtain the dynamic attenuation coefficient. The time-series cumulative scaling factor at time zero is set to be equal to the frequency shift scaling factor at the corresponding time. For time steps greater than zero, the dynamic attenuation coefficient is calculated by multiplying the frequency shift scaling factor of the current time step by the value 1 minus the dynamic attenuation coefficient, multiplying the difference by the time-series cumulative scaling factor of the previous time step, and outputting the time-series cumulative scaling factor of the current time step. Along the spatial dimension, calculate the cosine similarity of the rate of change of features between the current spatial position and the neighboring spatial positions, add a value of 1 as the propagation weight, multiply the propagation weight by the relative velocity scalar of the neighboring spatial positions and accumulate them, divide by the sum of the propagation weights to perform scale normalization, extract the larger of the absolute values ​​of the positive and negative values ​​of the normalization result, if the absolute value of the negative value is larger, inherit the negative sign, otherwise inherit the positive sign, combine the larger value to reconstruct the spatial correction velocity scalar with the direction determined and the scale aligned, and recalculate the spatial correction scaling factor based on the maximum value of the sum of the baseline propagation velocity scalar, the zero-prevention offset, the absolute value of the baseline propagation velocity scalar minus the spatial correction velocity scalar, and the zero-prevention offset. The target frequency shift scaling factor is obtained by multiplying the temporal cumulative scaling factor by 0.6 and adding the spatial correction scaling factor by 0.4. The target frequency shift scaling factor is then broadcast and expanded along the feature channels. Channel-by-channel weighted purification is performed on the spatiotemporal fusion feature sequence to output the Doppler modulation feature sequence. Subsequently, adaptive average pooling is performed along the spatial dimension to compress the target frequency shift scaling factor into a single scalar to obtain the frame-level frequency shift vector. The outer product of the frame-level frequency shift vector and its transpose is calculated to generate a frequency shift modulation matrix with a size equal to the product of time steps. The frequency shift modulation matrix is ​​then multiplied element-wise with the original temporal attention weight map to output the modulation temporal attention weight map. The saliency keyframe decoding output layer uses the modulation temporal attention weight map as a dynamic prior. It reconstructs and maps the spatial and channel dimensions of the Doppler modulation feature sequence to the column dimension, making the time dimension lie in the row dimension to form a feature matrix. The modulation temporal attention weight map is left-multiplied by this feature matrix to perform matrix multiplication. The token features are weighted, aggregated, and fused along the time dimension to restore the original shape. A 1x1 convolution is performed to reconstruct and focus the spatial features. The hyperbolic tangent function is used to output a semantic-level visual saliency map in the range of -1 to +1. Depth smoothing constraints are applied in combination with the modulation temporal attention weight map to output depth map data. Then, the absolute value of the difference between the attention response of adjacent time steps is calculated along the time dimension based on the modulation temporal attention weight map. The absolute value of the difference between the depth values ​​of adjacent time steps is calculated in combination with the depth map data along the time dimension as the depth discontinuity jump. The dynamic response mutation threshold is set to 0.5 mm and the depth jump threshold is set to 0.2 mm. When the absolute value of the attention response difference is greater than 0.5 and the depth discontinuity jump is greater than 0.2 mm, the jump moment is located. The video frame corresponding to the jump moment is extracted to output the video keyframe sequence.

[0020] In the specific implementation process, the core of the invention lies in the deep transformation of the spatiotemporal feature extraction mechanism of the traditional TimeSformer model inspired by acoustic physics. The traditional TimeSformer adopts an equal allocation of time attention, which cannot perceive the dynamic evolution trend of features in the spatiotemporal domain, resulting in a lag in feature capture of high-speed motion or sudden change scenes and easy interference from static background. This invention inherits the spatiotemporal separation attention computation paradigm of the traditional architecture to ensure the extraction of basic multi-dimensional related features, but innovatively introduces a spatiotemporal Doppler frequency shift layer for core modification. It maps the frequency shift phenomenon caused by relative motion in acoustics to the visual feature domain. By quantifying the feature change rate of tokens at the same spatial position in adjacent time steps, it derives the signed relative velocity scalar and uses it to calculate the frequency shift scaling factor that reflects the dynamic evolution rate of features. Then, along the time dimension, it obtains the temporal cumulative scaling factor through exponential moving average to perceive long-term dynamic trends. Along the spatial dimension, it obtains the spatial correction scaling factor by aggregating neighborhood information through feature change rate similarity to enhance local motion consistency. Finally, it fuses and generates the target frequency shift scaling factor to modulate the original temporal attention weight map element by element. This modification effectively overcomes the shortcomings of the static and even distribution of spatiotemporal attention in traditional models, enabling the model to adaptively focus on motion regions with significant dynamic evolution characteristics. It not only accurately locates and outputs video keyframe sequences based on temporal abrupt changes in modulation weights, avoiding the processing of redundant frames, but also outputs high-fidelity semantic-level visual saliency maps and depth maps through dynamic geometric constraints and feature reconstruction. This significantly improves the agility and accuracy of spatiotemporal perception in live streams, providing high-quality and dynamic prior-rich input data for subsequent 3D scene reconstruction and blind zone mapping.

[0021] In this embodiment, S2 specifically includes: S21. Read the video keyframe sequence and the corresponding depth map data, and calculate the depth based on the camera intrinsic parameter matrix. Figure 2 The pixel and depth values ​​are back-projected into a 3D space, and the red, green and blue color values ​​of the corresponding pixels are fused to output a 3D spatial point cloud carrying the colors. The sum of the absolute values ​​of the color differences between each point in the point cloud and its four nearest neighbors is calculated as the photometric matching cost, and the sum of the inverses of the spatial distances is calculated as the geometric matching cost. The joint matching cost is obtained by accumulating these values. Neighboring points with a joint matching cost less than 5.0 are interconnected and graph cut optimization fusion is performed to output 3D scene features.

[0022] S22. Extract the mean of the 3D coordinates of the 3D scene features as the spatial center point. Obtain the unit observation direction vector by subtracting the values ​​from the keyframe camera's optical center pointing to the spatial center point and normalizing the result. This vector serves as the third row of the orthogonal rotation matrix. Select the unit horizontal vector perpendicular to the third row as the first row. Perform cross product and normalization on the third and first rows to obtain the second row. Concatenate the orthogonal rotation matrix with the translation vector from the camera's optical center pointing to the spatial center point to output the initial projection matrix. Perform singular value decomposition on the distance matrix formed by the pairwise spatial distances of the 3D scene feature points, extracting the maximum... The left singular vector corresponding to the singular value is set as the principal axis of the spatial distribution and is set as the X-axis of the local coordinate system. The cross product of the principal axis of the spatial distribution and the height coordinate axis is set as the Y-axis of the local coordinate system. The cross product of the X-axis and the Y-axis of the local coordinate system is set as the Z-axis of the local coordinate system. Together with the spatial center point, an alignment transformation matrix is ​​constructed. The 3D scene features are multiplied by the inverse of the alignment transformation matrix. At the same time, the initial projection matrix is ​​right-multiplied by the alignment transformation matrix to map to the synchronous coordinate system, eliminating absolute scale ambiguity. The aligned 3D scene features and the aligned initial projection matrix are output.

[0023] S23. Expand the aligned 3D scene features by a homogeneous dimension of 1.0, multiply by the aligned initial projection matrix to obtain the two-dimensional projection coordinates, calculate the sum of the absolute values ​​of the horizontal and vertical differences between the two-dimensional projection coordinates and the original pixel coordinates of the keyframe, accumulate them and divide by the total number of points to obtain the average reprojection error; set the error threshold to 2.0 pixels. When the error is greater than 2.0 pixels, calculate the partial derivative of the aligned 3D scene features and the aligned initial projection matrix as the gradient value, and perform gradient descent updates with a spatial step size of 0.01 mm and a dimensionless step size of 0.001 mm per iteration, respectively. Iterate until the average reprojection error is less than or equal to 2.0 pixels, and output the optimized camera projection parameters and 3D scene features.

[0024] In this embodiment, S3 specifically includes: S31. Load the live streaming behavior big data pool and extract multi-source heterogeneous behavior data. Calculate the timestamp difference and spatial coordinate offset of each source behavior data. Truncate and align sequences with timestamp differences greater than 0.0 seconds. Subtract 0.0 pixels from the spatial coordinate offset to achieve translational alignment. Divide the aligned values ​​by the sum of their maximum absolute values ​​and the value 0.001 to achieve normalization mapping. Multiply by the value 0.5 and add the value 0.5 to offset to the 0.0 to 1.0 range. Apply the Sigmoid activation function to calculate the result of 1.0 divided by 1.0 plus the negative power of the natural constant e. Output the structured behavior feature sequence.

[0025] S32. The structured behavioral feature sequence is compressed into a single temporal response scalar by calculating the mean along the feature channel dimension. This response scalar is multiplied by the global spatial resolution value corresponding to the visual saliency map to map a two-dimensional behavioral response matrix to achieve spatial dimension expansion. The two-dimensional behavioral response matrix is ​​multiplied pixel by pixel by the visual saliency map to obtain the pixel-level attention response intensity of the visual saliency map. The attention response intensity is arranged according to the pixel coordinates to generate a spatial attention distribution. The mean value of the spatial attention distribution is calculated and 1.5 times the standard deviation value is added as the high response screening threshold. Pixels in the spatial attention distribution with values ​​greater than the high response screening threshold are marked as 1.0, and pixels with values ​​less than or equal to the high response screening threshold are marked as 0.0 to generate a binary mask map. Pixels with values ​​of 1.0 in the binary mask map are extracted and eight-neighbor connected component detection is performed. Discrete connected components with an area of ​​less than 100.0 pixels are removed, and the visual attention region is output.

[0026] S33. Perform reverse mask filtering on the visual saliency map based on the visual attention region. Replace the value 1.0 in the binary mask image corresponding to the visual attention region with the value 0.0 and the value 0.0 with the value 1.0 to generate a reverse mask. Multiply the reverse mask by the visual saliency map to extract low response regions with values ​​greater than 0.0. Calculate the mean and variance of the horizontal and vertical pixel coordinates of all pixels with values ​​other than 0.0 in the low response region. Remove outliers whose horizontal and vertical coordinates deviate from the mean by more than 3.0 times the variance. Extract the horizontal and vertical coordinate values ​​of the remaining low response pixels and combine them into a two-dimensional coordinate pair to output the 2D visual blind zone.

[0027] In this embodiment, S4 specifically includes: S41. Based on the 2D visual blind zone and camera projection parameters, perform back projection ray calculation to solve the ray clusters of the two-dimensional pixels corresponding to the 2D visual blind zone in the three-dimensional space. Calculate the Euclidean distance between each ray in the ray cluster and the mean point of the three-dimensional coordinates of the 3D scene features. Determine that the ray is less than 50.0 mm away as intersecting. Extract the 3D scene feature points on the intersecting rays that are closest to the mean point and output the three-dimensional visual blind zone point cloud.

[0028] S42. Calculate the mean Euclidean distance between each point in the 3D visual blind zone point cloud and its 10 nearest neighbors as a local spatial distribution statistical feature. Perform density clustering to extract point cloud clusters for points with local spatial distribution statistical features less than 5.0 mm. Calculate the maximum and minimum 3D coordinate differences of all points in each point cloud cluster to construct the bounding boundary. Fit a plane to each point and its 8 nearest neighbors within the point cloud cluster to obtain the surface normal vector. Calculate the cosine of the angle between all surface normal vectors within the point cloud cluster and the cluster average normal vector. Subtract the square of the angle cosine from 1.0 to obtain the normal vector dispersion. Subtract the normal vector dispersion from 1.0 to obtain the spatial availability metric. Output the spatial availability metric.

[0029] S43. Calculate the 75.0 percentile of the spatial availability metric of all point cloud clusters as the high availability screening threshold. Retain point cloud clusters with spatial availability metrics greater than the high availability screening threshold as high availability 3D visual blind zone point cloud clusters. Calculate the arithmetic mean of the 3D coordinates of all points within the boundary of the high availability 3D visual blind zone point cloud clusters as the centroid coordinates. Extract the average normal vector of the high availability 3D visual blind zone point cloud clusters and multiply it by the vertical offset distance of 100.0 mm. Add the product result to the centroid coordinates to calculate the orientation anchor point. Combine the centroid coordinates and the orientation anchor point to generate the initial spatial position of the material to be recommended.

[0030] In this embodiment, S5 specifically includes: S51. Based on 3D scene features and initial spatial position, extract 3D scene feature points within a 50.0 mm radius neighborhood of the initial spatial position. Fit a local quadratic surface to the neighborhood points to calculate the principal curvature value as the curvature distribution. Calculate the depth difference between the line connecting the initial spatial position and the neighborhood points and divide it by the horizontal distance to obtain the occlusion depth gradient. Concatenate the principal curvature value and the occlusion depth gradient to output the spatial constraint features.

[0031] S52. Based on the principal curvature values ​​and occlusion depth gradients of spatial constraint features, calculate the principal curvature values ​​multiplied by the offset step size of 10.0 mm and accumulate them in the opposite direction of the surface normal vector to obtain the local offset vector. Divide the occlusion depth gradient by the gradient scaling factor of 0.1 to convert it into a rotation angle. Calculate the rotation angle divided by the cosine of 2.0 as the real part and the rotation angle divided by the sine of 2.0 as the imaginary part to construct a rotation correction quaternion. Add the local offset vector to the centroid coordinates of the initial spatial position to obtain the correction translation amount. Multiply the rotation correction quaternion by the original quaternion of the initial spatial position to obtain the correction rotation amount. Concatenate the correction translation amount and the correction rotation amount to perform spatial geometric deformation compensation and output the fine-tuned spatial pose.

[0032] S53. Combine the camera projection parameters to map the centroid coordinates and orientation anchor points of the fine-tuned spatial pose to the image plane to extract the projection contour. Calculate the minimum distance from each vertex on the projection contour to the four boundary lines of the image to form a distance set. Calculate the mean of the distance set minus 1.5 times the standard deviation as the boundary safety distance. Divide the boundary safety distance by the distance from the center of the projection contour to the nearest boundary line of the image to obtain the scaling factor. Multiply the difference between the horizontal and vertical coordinates of the center of the projection contour and the horizontal and vertical coordinates of the center of the image by the translation compensation coefficient 0.2 to obtain the translation compensation amount. Add the translation compensation amount to the corrected translation amount of the fine-tuned spatial pose. Extract the rotation angle scalar corresponding to the corrected rotation amount, multiply it by the scaling factor, and keep the rotation axis unchanged to convert it back to a quaternion. Replace the corresponding parameters in the fine-tuned spatial pose for secondary correction and output the 3D overlay parameters.

[0033] In this embodiment, S6 specifically includes: S61. Extract the 2D pixel brightness gradient and 3D scene feature data of the video frame to be processed. Extract the horizontal and vertical components of the 2D pixel brightness gradient. Concatenate the horizontal and vertical components with the value 1.0 in sequence to form the 3D initial gradient vector. Calculate the Euclidean norm of the 3D initial gradient vector and add the value 1.0 as the gradient zero-norm. Divide the 3D initial gradient vector by the gradient zero-norm to perform normalization and obtain the 3D gradient vector. Based on the camera intrinsic parameter matrix and depth value in the 3D scene feature data, back-project the 3D gradient vector into 3D space to obtain the spatial gradient direction vector. Calculate the dot product of the spatial gradient direction vector and the corresponding surface normal vector. Extract the result when the dot product is negative. The corresponding spatial gradient direction vector is used as the incident light direction vector in the opposite direction. The brightness value of the corresponding pixel is obtained by tracing along the positive direction of the surface normal vector in the 3D scene feature data. The brightness value is calculated by dividing it by the maximum brightness value of the whole frame and adding 0.01 to obtain the normalized observed light flux. Based on the diffuse reflection assumption, the cosine of the angle between the incident light direction vector and the corresponding surface normal vector is calculated and added to 0.01 as the incident light attenuation ratio. The normalized observed light flux is divided by the incident light attenuation ratio to obtain the initial reflectivity value. The minimum value between the initial reflectivity value and the value 1.0 is extracted and truncated to obtain the reflectivity value. The incident light direction vector and reflectivity value of all spatial points are aggregated to output the ambient lighting information. Based on 3D scene features and corresponding ambient lighting information, 3D scene feature points within a 100.0 mm neighborhood corresponding to the spatial location of the 3D overlay parameters are extracted. The covariance matrix of the incident light direction vector of the neighborhood points is calculated, and the eigenvector corresponding to the largest eigenvalue is solved as the statistical principal axis of the incident light direction. The arithmetic mean of the reflectance values ​​of the neighborhood points is calculated as the ambient reflectance distribution. The incident light direction statistical principal axis and the ambient reflectance distribution are concatenated to output the local lighting parameters.

[0034] S62. Perform physical interaction calculations on the incident light direction statistical principal axis and ambient reflectivity distribution in the local lighting parameters with the material properties of the material to be recommended. Calculate the ambient reflectivity distribution divided by the value 3.14159 and add the value 0.05 as the incident light attenuation coefficient. Calculate the dot product of the surface normal vector of the material to be recommended and the opposite direction of the incident light direction statistical principal axis, multiply the result by the specular sharpness coefficient 50.0 to obtain the specular reflection distribution. Multiply the original texture color value of the material to be recommended by the incident light attenuation coefficient to perform photometric mapping and add the specular reflection distribution to perform texture recoloring. Apply the ReLU activation function to calculate the maximum value of the input value and the value 0.0, truncate the negative value, and calculate the minimum value of the truncation result and the value 1.0 to limit color overflow. Output the lighting-adapted material.

[0035] S63. Based on the incident light direction statistical principal axis in the local illumination parameters and the orientation anchor point in the 3D overlay parameters, calculate the projection intersection point of the orientation anchor point on the 3D scene features in the opposite direction of the incident light direction statistical principal axis. Extract the three-dimensional vertices of the edge of the material to be recommended and project them onto the two-dimensional plane where the projection intersection point is located along the incident light direction statistical principal axis to obtain the two-dimensional projection contour. Extend the two-dimensional projection contour outward by 100.0 mm pixel distance to obtain the occluded projection area. Calculate the negative number of the distance from each pixel in the occluded projection area to the projection intersection point and multiply it by the shadow attenuation coefficient 0.01 to obtain the shadow gradient. Perform shadow gradient rendering on the occluded projection area and overlay the rendering result onto the corresponding pixel position of the lighting adaptation material to output the 3D rendered material.

[0036] In this embodiment, S7 specifically includes: S71. Based on the 3D overlay parameters, map the 3D rendered material to the corresponding pixel coordinates of the original video frame, compare the pixel depth value of the 3D rendered material with the corresponding depth value of the 3D scene features, remove the rendered pixels with depth values ​​greater than the scene depth and perform a depth occlusion test, and perform edge feathering and fusion by multiplying the alpha value of the rendered pixel edge within 5.0 pixels of the edge of the rendered pixel that passed the depth occlusion test by the ratio of the distance from the pixel to the edge divided by 5.0 pixels, and generate a spatially fused image frame.

[0037] S72. Calculate the histogram of the luminance components of the spatially fused image frames and solve for the variance as the statistical distribution feature of image quality. Set the basic quantization parameter value to 30.0. Calculate the quotient of the statistical distribution feature of image quality divided by the value 50.0 and apply the Sigmoid activation function to calculate the negative quotient of 1.0 divided by 1.0 plus the natural constant e. Multiply the calculation result by the value 10.0 and add the basic quantization parameter value 30.0 to obtain the adaptive quantization parameter. Set the basic frame-level bit count value to 50000.0 bits. Calculate the basic frame-level bit count value multiplied by the value 30.0 and divided by the adaptive quantization parameter to obtain the target bit rate control coefficient. Concatenate the adaptive quantization parameter and the target bit rate control coefficient to output the video encoding control parameter.

[0038] S73. Read the current network transmission bandwidth value and calculate the bandwidth matching ratio by dividing the current network transmission bandwidth value by the target bitrate control coefficient. Set the network congestion threshold value to 0.8. When the bandwidth matching ratio is less than the network congestion threshold value of 0.8, multiply the adaptive quantization parameter in the video coding control parameters by 1.2 and update it as the congestion quantization parameter. When the bandwidth matching ratio is greater than or equal to the network congestion threshold value of 0.8, retain the original value of the video coding control parameters as the congestion quantization parameter. Use the congestion quantization parameter and the target bitrate control coefficient to perform single-pass encoding compression on the spatial fusion image frame to generate a video bitstream. Encapsulate the video bitstream into a real-time transmission protocol data packet and send it down through the transmission channel to output a live stream suitable for terminal decoding and rendering.

[0039] In this embodiment, S8 specifically includes: S81: Receive the live stream and strip the real-time transmission protocol packet header to perform protocol decryption and decapsulation operations. Extract the video frame to be transmitted and the encoded metadata. Input the video frame to be transmitted into the hardware decoder to perform decoding operations. Multiply the decoded YUV format pixel data by the color conversion matrix and divide by the value 255.0 to perform normalization. Apply the ReLU activation function to calculate the conversion value and truncate the negative value by the maximum value of 0.0. Output the decoded image frame in RGB format.

[0040] S82. Input the decoded image frame into the graphics rendering pipeline, extract the display timestamp of the decoded image frame, calculate the standard deviation of the difference between the display timestamps of two adjacent frames as the display timestamp statistical distribution feature, set the baseline vertical synchronization beat value to 16.67 milliseconds, calculate the product of the baseline vertical synchronization beat value and the display timestamp statistical distribution feature multiplied by 0.5 to obtain the adaptive vertical synchronization beat, align the rendering timing of the decoded image frame to the adaptive vertical synchronization beat to execute the screen drawing, and output the interactive display interface.

[0041] S83. Capture the touch event stream on the interactive display interface, extract the screen horizontal and vertical coordinates of each touch point in the touch event stream to form a spatial coordinate distribution, calculate the mean of the timestamp difference between adjacent touch events as a time interval statistical feature, and concatenate the mean of the horizontal and vertical coordinates of the spatial coordinate distribution with the time interval statistical feature in sequence to form an interactive behavior vector with a dimension of 3. Encapsulate the interactive behavior vector into a transmission control protocol data packet and send it back to the server to update the live broadcast behavior big data pool.

[0042] refer to Figure 3 A live streaming content recommendation system based on big data analysis, specifically including the following modules: The spatiotemporal feature extraction and modulation perception module is used to collect live video data. It performs spatiotemporal feature extraction by improving the TimeSformer model to obtain the original temporal attention weights. It introduces a spatiotemporal Doppler frequency shift modulation mechanism, extracts the relative velocity scalar based on the feature change rate mapping and derives the frequency shift scaling factor. After spatiotemporal joint calibration, it modulates the original temporal attention weights and outputs a visual saliency map, depth map and video key frame sequence through significant spatiotemporal perception. The 3D scene reconstruction and parameter estimation module is used to perform 3D scene analysis on video keyframe sequences and corresponding depth map data, reconstruct the 3D spatial information of the live scene, output 3D scene features, and estimate the corresponding camera projection parameters. The behavioral data fusion and blind spot determination module is used to load the live broadcast behavior big data pool and extract multi-source heterogeneous behavioral data. It combines 3D scene features and visual saliency maps to calculate the visual attention area and user attention distribution information, and determines the 2D visual blind spot by excluding the visual attention area. The blind spot spatial mapping and position calculation module is used to map the 2D visual blind spot to a three-dimensional space and perform spatial availability assessment based on 3D scene features, 2D visual blind spots and camera projection parameters, so as to calculate the initial spatial position of the material to be recommended. The spatial position fine-tuning and parameter determination module is used to fine-tune and optimize the initial spatial position by combining 3D scene features, and determine the 3D overlay parameters of the material to be recommended in three-dimensional space to conform to the composition rules. The lighting rendering and shadow blending module is used to perform lighting rendering on the recommended material based on the 3D scene characteristics and corresponding ambient lighting information, combined with 3D overlay parameters, to blend shadows and generate 3D rendered material. The spatial fusion and live stream output module is used to fuse 3D rendered materials into a three-dimensional space based on 3D overlay parameters to generate live recommendation frames, and transmit them to the client via the network to output the live stream; The client-side interaction and data feedback module is used to parse the live stream, perform real-time rendering and interactive display, respond to interactive operations to generate real-time feedback data, and send it back to the server to update the live stream behavior big data pool.

[0043] Example 1: To verify the feasibility of this invention in immersive live streaming and real-time interactive recommendations, the method of this invention was applied to the intelligent 3D virtual material delivery system of a leading live streaming platform (hereinafter referred to as "Platform L"). In traditional live streaming material recommendation systems, saliency detection based on 2D image planes and preset rule mapping methods are typically used to overlay and recommend virtual goods or interactive props. These methods not only fail to effectively handle the complex three-dimensional spatial geometry and dynamic lighting changes in live streaming scenes, but also easily ignore the dynamic semantic relationships of the viewer's gaze over time, easily leading to severe perspective distortion, depth penetration, and light and shadow fragmentation between virtual materials and real scenes, resulting in a stiff recommendation effect and a poor user experience. To solve the above problems, Platform L decided to adopt the live streaming material recommendation method and system based on big data analysis proposed in this invention.

[0044] During implementation, Platform L first utilizes its data acquisition system to acquire real-time live stream video data and viewer interaction logs, including events such as sending bullet comments, liking, gift donations, and gaze placement. After data cleaning and multi-source heterogeneous fusion processing, a high-quality live stream behavior big data pool is formed. Simultaneously, Platform L improves the TimeSformer model to extract spatiotemporal features from the live stream and introduces a spatiotemporal Doppler frequency shift modulation mechanism. Based on the feature rate of change mapping, it extracts a relative velocity scalar and derives the frequency shift scaling factor. It then performs spatiotemporal joint calibration and modulation on the original temporal attention weights, accurately outputting visual saliency maps, depth maps, and video keyframe sequences.

[0045] Platform L performs 3D scene analysis on video keyframe sequences and depth map data to reconstruct the 3D spatial information of the live streaming scene and estimate camera projection parameters. It calculates the visual attention area by combining 3D scene features and visual saliency maps, and uses a reverse mask to filter out high-response areas, accurately identifying 2D visual blind spots. Subsequently, based on 3D scene features and camera projection parameters, the 2D visual blind spots are mapped to 3D space, and spatial availability is assessed. The initial spatial position of the recommended material is calculated, and 3D overlay parameters are fine-tuned and optimized using composition rules. Finally, based on ambient lighting information and 3D overlay parameters, the recommended material undergoes lighting rendering and shadow blending to generate 3D rendered material. This material is then blended into 3D space according to the 3D overlay parameters to generate live streaming recommendation frames, which are then sent to the client. The client responds to interactive operations, generating real-time feedback data, which is then sent back to the server to update the live streaming behavior big data pool, forming a closed-loop optimization.

[0046] During implementation, the technical team at Platform L discovered that, compared to traditional 2D planar mapping and static rule-based recommendation methods, the method of this invention significantly improves the visual realism of virtual material recommendations and user engagement. Traditional methods cannot accurately depict the 3D spatial usability and dynamic lighting interactions of live streaming scenes, while the method of this invention effectively achieves physical-level deep fusion of virtual materials and real scenes in terms of perspective, occlusion, and lighting through spatiotemporal Doppler frequency shift modulation and blind zone 3D spatial mapping.

[0047] To further verify the actual performance of the method of the present invention, platform L conducted a detailed comparative test between the method of the present invention and the traditional method. The specific performance data is shown in Table 1: Table 1 Performance Comparison of Platform L Live Stream Material Recommendation Methods As shown in Table 1, the performance of the live streaming content recommendation system was comprehensively improved after applying the method of this invention. The click-through rate of content recommendations increased from 3.2% in the traditional method to 6.1%, and the conversion rate increased from 0.85% to 1.64%, significantly improving the commercial conversion value of recommendations and directly increasing the revenue of the platform and streamers. Revenue per thousand impressions (RPM) and return on investment (ROI) of virtual content also achieved significant increases of nearly 90%. The single-frame 3D fusion rendering time decreased from 150 milliseconds in the traditional method to 38 milliseconds, and the concurrent live streaming processing capacity increased from 2,000 streams to 8,500 streams, resulting in a significant leap in recommendation rendering efficiency and system concurrent response capability. In addition, the complaint rate for content occlusion and inconsistency dropped sharply from 2.8% to 0.3%, and the visual realism score of virtual-real fusion was greatly improved, effectively enhancing the immersive experience for viewers. Viewer satisfaction with recommendations also improved significantly, from 68.5% to 89.2%.

[0048] Through the method of this invention, platform L successfully achieved three-dimensional, precise, and personalized recommendations for live streaming virtual materials, effectively enhancing the visual immersion and audience engagement of the material delivery, ensuring the commercial benefits for both the platform and the broadcasters, significantly improving the intelligence and automation level of live streaming recommendations, significantly reducing the display costs of invalid and incongruous materials, enhancing the system's real-time spatial computing capabilities and robustness, and providing strong technical support for the live streaming metaverse and intelligent interactive operation.

[0049] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A live streaming content recommendation method based on big data analysis, characterized in that, Includes the following steps: S1. Collect live video data, perform spatiotemporal feature extraction by improving the TimeSformer model, and obtain the original time attention weights; introduce a spatiotemporal Doppler frequency shift modulation mechanism, extract the relative velocity scalar based on the feature change rate mapping and derive the frequency shift scaling factor, modulate the original time attention weights after spatiotemporal joint calibration, and output visual saliency map, depth map and video key frame sequence after significant spatiotemporal perception. S2. Perform 3D scene analysis on the video keyframe sequence and corresponding depth map data, reconstruct the 3D spatial information of the live scene, output 3D scene features, and estimate the corresponding camera projection parameters. S3. Load the live streaming behavior big data pool and extract multi-source heterogeneous behavior data. Combine 3D scene features and visual saliency map to calculate the visual attention area and user attention distribution information. Then, determine the 2D visual blind spot by excluding the visual attention area. S4. Based on 3D scene features, 2D visual blind spots and camera projection parameters, map the 2D visual blind spots to three-dimensional space and perform spatial availability assessment to calculate the initial spatial position of the material to be recommended. S5. Fine-tune and optimize the initial spatial position by combining 3D scene features to determine the 3D overlay parameters of the recommended material that conform to the composition rules in three-dimensional space. S6. Based on 3D scene features and video key frame sequences, ambient lighting information is generated, and combined with 3D overlay parameters, the recommended material is rendered with lighting, shadows are blended, and 3D rendered material is generated. S7. Based on the 3D overlay parameters, the 3D rendered materials are blended into the three-dimensional space to generate live recommendation frames, which are then transmitted to the client via the network to output the live stream; S8: The client receives and parses the live stream, performs real-time rendering and interactive display, responds to interactive operations to generate real-time feedback data, and sends it back to the server to update the live stream behavior big data pool.

2. The live streaming content recommendation method based on big data analysis according to claim 1, characterized in that, The improved TimeSformer model includes a spatiotemporal embedding coding layer, a separate attention computation layer, a spatiotemporal Doppler frequency shift layer, and a saliency keyframe decoding output layer: The spatiotemporal embedding coding layer is used to perform temporal sampling on the input live stream video data to generate a video frame sequence, and to perform spatial segmentation and feature mapping on the video frames, injecting spatiotemporal location information to generate a video token sequence containing spatiotemporal information. The separate attention calculation layer is used to receive the video token sequence, calculate the correlation between each token along the spatial dimension to extract the global context features of the screen, and then calculate the correlation between each token along the temporal dimension to extract the dynamic change features between frames. After feature fusion, it outputs the spatiotemporal fusion feature sequence and the corresponding original temporal attention weight map. The spatiotemporal Doppler frequency shift layer is used to introduce an acoustic spatiotemporal Doppler frequency shift modulation mechanism, and the specific execution process includes: The feature change rate of the token at the same spatial location in adjacent time steps is calculated based on the spatiotemporal fusion feature sequence. The feature change rate is mapped to a signed relative velocity scalar according to the feature evolution direction, and the baseline propagation velocity scalar is accumulated along the temporal dimension. Calculate the algebraic difference between the reference propagation speed scalar and the relative speed scalar, obtain the ratio between the reference propagation speed scalar and the algebraic difference, and output the frequency shift scaling factor; Along the time dimension, a dynamic attenuation coefficient is generated based on the relative velocity scalar of the current time step. An exponential moving average is performed on the frequency shift scaling factor between the previous time step and the current time step, and the time-series cumulative scaling factor is output. Along the spatial dimension, propagation weights are constructed based on spatial similarity of feature change rates. Spatial neighborhood feature aggregation is performed to output a spatial correction velocity scalar. The spatial correction scaling factor is then recalculated based on the spatial correction velocity scalar. The target frequency shift scaling factor is obtained by fusing the temporal cumulative scaling factor and the spatial correction scaling factor. The target frequency shift scaling factor is then multiplied element-wise with the original temporal attention weight map to output the modulation temporal attention weight map. The saliency keyframe decoding output layer is used to reconstruct and focus spatial features on the spatiotemporal fusion feature sequence using the modulation temporal attention weight map as a dynamic prior to generate a semantic-level visual saliency map, and apply dynamic geometric constraints to output depth map data; then, based on the dynamic response abrupt change of the modulation temporal attention weight map along the time dimension, it locates the jump moment to output the video keyframe sequence.

3. The live streaming content recommendation method based on big data analysis according to claim 1, characterized in that, S2 specifically includes: S21. Based on the video keyframe sequence and the corresponding depth map data, perform multi-view feature matching and feature fusion to reconstruct the three-dimensional spatial information of the live scene and output 3D scene features. S22. Solve the initial projection matrix based on the initial spatial correspondence between 3D scene features and video keyframe sequences, and extract the spatial distribution principal axis based on the global geometric topology distribution statistical features of 3D scene features to construct a scene local coordinate system to eliminate absolute scale ambiguity. S23. Based on the initial projection matrix, backproject the 3D scene features to the video keyframe sequence to calculate the reprojection error distribution. Based on the reprojection error distribution, perform gradient backpropagation update on the initial projection matrix and 3D scene features, and output the optimized camera projection parameters and 3D scene features.

4. The live streaming content recommendation method based on big data analysis according to claim 1, characterized in that, S3 specifically includes: S31. Load the live streaming behavior big data pool and extract multi-source heterogeneous behavior data. Perform spatiotemporal alignment and normalization mapping on the multi-source heterogeneous behavior data. Based on the timestamp, backproject the structured behavior feature sequence to the 2D image space to generate a 2D behavior heatmap with the same resolution as the visual saliency map. S32. Jointly correlate the 2D behavior heatmap and the visual saliency map in terms of spatial dimensions, calculate the pixel-level attention response intensity and generate the spatial attention distribution, filter the high-response connected components based on the numerical statistical features of the spatial attention distribution, and output the visual attention region. S33. Perform reverse masking on the visual saliency map based on the visual attention region, extract the spatial coordinates of the low response region, and output the 2D visual blind spot.

5. The live streaming content recommendation method based on big data analysis according to claim 1, characterized in that, S4 specifically includes: S41. Based on the 2D visual blind zone and camera projection parameters, perform back projection ray calculation to solve the ray clusters of the two-dimensional pixels corresponding to the 2D visual blind zone in the three-dimensional space, perform intersection operation between the ray clusters and the 3D scene features, and output the three-dimensional visual blind zone point cloud. S42. Calculate the local spatial distribution statistical features of the point cloud in the 3D visual blind zone, extract the bounding boundary and surface normal vector distribution of the point cloud cluster based on the local spatial distribution statistical features, evaluate the spatial availability based on the dispersion of the surface normal vector distribution, and output the spatial availability metric. S43. Based on the distribution statistical quantiles of spatial availability metrics, select high-availability 3D visual blind spot point cloud clusters, and combine the enclosing boundary and surface normal vector distribution of the high-availability 3D visual blind spot point cloud clusters to calculate the centroid coordinates and orientation anchor points, generating the initial spatial position of the material to be recommended.

6. The live streaming content recommendation method based on big data analysis according to claim 1, characterized in that, S5 specifically includes: S51. Based on 3D scene features and initial spatial position, calculate the local neighborhood geometric features of the initial spatial position in three-dimensional space, extract the curvature distribution and occlusion depth gradient of the neighborhood scene surface, and output the spatial constraint features. S52. Based on the curvature distribution and occlusion depth gradient of the spatial constraint features, calculate the local offset vector and rotation correction quaternion of the initial spatial position, perform spatial geometric deformation compensation on the initial spatial position, and output the fine-tuned spatial pose. S53. Combine the camera projection parameters to map the fine-tuned spatial pose onto the image plane to extract the projection contour. Construct a composition grid reference line based on the image boundary and the visual interest area. Calculate the offset of the projection contour relative to the composition grid reference line. Generate a scaling factor and translation compensation based on the offset. Perform secondary correction on the fine-tuned spatial pose and output 3D overlay parameters.

7. The live streaming content recommendation method based on big data analysis according to claim 1, characterized in that, S6 specifically includes: S61. Based on 2D pixel brightness gradient and 3D scene feature data, the 2D gradient is mapped to 3D space by combining camera imaging geometry, and the incident light direction vector is estimated by using the geometric constraints of 3D space gradient and surface normal vector. S62. Based on the surface normal vector sampling of 3D scene feature data, the observed light flux is obtained, and under the assumption of diffuse reflection, the reflectivity value is obtained by inversion based on the attenuation relationship between the observed light flux and the incident light direction, and the ambient lighting information is generated by aggregation. S63. Based on 3D scene features and ambient lighting information, extract the target point set in the spatial neighborhood corresponding to the 3D overlay parameters, extract the main direction of the incident light direction vector of the target point set to obtain the statistical main axis of the incident light direction, perform statistical averaging on the reflectivity values ​​to obtain the ambient reflectivity distribution, and stitch together to output the local lighting parameters; S64. Perform physical interaction calculations between the local lighting parameters and the material properties of the material to be recommended, solve the incident light attenuation coefficient and specular reflection distribution on the surface of the material to be recommended, perform photometric mapping and texture recoloring on the material to be recommended, and output lighting-adapted material. S65. Based on the incident light direction statistical principal axis in the local lighting parameters and the orientation anchor point in the 3D overlay parameters, calculate the occlusion projection area of ​​the material to be recommended in three-dimensional space, perform shadow gradient rendering on the occlusion projection area and overlay it onto the lighting adaptation material, and output the 3D rendered material.

8. The live streaming content recommendation method based on big data analysis according to claim 1, characterized in that, Specifically, S7 includes: S71. Based on the 3D overlay parameters, the 3D rendered material is mapped to the corresponding pixel coordinates of the original video frame. According to the 3D scene features and depth map data, the 3D rendered material and the original video frame are subjected to joint depth occlusion test and edge feathering fusion to generate a spatial fusion image frame. S72. Perform image quality statistical distribution evaluation on the spatially fused image frame, adaptively calculate the coding quantization parameters and bit rate control coefficients based on the image quality statistical distribution characteristics, and output the video frame to be transmitted. S73. Perform network-state-aware bitrate allocation and protocol encapsulation on the video frames to be transmitted, and send them down through the transmission channel and output a live stream suitable for terminal decoding and rendering.

9. The live streaming content recommendation method based on big data analysis according to claim 1, characterized in that, S8 specifically includes: S81: Receive the live stream and perform protocol decryption and decapsulation operations, extract the video frame to be transmitted and the encoded metadata, perform hardware decoding and pixel format conversion on the video frame to be transmitted, and output the decoded image frame; S82. Input the decoded image frame into the graphics rendering pipeline, extract the display timestamp statistical distribution characteristics of the decoded image frame, adaptively adjust the vertical synchronization beat and frame rendering timing based on the display timestamp statistical distribution characteristics, and execute screen drawing to output the interactive display interface. S83. Capture the touch event stream on the interactive display interface, analyze the spatial coordinate distribution and time interval statistical characteristics of the touch event stream, structure the spatial coordinate distribution and time interval statistical characteristics into interactive behavior vectors, and send the interactive behavior vectors back to the server to update the live broadcast behavior big data pool.

10. A live streaming content recommendation system based on big data analysis, comprising the live streaming content recommendation method based on big data analysis as described in any one of claims 1 to 8, characterized in that, Includes the following modules: The spatiotemporal feature extraction and modulation perception module is used to collect live video data. It performs spatiotemporal feature extraction by improving the TimeSformer model to obtain the original temporal attention weights. It introduces a spatiotemporal Doppler frequency shift modulation mechanism, extracts the relative velocity scalar based on the feature change rate mapping and derives the frequency shift scaling factor. After spatiotemporal joint calibration, it modulates the original temporal attention weights and outputs a visual saliency map, depth map and video key frame sequence through significant spatiotemporal perception. The 3D scene reconstruction and parameter estimation module is used to perform 3D scene analysis on video keyframe sequences and corresponding depth map data, reconstruct the 3D spatial information of the live scene, output 3D scene features, and estimate the corresponding camera projection parameters. The behavioral data fusion and blind spot determination module is used to load the live broadcast behavior big data pool and extract multi-source heterogeneous behavioral data. It combines 3D scene features and visual saliency maps to calculate the visual attention area and user attention distribution information, and determines the 2D visual blind spot by excluding the visual attention area. The blind spot spatial mapping and position calculation module is used to map the 2D visual blind spot to a three-dimensional space and perform spatial availability assessment based on 3D scene features, 2D visual blind spots and camera projection parameters, so as to calculate the initial spatial position of the material to be recommended. The spatial position fine-tuning and parameter determination module is used to fine-tune and optimize the initial spatial position by combining 3D scene features, and determine the 3D overlay parameters of the material to be recommended in three-dimensional space to conform to the composition rules. The lighting rendering and shadow blending module is used to perform lighting rendering on the recommended material based on the 3D scene characteristics and corresponding ambient lighting information, combined with 3D overlay parameters, to blend shadows and generate 3D rendered material. The spatial fusion and live stream output module is used to fuse 3D rendered materials into a three-dimensional space based on 3D overlay parameters to generate live recommendation frames, and transmit them to the client via the network to output the live stream; The client-side interaction and data feedback module is used to parse the live stream, perform real-time rendering and interactive display, respond to interactive operations to generate real-time feedback data, and send it back to the server to update the live stream behavior big data pool.

Images