Entertainment information transmission system and method fusing multi-terminal real-time interactive pushing

Abstract

The invention relates to the technical field of information transmission, in particular to an entertainment information transmission system and method fusing multi-terminal real-time interactive pushing, and the system comprises a time sequence collection module, an audio and video restoration module, a task generation module, a pushing adjustment module and a synchronous transmission module. According to the method, the periodic rendering parameters of each terminal are dynamically extracted and the uncertainty measurement is constructed, so that the content pushing rhythm can be adjusted according to the transmission state difference of the terminal in different time periods, and the time sequence completion and the synchronous distribution of the fault picture are realized in combination with the frame-level image restoration and content mapping rule; according to the pushing time efficiency and the terminal response dynamic state, the corresponding relation between the content and the time window is established, the node jump change and the time consumption trend in the transmission path are further subjected to correlation analysis, so that the path fluctuation area is accurately judged, and the rhythm fine tuning of the terminal task is completed in cooperation with the offset calculation in the period.

Description

Technical Field

[0001] This invention relates to the field of information transmission technology, and in particular to an entertainment information transmission system and method that integrates real-time interactive push from multiple terminals. Background Technology

[0002] The field of information transmission technology refers to a collection of technologies that utilize communication, computers, networks, and other means to achieve efficient and reliable transmission of information such as data, voice, and images between various media.

[0003] Among them, entertainment information transmission systems refer to system platforms designed for home and individual users, used to realize content presentation and user interaction in application scenarios such as audiovisual entertainment and interactive entertainment. These systems primarily cater to specific entertainment needs such as karaoke, video-on-demand, and music playback. They typically employ a dual-system architecture to handle audio and video playback and user interface interaction separately. Audio signals are processed and decoded using embedded chips, and sound and video output is achieved through terminal hardware.

[0004] Because existing entertainment information transmission technologies employ a dual-system architecture to handle audio / video playback and interface interaction separately, the terminal only passively receives and outputs the transmitted content locally. It lacks the ability to adjust in real time based on transmission link fluctuations and terminal rendering stability. When faced with network fluctuations, uneven terminal load, or content frame loss, it cannot adjust the push rhythm according to the actual response of the playback terminal. This leads to problems such as task window misalignment and screen content delay in cross-terminal scenarios. For example, in a karaoke scenario, if the synchronization error between the accompaniment and subtitle ends continues to accumulate, it will cause interactive chaos. Existing technologies also do not provide a frame tear repair mechanism. If a frame is missing during video playback, the screen tearing cannot be reconstructed, ultimately resulting in poor playback content continuity and unstable user experience. Summary of the Invention

[0005] To address the technical problems existing in the prior art, embodiments of the present invention provide an entertainment information transmission system and method integrating real-time interactive push from multiple terminals. The technical solution is as follows: On the one hand, it provides an entertainment information transmission system that integrates real-time interactive push from multiple terminals. This system includes: The timing acquisition module acquires multi-terminal rendering timing parameters within the current period of multiple terminals, analyzes the response change magnitude of each type of rendering timing parameter in the corresponding terminal, and outputs the terminal transmission status parameter group. The audio and video restoration module acquires the original audio and video content, identifies and restores the grayscale and texture structure parameters of pixel blocks in the frame-fault missing areas of the original audio and video content, and outputs the restored video clip. The task generation module identifies and sorts each repaired video segment that can be allocated to the terminal's transmission window in the current period based on the terminal transmission status parameter group, and generates an entertainment information content push sorting table. The push adjustment module schedules the instruction response time and task execution window of each terminal according to the entertainment information content push sorting table, and generates terminal push control results. The synchronous transmission module, based on the terminal push control results, analyzes the number of node jumps and path consumption time of each entertainment information content segment in the cross-terminal transmission path, and pushes out multi-terminal synchronized content to obtain the entertainment information transmission results.

[0006] As a further aspect of the present invention, the terminal transmission status parameter group includes response time difference value, rendering timing distribution characteristics, and waiting time statistics. The repaired video segment specifically includes structurally continuous segment, texture matching segment, and grayscale boundary fusion segment. The entertainment information content push sorting table specifically includes content segment timing position, terminal scheduling order, and task allocation relationship. The terminal push control result includes execution time window, instruction adjustment order, and rendering response offset. The entertainment information transmission result specifically refers to synchronous content output status, path jump structure, and cross-terminal transmission delay data.

[0007] As a further aspect of the present invention, the timing acquisition module includes: The rendering parameter acquisition submodule acquires the rendering timing parameters of multiple terminals in the current period, performs structured extraction and summarization on a periodic basis within each terminal, and obtains a set of rendering timing parameter sequences. The periodic difference calculation submodule, for each type of rendering timing parameter in the rendering timing parameter sequence set, divides the sliding window of each type of rendering timing parameter according to the period, counts the change segments of the sliding window of each type of rendering timing parameter, and obtains rendering timing change data. The state parameter output submodule, based on the rendering timing change data, for each type of rendering timing parameter in the current terminal, statistically analyzes the frequency of occurrence of each difference within the sliding window to construct a difference probability distribution, calculates the maximum entropy of the difference probability distribution, characterizes the uncertainty of the change of each type of rendering timing parameter in the corresponding terminal, and generates a terminal transmission state parameter group.

[0008] As a further aspect of the present invention, the rendering timing parameters include audio decoding time, video rendering time, and buffer waiting time.

[0009] As a further aspect of the present invention, the audio and video restoration module includes: The frame missing recognition submodule acquires the original audio and video content and performs pixel integrity detection. Based on the continuity of pixel distribution between the original audio and video frames, it determines whether there is a structural break in the current frame, locates the position range of the frame break missing area, and obtains the frame break missing area location result. The image structure extraction submodule calls the frame tomography missing region localization result, collects the gray level and texture structure parameters of pixel blocks in the frame images adjacent to the missing region, and constructs an inter-frame structure feature parameter set; The vector fitting and repair submodule estimates the pixel grid multi-scale vector between adjacent frames using the pyramid optical flow method based on the inter-frame structural feature parameter set. It matches a stable twisted vector segment as a reference and, combined with the missing area of ​​the frame tortuosity, performs line direction fitting based on vector continuity on the missing area of ​​the current frame to obtain the repaired video segment.

[0010] As a further aspect of the present invention, the inter-frame structural feature parameter set is composed of three feature parameters extracted from the grayscale and texture structure parameters of the pixel blocks collected in the adjacent frame images of the missing region: the horizontal grayscale gradient, the vertical texture jump amplitude, and the slope of the grayscale boundary within the block.

[0011] As a further aspect of the present invention, the task generation module includes: The window matching and recognition submodule obtains the available transmission window range of each terminal in the current period in the terminal transmission status parameter group, detects the transmission timestamp corresponding to each repaired video segment, determines whether it falls into the terminal's available transmission range, and obtains an allocable segment window lookup table. The content periodic mapping submodule calls the allocable segment window lookup table, combines the resource response feature parameters of each terminal in the terminal transmission status parameter group, and maps the allocable video segments periodically to obtain the transmission task mapping result. The sequential sorting output submodule sorts the content segments according to the priority requirements of the corresponding window of each terminal based on the transmission task mapping results, and arranges the task structure under each terminal in a unified manner to generate an entertainment information content push sorting table.

[0012] As a further aspect of the present invention, the push adjustment module includes: The instruction time detection submodule extracts the terminal identifier and sequence number corresponding to each content segment in the entertainment information content push sorting table, obtains the instruction response time of each terminal in the current period, and establishes a matching relationship between content segments and response times to obtain terminal response time distribution data. The task window comparison submodule calls the terminal response time distribution data, and according to the task execution window boundary corresponding to each segment in the entertainment information content push sorting table, compares the instruction response time with the task window interval position, identifies the degree of time offset between the two, and obtains the terminal task window offset data. The synchronous control output submodule adjusts the position of each terminal's push rhythm in the current cycle based on the terminal task window offset data, corrects the task execution window alignment state, and generates terminal push control results.

[0013] As a further aspect of the present invention, the synchronization transmission module includes: The path node acquisition submodule, based on the terminal push control results, obtains the target terminal identifier of each entertainment information content segment, as well as the node sequence and path time information that the content segment passes through during cross-terminal transmission, and obtains the segment transmission path dataset. The path feature analysis submodule counts the number of jumps for each node in the fragment transmission path dataset, calculates the average transmission time for the corresponding path, and identifies fluctuating segments in the transmission path based on the changing trends of the number of jumps and the transmission time, thereby obtaining the path transmission feature analysis results. The synchronous output execution submodule, based on the path transmission feature analysis results and combined with the adjusted push rhythm in the terminal push control results, synchronously configures the output rhythm of entertainment information content segments, executes unified push operations between multiple terminals, and obtains entertainment information transmission results.

[0014] On the other hand, an entertainment information transmission method integrating real-time interactive push from multiple terminals, which is executed based on the aforementioned entertainment information transmission system integrating real-time interactive push from multiple terminals, includes the following steps: S1: Obtain the multi-terminal rendering timing parameters in the current period of multiple terminals, analyze the response change magnitude of each type of rendering timing parameter in the corresponding terminal, and output the terminal transmission status parameter group. S2: Obtain the original audio and video content, identify and repair the grayscale and texture structure parameters of the pixel blocks in the frame fault missing area of ​​the original audio and video content, and output the repaired video clip; S3: Based on the terminal transmission status parameter group, identify and sort each repaired video segment that can be allocated to the terminal's transmission window in the current period, and generate an entertainment information content push sorting table; S4: Based on the entertainment information content push sorting table, schedule the instruction response time and task execution window of each terminal to generate terminal push control results; S5: Based on the terminal push control results, analyze the number of node jumps and path consumption time of each entertainment information content segment in the cross-terminal transmission path, and push out multi-terminal synchronized content to obtain the entertainment information transmission results.

[0015] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: By dynamically extracting periodic rendering parameters from each terminal and constructing an uncertainty metric, the content push rhythm can be adjusted according to the differences in transmission status of the terminal in different time periods. Combined with frame-level image restoration and content mapping rules, the timing of discontinuous images can be completed and synchronously distributed. The correspondence between content and time windows can be dynamically established based on push timeliness and terminal response. Furthermore, the node jump changes and time consumption trends in the transmission path can be correlated and analyzed to accurately determine the path fluctuation area. With the offset calculation within the period, the rhythm of terminal tasks can be fine-tuned. The multi-terminal content synchronization process can be completed by matching the timing of path nodes and coordinating the rhythm of output tasks. Under continuous push conditions, problems such as sudden changes in timeliness, frame rendering jitter, and transmission delay drift can be dynamically identified. A stable cross-terminal scheduling sequence can be constructed by updating the task rhythm and reconstructing the content priority, so as to achieve continuous adaptation and rhythm synchronization of periodic task windows, effectively solving problems such as content playback misalignment, task conflict, and response drift in a multi-terminal distributed environment. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of an entertainment information transmission system that integrates real-time interactive push from multiple terminals, provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the system framework of the present invention; Figure 3 This is a flowchart of the timing acquisition module in this invention; Figure 4 This is a flowchart of the audio and video restoration module in this invention; Figure 5 This is a flowchart of the task generation module in this invention; Figure 6 This is a flowchart of the push adjustment module in this invention; Figure 7 This is a flowchart of the synchronous transmission module in this invention. Detailed Implementation

[0018] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0019] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0020] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0021] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0022] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0023] This invention provides an entertainment information transmission system that integrates real-time interactive push from multiple terminals, such as... Figure 1-2 The diagram shown illustrates an entertainment information transmission system that integrates real-time interactive push notifications across multiple terminals. This system includes: The timing acquisition module acquires multi-terminal rendering timing parameters within the current period of multiple terminals, analyzes the response change magnitude of each type of rendering timing parameter in the corresponding terminal, and outputs the terminal transmission status parameter group. The audio and video restoration module acquires the original audio and video content, identifies and restores the grayscale and texture structure parameters of pixel blocks in the frame-fault missing areas of the original audio and video content, and outputs the restored video clip. The task generation module identifies and sorts each repaired video segment that can be allocated to the terminal's transmission window within the current period based on the terminal transmission status parameter group, and generates an entertainment information content push sorting table. The push adjustment module schedules the instruction response time and task execution window of each terminal according to the entertainment information content push sorting table, and generates terminal push control results. The synchronous transmission module analyzes the number of node jumps and path consumption of each entertainment information content segment in the cross-terminal transmission path based on the terminal push control results, and pushes out the multi-terminal synchronized content to obtain the entertainment information transmission results.

[0024] The terminal transmission status parameter group includes response time difference value, rendering timing distribution characteristics, and waiting time statistics. The repaired video segments are specifically structurally continuous segments, texture matching segments, and grayscale boundary fusion segments. The entertainment information content push sorting table specifically includes the content segment timing position, terminal scheduling order, and task allocation relationship. The terminal push control results include execution time window, instruction adjustment order, and rendering response offset. The entertainment information transmission results specifically refer to the synchronous content output status, path jump structure, and cross-terminal transmission delay data.

[0025] Specifically, such as Figure 2 , 3 As shown, the timing acquisition module includes: The rendering parameter acquisition submodule acquires the rendering timing parameters of multiple terminals in the current period, performs structured extraction and summarization on a periodic basis within each terminal, and obtains a set of rendering timing parameter sequences. Rendering timing parameters include audio decoding time, video rendering time, and buffer wait time; To obtain rendering timing parameters for the current period across multiple terminals, the first step is to call the media decoder log interface built into the operating system or player on each terminal to extract audio decoding time, video rendering time, and buffer waiting time. Record the occurrence time and duration of each parameter on the timeline. Within each terminal, these parameters are grouped and statistically analyzed in fixed periods (e.g., 100ms per period). The period boundary is advanced using a sliding window method, while invalid data such as dropped or silent frames are removed. Then, the rendering parameter data for each period is structured and encapsulated, for example, in the form of {period number: {audio decoding time: Xms, video rendering time: Yms, buffer waiting time: Zms}}. The encapsulated data from multiple periods are then arranged chronologically to form a sequence set. In real-world scenarios, such as online video conferencing, different terminals may capture video with different frame rates. In a certain period, a terminal might have a video rendering time of 22ms, an audio decoding time of 12ms, and a buffer waiting time of 15ms. After structured extraction and encapsulation, subsequent period change analysis and cross-terminal comparison are facilitated, ultimately yielding a sequence set of rendering timing parameters.

[0026] The periodic difference calculation submodule divides each type of rendering timing parameter in the rendering timing parameter sequence set into a sliding window according to the period, counts the change segments of the sliding window of each type of rendering timing parameter, and obtains the rendering timing change data. For each type of parameter in the acquired rendering timing parameter sequence set, period difference calculation is performed. First, parameters such as video rendering time are sorted by period, and a sliding window length is set (e.g., 10 periods). The window slides sequentially with a step size of 1, calculating the difference between the current period value and the previous period value within the window. The changing trend of all differences within the window is statistically analyzed. Jump values ​​or stable values ​​need to be labeled separately. If the video rendering time sequence of a certain terminal is [22ms, 24ms, 28ms, 26ms...], then the period difference calculation is performed at each step in the sliding window, such as Δ1 = 24-22=2ms, Δ2 = 28-24=4ms, Δ3 = 26-28=-2ms, and record them sequentially as the difference sequence {2,4,-2,...}. This process requires identifying trend change segments through time-series comparison. In practical applications, such as during live streaming monitoring, if the period difference is continuously positive, it is considered a cumulative delay trend; if it is negative, it is considered a rendering drop-off. A change segment that continues with the same trend for more than 3 periods constitutes a stable change segment. During this process, a threshold needs to be set to determine whether the difference fluctuation is significant. If the absolute value of the change exceeds 5ms, it is considered a trend jump; otherwise, it is considered noise disturbance. The statistically obtained change data sequences of video rendering, audio decoding, and buffering will be summarized.

[0027] The state parameter output submodule, based on the rendering timing change data, calculates the frequency of occurrence of each difference within the sliding window for the periodic difference sequence of each type of rendering timing parameter in the current terminal, constructs the difference probability distribution, calculates the maximum entropy of the difference probability distribution, characterizes the uncertainty of the change of each type of rendering timing parameter in the corresponding terminal, and generates a terminal transmission state parameter group. Based on the rendering timing data, a set of terminal transmission status parameters is generated. First, for each type of rendering timing parameter (e.g., video rendering time, audio decoding time, buffer waiting time), the set of periodic difference values ​​is obtained from the previous stage. Then, for the current terminal, the period is divided according to a fixed sliding window (e.g., one window every 10 periods). Within each window, the set of differences for that rendering parameter is extracted item by item. The difference data is divided into fixed numerical intervals, and each interval represents a statistical range. For example, when the difference changes at the millisecond level, it can be divided into 10 intervals, each interval with a width of 5ms.

[0028] After statistically analyzing the difference set of each type of rendering parameter, the number of data points falling within each interval is recorded. Then based on the total number of samples (That is, the total number of all differences within the sliding window) Calculate the probability of each interval. ,Right now: Next, we will analyze the probability distribution of each interval. Substituting into the information entropy formula, the uncertainty of the change of this type of rendering parameter under the current sliding window is calculated. The maximum entropy calculation formula used is: ; in, : Indicates the degree of uncertainty of the period difference change of a certain type of rendering timing parameter (such as audio decoding time) in the current terminal within the sliding window, that is, the complexity of the change trend of the parameter in the current period; : Indicates the number of the difference interval, such as interval 1, interval 2, numbered from smallest to largest difference range; : Represents the total number of intervals, usually set to 10, 20, etc. according to the desired precision, reflecting the degree of dispersion of the difference range; : for the first The probability of the difference occurring within the nth interval, i.e., the probability of the nth interval... The number of difference samples in the interval is divided by the total number of samples in the current sliding window; : for the difference falling into the first The number of samples in the interval; : This represents the total number of difference samples within the current sliding window, satisfying... .

[0029] In this way, the audio decoding time difference sequence, video rendering time difference sequence, and buffer waiting time difference sequence are statistically analyzed and entropy is calculated. These three sets of entropy values ​​will constitute three independent parameters in the terminal's current periodic state parameter group. Each entropy value individually reflects the rendering timing change of its corresponding type, and the overall result is used to feedback the current terminal's transmission and processing status. The final generated format is as follows: {Terminal ID: xxx, Video Rendering Entropy:} Audio decoding entropy: Buffer wait entropy: }

[0030] Suppose that the sequence of "video rendering time difference" obtained on the terminal device within a certain period of time is (unit: milliseconds): The sequence comes from 10 periods covered by the sliding window, meaning the total number of samples is: .

[0031] Divide the difference into intervals, setting each interval to 5ms, and set the total number of intervals to: The corresponding intervals are as follows: , , , For the others (values ​​that do not fall into any of the above intervals), the next step is to count the number of data points within each difference interval. This is used to calculate the probability of each interval. Interval 1 ( There are 2: -5, -5 → Interval 2 ( There are 3 '0's, '0's in total → Interval 3 ( There are 3 in total: 5, 5, 5 → Interval 4 ( There are 2: 10, 10 → Other intervals: No data → ,therefore: , , , , .

[0032] Substituting the values ​​into the information entropy formula, we can calculate the entropy value. The entropy value calculation formula is as follows: ; like Then this term is defined as 0, because ; Substitute each item in turn: Item 1: ; Item 2: ; Item 3: Same as above (also 0.3): Result is the same as Item 2 → 0.5211; Item 4: ; Item 5: ; Summing gives the final entropy value. .

[0033] Ultimately, the maximum entropy value of the difference in video rendering time within the sliding window is: This value, as a member of the terminal status parameters, is used to reflect the periodic fluctuation characteristics of the current terminal video rendering. Its value is not used for any purposeful explanation; it is only output as a statistical indicator to the terminal transmission status parameter group, and together with other parameters (such as audio decoding time entropy and buffer waiting time entropy), it constitutes a complete status description.

[0034] Specifically, such as Figure 2 , 4 As shown, the audio and video repair module includes: The frame missing recognition submodule acquires the original audio and video content and performs pixel integrity detection. Based on the continuity of pixel distribution between the original audio and video frames, it determines whether there is a structural break in the current frame, locates the position range of the frame break missing area, and obtains the frame break missing area location result. The system receives frame-by-frame data of the original audio and video content. During frame-by-frame analysis, each frame is traversed and detected using a pixel-level integrity detection method. This involves preprocessing the pixel distribution map of the current frame with two-dimensional mean filtering and edge-preserving filtering to extract the main structural region and edge change region of the pixel distribution. Then, a sliding comparison is performed on the pixel gradient change rate between the current frame and the adjacent frames. If a sudden pixel gradient change or structural break is found in a certain region, and the difference between this region and the corresponding region in the previous and next frames is above a set threshold range (e.g., the pixel grayscale mean square error is greater than 120, or the structural similarity SSIM is less than 0.4), it is determined that there is a frame structure fault in this region. Subsequently, the fault region is marked with a binary mask and its spatial coordinate range is located using a boundary tracking algorithm, such as the start and end positions of a rectangular region. Then, based on its contour boundary, the upper left and lower right pixel indices of the fault region are marked in the intra-frame coordinate system as the final location result of the fault missing region.

[0035] The image structure extraction submodule calls the frame tomography missing region localization results, collects the grayscale and texture structure parameters of pixel blocks in the frame images adjacent to the missing region, and constructs an inter-frame structure feature parameter set; The inter-frame structural feature parameter set is composed of three feature parameters extracted from the gray level and texture structure parameters of the pixel blocks in the adjacent frame images of the missing region: horizontal gray level gradient, vertical texture jump amplitude, and slope of gray level boundary within the block. After receiving the location information of the missing tomographic region, the image position corresponding to the region in the previous and next frames is selected. The gray-scale mean, gray-scale gradient value and texture edge amplitude of each pixel block are extracted block by block. In the process, the corresponding position region in adjacent frames is first divided into several image blocks of equal size, such as 8×8 pixels per block. Then, the gray-scale gradient change rate is calculated for each block as the horizontal gray-scale gradient parameter, and the change amplitude of the texture response function in the vertical direction is calculated as the vertical texture jump amplitude. Then, the slope of the gray-scale boundary in the block is fitted by the linear regression result of the gray-scale change slope of the pixels in the block. All feature parameters must meet the extraction stability judgment conditions, such as gray-scale standard deviation greater than 10 and texture direction response ratio greater than 1.5. Image blocks with edge occlusion or brightness flicker are excluded. Finally, an inter-frame structural feature parameter set is formed, which includes three features: horizontal gray-scale gradient, vertical texture jump amplitude and slope of gray-scale boundary in the block.

[0036] The vector fitting and repair submodule estimates the pixel grid multi-scale vector between adjacent frames using the pyramid optical flow method based on the inter-frame structural feature parameter set. It matches stable twisted vector segments as references and, combined with the missing frame tomography region, performs line direction fitting based on vector continuity on the missing region of the current frame to obtain the repaired video segment. During frame restoration, based on the inter-frame structural feature parameter set acquired in the previous stage, information such as pixel block grayscale gradients, texture jumps, and grayscale boundary slopes collected in adjacent frames is used to establish multi-scale pixel displacement vector relationships through the pyramid optical flow method to restore the missing regions of the current frame. First, a pyramid structure is constructed for each adjacent frame image, and each layer is scaled down to a smaller size than the original image. , , …multiple resolution levels are obtained proportionally, and each layer of image is used to estimate motion trends at different scales.

[0037] Within each pyramid layer, the submodule performs optical flow vector calculations for each pixel or pixel block between frames, obtaining the motion displacement relationship from the previous frame to the next frame, represented as a two-dimensional displacement vector for each point in the optical flow vector field: ; in: : Indicates the image coordinate position is The two-dimensional movement vector of a pixel between frames is used to indicate how it should be reconstructed at the missing location in the current frame; : The horizontal and vertical coordinate indices of the current pixel in the image; : This represents the horizontal (x-axis) displacement of the pixel, indicating the distance (in pixels) the point moves in the left and right directions between two frames. : This represents the vertical (y-axis) displacement of the pixel, indicating the vertical distance (in pixels) that the point moves between two frames.

[0038] The two-dimensional vectors mentioned above are obtained by solving the optical flow constraint equation, which is as follows: ; in, : This is the grayscale gradient of the current pixel in the horizontal direction (x direction), that is, the rate of change of pixel grayscale along the horizontal direction, which represents the intensity of the horizontal edge features of the pixel in the current frame; : This is the grayscale gradient of the current pixel in the vertical direction (y direction), that is, the rate of change of pixel grayscale along the vertical direction, reflecting the existence and intensity of the vertical edge; : This is the grayscale time difference of the pixel between consecutive frames, that is, the change in grayscale value of the pixel at the same position between two frames, used to measure the brightness change trend of the pixel due to motion; : These are the components of the optical flow in the x-axis and y-axis directions, respectively, which are the horizontal and vertical displacements that need to be solved.

[0039] This equation reflects the requirement that the same pixel should maintain consistent grayscale in adjacent frames under the assumptions of constant illumination and small displacement; that is, it establishes a constraint relationship based on the principle of grayscale consistency. By minimizing the residual through a local window, the grayscale value can be estimated. Value pairs allow the motion trend of a pixel to be determined between consecutive frames.

[0040] During the repair phase, vector fitting is based on these Vector inpainting maps the content of reference blocks in adjacent frames to the missing regions in the current frame. Linear interpolation is used to maintain edge integrity, with particular emphasis on directional consistency and amplitude stability checks in distorted vector regions. Criteria for participation include a directional deviation not exceeding 15 degrees and a displacement difference less than 2 pixels. This process restores the continuity and structural consistency of the missing frame's tortuous regions. Finally, the repaired results are stitched together to form a complete frame and inserted into the original video sequence, creating a playable video segment.

[0041] Optical flow constraint equation: ; Select three pixels in the image that are located at the same or nearby positions in adjacent frames, and obtain their spatial and temporal gradient values ​​(all values ​​below are calculated by the image convolutional function): Pixel A: Location Horizontal gradient: Vertical gradient: Time gradient: ; Substitute into the formula: , ; Pixel B: Position , , , ; Substitute into the formula: , ; Pixel C: Position , , , ; Substitute into the formula: , ; Three sets of formulas summarized , , ; The solution method involves a system of linear equations derived from image gradient calculations, used to solve for two variables. (That is, the horizontal and vertical optical flow vectors of this pixel). Since it involves solving three linear equations for two unknowns, the least squares method is used to minimize the sum of squared errors.

[0042] Least squares solution, constructed in matrix form: ; Normal system of equations: ; Calculation results: ; If we solve for: , ; The optical flow vector is: The result indicates that the pixel shifts by 2 pixels horizontally and 1 pixel vertically between adjacent frames. This vector is used to propagate along the rows within the missing region, achieving fitting and repair based on vector continuity.

[0043] In the entire vector fitting and restoration process, the final obtained two-dimensional displacement vector represents the specific motion path of each pixel in the current frame between adjacent frames, and is the fundamental data for determining how to reconstruct the content of the missing region. This displacement vector is obtained by constructing optical flow constraints and combining the spatial gradient and temporal grayscale changes of multiple pixels, using the least squares method to solve multiple linear equations, so that the result has optimal matching characteristics in the local range. Its horizontal component is used to represent the distance the pixel moves laterally, and its vertical component is used to represent the magnitude of the pixel's vertical movement. This vector not only reflects the structural changes of the local image, but also reflects the displacement trend of the image content between frames. The overall calculation process includes: constructing an image pyramid for multi-resolution processing, extracting the grayscale change information of the target pixel and its neighborhood in adjacent frames, calculating the spatial and temporal gradient of each pixel, establishing multiple linear optical flow equations within a local window, obtaining the corresponding displacement vector by solving them, and then performing vector interpolation and pixel restoration on the missing region.

[0044] Specifically, such as Figure 2 , 5 As shown, the task generation module includes: The window matching and recognition submodule obtains the available transmission window range for each terminal in the current period from the terminal transmission status parameter group, detects the transmission timestamp corresponding to each repaired video segment, determines whether it falls into the terminal's available transmission range, and obtains an allocable segment window lookup table. First, the transmission status parameter group of each terminal is traversed and parsed to extract the transmission availability window information for that terminal within the current period. This window information is typically determined by a combination of factors such as network bandwidth status, processing load level, and jitter stability. The available transmission window range is determined by preset rules; for example, a terminal is considered to be in an available transmission range only if its bandwidth is greater than 2Mbps, its average processing latency is less than 50ms, and its frame rendering entropy value is less than 1.8 within a certain period. Then, the original transmission timestamps of each repaired video segment are extracted. These timestamps are bound during video frame sequence encoding and represent the theoretical start time of the segment's transmission. The module compares the timestamps with the time range of each terminal's available transmission window to determine whether each video segment falls within the corresponding window range. If the timestamp is between the start and end times of the window, it is considered a successful match, and the matching relationship is recorded. All successfully matched video segments and terminal window pairs form a set of reference entries and are written into an allocable segment window lookup table. This lookup table is used for subsequent mapping operations and is key input data to ensure accurate placement of the distribution task.

[0045] The content periodic mapping submodule calls the allocable segment window lookup table, combines the resource response characteristic parameters of each terminal in the terminal transmission status parameter group, and maps the allocable video segments periodically to obtain the transmission task mapping results. First, the allocable segment window lookup table is invoked, and the resource response characteristic parameters recorded in the transmission status parameter group of each terminal are read synchronously. These parameters are used to measure the terminal's response efficiency and content processing capability when receiving tasks, and their composition may include terminal decoding rate, remaining buffer space, frame recovery latency, etc. For each video segment to be allocated, its adaptation period is determined based on the resource response characteristics of its target terminal and its transmission timestamp period. For example, if a terminal's buffer capacity is at a high threshold (e.g., remaining buffer space exceeds 80MB), then the terminal is allowed to prioritize the concurrent transmission of multiple segments; if its average decoding rate is lower than a set threshold (e.g., lower than 24fps), then continuous mapping will be skipped in the periodic mapping. The module performs periodic mapping on the segment allocation in the available window according to the periodic advancement method, corresponding to each segment being assigned to a specific periodic transmission task of a terminal. Finally, the terminal to which each segment belongs and the time period information to which it is mapped are generated, and the results of the transmission task mapping are summarized.

[0046] The sequential sorting output submodule sorts the content segments according to the priority requirements of each terminal window based on the transmission task mapping results, arranges the task structure under each terminal in a unified manner, and generates an entertainment information content push sorting table. After receiving the transmission task mapping results, a unified sorting process is initiated for the video content segments assigned to each terminal. First, the module categorizes all transmission segments by terminal number, ensuring that each terminal's corresponding transmission task set is processed independently. Within each terminal's task set, the attribute information carried by each segment is extracted, including content importance tags, timeliness tags, and entertainment level tags. The content importance tag is set by the content generation source; for example, key plot content is marked as level one importance, auxiliary background shots as level three, and advertising segments as level four. The timeliness tag indicates the content's time sensitivity; for example, live stream comments are marked as high timeliness, and pre-loaded segments as low timeliness. The entertainment level tag reflects the content's audience preferences; for example, immersive content is marked as high level.

[0047] Subsequently, the module sorts content according to a preset multi-level priority rule. The sorting rule first assigns a base score based on content importance tags, with primary importance tags receiving the highest priority. Next, it considers timeliness tags, prioritizing high-timeliness content. The sorting result is also fine-tuned based on the content's entertainment level. In the rule settings, if multiple segments have the same importance and timeliness tags, the module will also perform secondary sorting based on the order of appearance or content number to ensure that content of the same priority is pushed in a stable order.

[0048] To further refine the sorting strategy, the module has set the weights of all sorting indicators as adjustable parameters in the initial configuration, such as importance accounting for 70% of the sorting evaluation, timeliness accounting for 20%, and entertainment level accounting for 10%. During execution, multiple attribute tags for each segment are scored separately, and the scores are combined in a weighted manner to form the sorting criteria. After sorting, the module organizes the sequence information of the terminal's task segments into a standard format for output, ensuring that subsequent push logic can execute content distribution based on this order. Finally, the sorting structures of all terminals are integrated into a unified format entertainment information content push sorting table.

[0049] Specifically, such as Figure 2 , 6 As shown, the push adjustment module includes: The instruction time detection submodule extracts the terminal identifier and sequence number corresponding to each content segment in the entertainment information content push sorting table, obtains the instruction response time of each terminal in the current period, and establishes a matching relationship between content segments and response times to obtain terminal response time distribution data. First, the module parses the entertainment information content push sorting table line by line, extracting the terminal identifier information associated with each content segment and its sequence number within the terminal. This sequence number identifies the segment's position in the terminal's push process. Next, the module obtains the instruction response time of each terminal within the current period by connecting to the terminal communication interface or scheduling records. This is the time interval between the terminal receiving the push instruction and issuing a response. This response time data must be recorded using a precise clock synchronization mechanism, ensuring a minimum sampling accuracy of no more than 10 milliseconds. Then, the module constructs a matching relationship for each segment, associating the segment's terminal identifier, sequence number, and the terminal's response time for the corresponding period into a mapping record. If a terminal contains multiple segments within the current period, the response time for each segment is recorded; if a terminal has no response data in a certain period, it is marked as an invalid period. All matching records of segments and terminal response times are finally aggregated to form terminal response time distribution data. In this data structure, each record clearly indicates the terminal number, segment number, and corresponding response delay.

[0050] The task window comparison submodule calls the terminal response time distribution data, pushes the task execution window boundary corresponding to each segment in the entertainment information content push sort table, compares the instruction response time with the task window interval position, identifies the time offset between the two, and obtains the terminal task window offset data. The module reads terminal response time distribution data and pairs it with task execution window boundary information in the entertainment information content push sorting table. Each content segment in the sorting table contains the start and end time intervals of its target terminal's task execution, indicating when the segment should ideally begin and complete its push. The module compares the actual response time of each segment with its task window, identifying whether there is premature execution, delayed response, or cross-cycle drift by determining whether the response time falls within the window interval and the offset of the response time from the start or end of the window. The degree of offset is graded by setting thresholds: an offset time within ±30 milliseconds is considered an alignment interval, exceeding ±30 milliseconds but not exceeding ±100 milliseconds is considered a slight offset, and exceeding ±100 milliseconds is considered a severe offset. The module records the specific offset duration, offset direction, and corresponding terminal information for each segment, aggregating them by terminal to form terminal task window offset data. This data clearly reflects the time position offset of the push task within the current cycle for each terminal.

[0051] The synchronous control output submodule adjusts the position of each terminal's push rhythm in the current cycle based on the terminal task window offset data, corrects the task execution window alignment, and generates terminal push control results. After acquiring the terminal task window offset data, the module performs rhythm correction operations at the terminal level. First, it analyzes the offset of all segments in the current cycle for each terminal. If there are consecutive slight or severe offsets in the offset records, the module identifies a systematic drift in the current rhythm of that terminal. The module performs displacement correction based on the offset direction. If the offset direction is "advanced," the entire push task window for that terminal is moved backward by a fixed step in the next cycle; if it is "delayed," the window's starting position is moved forward. The correction step size is set according to the degree of offset, for example, 20 milliseconds for slight offsets and 50 milliseconds for severe offsets, with redundancy margins added to prevent reverse drift. During the correction process, it ensures that the new task window does not conflict with tasks in adjacent cycles, and uses cycle merging or task buffering strategies to eliminate congestion when necessary. After all corrections are completed, the new push window for the current cycle of each terminal is updated, and all window start and end times, segment order, and rhythm adjustment results are uniformly packaged to generate the terminal push control results.

[0052] Specifically, such as Figure 2 , 7 As shown, the synchronous transmission module includes: The path node acquisition submodule, based on the terminal push control results, obtains the target terminal identifier of each entertainment information content segment, as well as the node sequence and path time information that the content segment passes through during cross-terminal transmission, and obtains the segment transmission path dataset. First, the module extracts the target terminal identifier for each entertainment information content segment from the terminal push control results. This identifier is used to track the final transmission destination of the content. Then, the module calls the network routing tracing mechanism or transmission link log interface to query the complete path sequence formed by the content segment from the source address, through intermediate network transmission nodes, to the target terminal. The identifier information for each path node includes node type, address label, transmission interface type, etc., and records the specific time point when the segment passes through each node, thereby calculating the time consumption between that node and the next node. The module records each path hop-by-hop from the start to the end, generating a path node sequence organized by segment, and simultaneously calculates the transmission time between all hops, forming a complete path time list for that segment. It also checks the integrity and consistency of the path data. If there are anomalies such as overlapping nodes, reversed transmission times, or abrupt path changes, the path is marked as an abnormal path and submitted to the anomaly handling module. The transmission node sequences and corresponding time information for all normal paths ultimately form the segment transmission path dataset.

[0053] The path feature analysis submodule counts the number of jumps for each node in the fragment transmission path dataset, calculates the average transmission time for the corresponding path, and identifies fluctuating segments in the transmission path based on the changing trends of the number of jumps and the transmission time, thereby obtaining the path transmission feature analysis results. After receiving the fragmented transmission path dataset, the module begins statistical and evaluation processing for each path. First, it analyzes the number of hops in each path, determined by the number of nodes traversed between the transmission origin and the terminal. These nodes typically include edge gateways, routing forwarding nodes, and switching bridging points. The module statistically analyzes the distribution of hop counts across all paths and calculates the overall average transmission time for each path, calculated as the sum of the times taken at each hop divided by the number of hops. To identify path stability, the module further combines the hop count and average time analysis, plotting a trend curve over time to determine if an increase in hop count is accompanied by an abnormal increase in transmission time. If a path segment shows a constant hop count across three consecutive data collections but experiences a surge in transmission time exceeding the average by more than 20%, the module marks it as a fluctuating segment. Fluctuating segments are identified based on two dimensions: first, a concentrated abnormal fluctuation in the time taken at a node within the path; and second, the overall path time continuously exceeds a stable range without a change in hop count. The stable range is calculated using statistical distribution, typically defined as a range where the median fluctuates within 15%. All identified fluctuation segment locations, fluctuation amplitudes, involved segments, and node information will be summarized to generate path transmission feature analysis results.

[0054] The synchronous output execution submodule, based on the path transmission characteristic analysis results and combined with the adjusted push rhythm in the terminal push control results, synchronously configures the output rhythm of entertainment information content segments, executes unified push operations between multiple terminals, and obtains entertainment information transmission results. The module receives the path transmission characteristic analysis results and combines them with the existing rhythm adjustment data in the terminal push control results to perform the final synchronization configuration of the output rhythm. The module performs comprehensive scheduling on a per-segment basis, considering multiple dimensions such as the target terminal, path node status, and task time window, paying particular attention to segments with fluctuating sections in the path. If a segment's path contains a significant fluctuating section and the distance between it and the starting point of the target terminal's original push window is less than 100 milliseconds, the module will delay the push time of that segment by a minimum scheduling step (e.g., 50 milliseconds) to avoid unstable network nodes. If the fluctuating area is at the end of the path and the terminal is in a high-response state, the module will push the segment ahead of schedule within permissible limits. All rhythm adjustment operations must ensure no segment overlap or terminal reception congestion. Finally, the module integrates the output time, terminal location, and path synchronization adjustment information of each segment into a complete push instruction set. This instruction set is executed via the push control interface to distribute unified content across terminals, ultimately obtaining the multi-terminal synchronized transmission results of entertainment information.

[0055] The entertainment information transmission method integrating real-time interactive push from multiple terminals is executed based on the aforementioned entertainment information transmission system integrating real-time interactive push from multiple terminals, and includes the following steps: S1: Obtain the multi-terminal rendering timing parameters in the current period of multiple terminals, analyze the response change magnitude of each type of rendering timing parameter in the corresponding terminal, and output the terminal transmission status parameter group. S2: Obtain the original audio and video content, identify and repair the grayscale and texture structure parameters of the pixel blocks in the frame fault missing area of ​​the original audio and video content, and output the repaired video clip; S3: Based on the terminal transmission status parameter group, identify and sort each repaired video segment that can be allocated to the terminal's transmission window in the current period, and generate an entertainment information content push sorting table; S4: Based on the entertainment information content push sorting table, schedule the instruction response time and task execution window of each terminal, and generate terminal push control results; S5: Based on the terminal push control results, analyze the number of node jumps and path consumption of each entertainment information content segment in the cross-terminal transmission path, and push out multi-terminal synchronized content to obtain the entertainment information transmission results.

[0056] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An entertainment information transmission system integrating real-time interactive push from multiple terminals, characterized in that, The system includes: The timing acquisition module acquires multi-terminal rendering timing parameters within the current period of multiple terminals, analyzes the response change magnitude of each type of rendering timing parameter in the corresponding terminal, and outputs the terminal transmission status parameter group. The audio and video restoration module acquires the original audio and video content, identifies and restores the grayscale and texture structure parameters of pixel blocks in the frame-fault missing areas of the original audio and video content, and outputs the restored video clip. The task generation module identifies and sorts each repaired video segment that can be allocated to the terminal's transmission window in the current period based on the terminal transmission status parameter group, and generates an entertainment information content push sorting table. The push adjustment module schedules the instruction response time and task execution window of each terminal according to the entertainment information content push sorting table, and generates terminal push control results. The synchronous transmission module, based on the terminal push control results, analyzes the number of node jumps and path consumption time of each entertainment information content segment in the cross-terminal transmission path, and pushes out multi-terminal synchronized content to obtain the entertainment information transmission results.

2. The entertainment information transmission system integrating multi-terminal real-time interactive push according to claim 1, characterized in that: The terminal transmission status parameter group includes response time difference value, rendering timing distribution characteristics, and waiting time statistics. The repaired video segment specifically includes structurally continuous segment, texture matching segment, and grayscale boundary fusion segment. The entertainment information content push sorting table specifically includes content segment timing position, terminal scheduling order, and task allocation relationship. The terminal push control result includes execution time window, instruction adjustment order, and rendering response offset. The entertainment information transmission result specifically refers to synchronous content output status, path jump structure, and cross-terminal transmission delay data.

3. The entertainment information transmission system integrating multi-terminal real-time interactive push according to claim 1, characterized in that: The timing acquisition module includes: The rendering parameter acquisition submodule acquires the rendering timing parameters of multiple terminals in the current period, performs structured extraction and summarization on a periodic basis within each terminal, and obtains a set of rendering timing parameter sequences. The periodic difference calculation submodule, for each type of rendering timing parameter in the rendering timing parameter sequence set, divides the sliding window of each type of rendering timing parameter according to the period, counts the change segments of the sliding window of each type of rendering timing parameter, and obtains rendering timing change data. The state parameter output submodule, based on the rendering timing change data, for each type of rendering timing parameter in the current terminal, statistically analyzes the frequency of occurrence of each difference within the sliding window to construct a difference probability distribution, calculates the maximum entropy of the difference probability distribution, characterizes the uncertainty of the change of each type of rendering timing parameter in the corresponding terminal, and generates a terminal transmission state parameter group.

4. The entertainment information transmission system integrating multi-terminal real-time interactive push according to claim 3, characterized in that: The rendering timing parameters include audio decoding time, video rendering time, and buffer waiting time.

5. The entertainment information transmission system integrating multi-terminal real-time interactive push according to claim 1, characterized in that: The audio and video repair module includes: The frame missing recognition submodule acquires the original audio and video content and performs pixel integrity detection. Based on the continuity of pixel distribution between the original audio and video frames, it determines whether there is a structural break in the current frame, locates the position range of the frame break missing area, and obtains the frame break missing area location result. The image structure extraction submodule calls the frame tomography missing region localization result, collects the gray level and texture structure parameters of pixel blocks in the frame images adjacent to the missing region, and constructs an inter-frame structure feature parameter set; The vector fitting and repair submodule estimates the pixel grid multi-scale vector between adjacent frames using the pyramid optical flow method based on the inter-frame structural feature parameter set. It matches a stable twisted vector segment as a reference and, combined with the missing area of ​​the frame tortuosity, performs line direction fitting based on vector continuity on the missing area of ​​the current frame to obtain the repaired video segment.

6. The entertainment information transmission system integrating multi-terminal real-time interactive push according to claim 5, characterized in that: The inter-frame structural feature parameter set is composed of three feature parameters: horizontal grayscale gradient, vertical texture jump amplitude, and intra-block grayscale boundary slope, extracted from the grayscale and texture structure parameters of pixel blocks collected in adjacent frame images of the missing region.

7. The entertainment information transmission system integrating multi-terminal real-time interactive push according to claim 1, characterized in that: The task generation module includes: The window matching and recognition submodule obtains the available transmission window range of each terminal in the current period in the terminal transmission status parameter group, detects the transmission timestamp corresponding to each repaired video segment, determines whether it falls into the terminal's available transmission range, and obtains an allocable segment window lookup table. The content periodic mapping submodule calls the allocable segment window lookup table, combines the resource response feature parameters of each terminal in the terminal transmission status parameter group, and maps the allocable video segments periodically to obtain the transmission task mapping result. The sequential sorting output submodule sorts the content segments according to the priority requirements of the corresponding window of each terminal based on the transmission task mapping results, and arranges the task structure under each terminal in a unified manner to generate an entertainment information content push sorting table.

8. The entertainment information transmission system integrating multi-terminal real-time interactive push according to claim 1, characterized in that: The push adjustment module includes: The instruction time detection submodule extracts the terminal identifier and sequence number corresponding to each content segment in the entertainment information content push sorting table, obtains the instruction response time of each terminal in the current period, and establishes a matching relationship between content segments and response times to obtain terminal response time distribution data. The task window comparison submodule calls the terminal response time distribution data, and according to the task execution window boundary corresponding to each segment in the entertainment information content push sorting table, compares the instruction response time with the task window interval position, identifies the degree of time offset between the two, and obtains the terminal task window offset data. The synchronous control output submodule adjusts the position of each terminal's push rhythm in the current cycle based on the terminal task window offset data, corrects the task execution window alignment state, and generates terminal push control results.

9. The entertainment information transmission system integrating multi-terminal real-time interactive push according to claim 1, characterized in that: The synchronization transmission module includes: The path node acquisition submodule, based on the terminal push control results, obtains the target terminal identifier of each entertainment information content segment, as well as the node sequence and path time information that the content segment passes through during cross-terminal transmission, and obtains the segment transmission path dataset. The path feature analysis submodule counts the number of jumps for each node in the fragment transmission path dataset, calculates the average transmission time for the corresponding path, and identifies fluctuating segments in the transmission path based on the changing trends of the number of jumps and the transmission time, thereby obtaining the path transmission feature analysis results. The synchronous output execution submodule, based on the path transmission feature analysis results and combined with the adjusted push rhythm in the terminal push control results, synchronously configures the output rhythm of entertainment information content segments, executes unified push operations between multiple terminals, and obtains entertainment information transmission results.

10. A method for transmitting entertainment information that integrates real-time interactive push from multiple terminals, characterized in that: The entertainment information transmission system integrating multi-terminal real-time interactive push according to any one of claims 1-9 includes the following steps: S1: Obtain the multi-terminal rendering timing parameters in the current period of multiple terminals, analyze the response change magnitude of each type of rendering timing parameter in the corresponding terminal, and output the terminal transmission status parameter group. S2: Obtain the original audio and video content, identify and repair the grayscale and texture structure parameters of the pixel blocks in the frame fault missing area of ​​the original audio and video content, and output the repaired video clip; S3: Based on the terminal transmission status parameter group, identify and sort each repaired video segment that can be allocated to the terminal's transmission window in the current period, and generate an entertainment information content push sorting table; S4: Based on the entertainment information content push sorting table, schedule the instruction response time and task execution window of each terminal to generate terminal push control results; S5: Based on the terminal push control results, analyze the number of node jumps and path consumption time of each entertainment information content segment in the cross-terminal transmission path, and push out multi-terminal synchronized content to obtain the entertainment information transmission results.

Images