Home multi-screen interaction display method and system

Abstract

This invention relates to the field of multi-screen interaction technology and discloses a method and system for home multi-screen interactive display. The method involves: in the TV standby state, acquiring the user's position coordinate sequence and scanning a list of available devices; counting the number of times the user's gaze falls on the TV screen area; when the number exceeds a preset gaze threshold, displaying a screen casting confirmation interface; after receiving user confirmation, selecting the optimal transmission protocol and establishing a device connection with a mobile terminal device in the list of available devices; requesting video data within a preset time range from the video server based on the mobile terminal device's current playback progress timestamp and writing it to a local buffer; when the buffering time reaches a preset value, sending a buffer ready signal; and generating a smooth transition image between the mobile terminal device and the TV based on the buffer ready signal. This invention achieves smooth image transitions between devices with different frame rates, solving the stuttering problem caused by frame rate mismatch in existing technologies.

Description

Technical Field

[0001] This invention relates to the field of multi-screen interaction technology, and in particular to a method and system for home multi-screen interactive display. Background Technology

[0002] With the increasing popularity of smart home devices, users are increasingly looking to switch between multiple screens such as mobile phones and televisions to watch content.

[0003] Existing screen mirroring technologies require users to manually trigger the mirroring operation, and cannot automatically detect switching needs based on user behavior; the mirroring protocol is singular and fixed, resulting in poor compatibility between different brands of terminal devices, with AirPlay on iOS devices and Miracast on Android devices being incompatible; during the switching process, the TV screen experiences a 3 to 8 second black screen waiting period, leading to a discontinuous visual experience; the playback progress synchronization accuracy is insufficient, resulting in 0.5 to 2 seconds of repetition or skipping of content after switching; and the frame rate difference between the mobile phone (30fps) and the TV (60fps) causes stuttering and repetition during the transition period. Summary of the Invention

[0004] The main objective of this invention is to provide a method and system for multi-screen interactive display in the home. This invention achieves smooth screen transitions between devices with different frame rates and solves the stuttering problem caused by frame rate mismatch in the prior art.

[0005] To achieve the above objectives, the present invention provides a method for multi-screen interactive display in the home, comprising the following steps: While the TV is in standby mode, the system collects the user's location coordinate sequence and scans the list of available devices. The system counts the number of times the user's gaze falls on the TV screen area. When the number of times exceeds a preset gaze threshold, a screen casting confirmation interface is displayed. After receiving confirmation from the user, the system selects the optimal transmission protocol and establishes a device connection with the mobile terminal device in the list of available devices. The optimal transmission protocol is used to connect and send probe data packets to calculate the baseline round-trip delay and network jitter value. Based on the current playback progress timestamp of the mobile terminal device, the video server is requested to write video data within a preset time range into the local buffer. When the buffer time reaches the preset value, a buffer ready signal is sent. Based on the buffer ready signal, a smooth transition image is generated between the mobile terminal device and the TV.

[0006] Optionally, in a first implementation of the first aspect of the present invention, in the television standby state, collecting the user's location coordinate sequence and scanning the list of available devices includes: While the TV is in standby mode, the user's location coordinate sequence is collected; The movement direction vector is calculated based on the user's position coordinate sequence, and the angle between the movement direction vector and the normal direction of the TV screen is calculated. Based on the included angle and the vertical distance between the user and the screen, the wireless network scanning module is triggered to send a device discovery broadcast packet; Receive response packets from surrounding mobile terminal devices and parse the list of available devices.

[0007] Optionally, in a second implementation of the first aspect of the present invention, calculating the movement direction vector based on the user's position coordinate sequence and calculating the angle between the movement direction vector and the normal direction of the television screen includes: Select the position coordinates at the first moment, the position coordinates at the second moment, and the position coordinates at the third moment from the position coordinate sequence, and calculate the movement direction vector based on the position coordinates at the third moment and the position coordinates at the first moment; Calculate the dot product of the movement direction vector and the normal direction of the TV screen. Calculate the first magnitude of the movement direction vector and the second magnitude of the normal direction of the TV screen, respectively. Divide the dot product by the product of the first magnitude and the second magnitude, and take the inverse cosine value to obtain the included angle.

[0008] Optionally, in a third implementation of the first aspect of the present invention, the number of times the gaze point is located in the television screen area is counted, and when the number exceeds a preset gaze threshold, a screen casting confirmation interface is displayed. After receiving user confirmation, the optimal transmission protocol is selected and a device connection is established with the mobile terminal device in the list of available devices, including: The system acquires images of the user's eyeballs and extracts the coordinates of the pupil center. Based on the pupil center coordinates and the relative position of the user's head and the screen, it calculates the intersection of the gaze direction vector and the TV screen plane to obtain the coordinates of the gaze point. The number of times the coordinates of the gaze point fall on the TV screen area within a preset time window is counted. When the number of times exceeds a preset gaze threshold, a screen casting confirmation interface is displayed on the TV screen. After receiving user confirmation, the optimal transmission protocol is selected and the mobile terminal device is determined from the list of available devices; The corresponding protocol stack is started according to the optimal transmission protocol, and a connection invitation is sent to the mobile terminal device to establish a device connection.

[0009] Optionally, in a fourth implementation of the first aspect of the present invention, after receiving user confirmation, selecting the optimal transmission protocol and determining the mobile terminal device from the list of available devices includes: Extract the mobile terminal devices and their corresponding content types from the list of available devices; When the content type is video, the encoding format, resolution, frame rate, and bit rate are extracted to form a video feature vector; when the content type is game, the game frame rate requirement is extracted. Based on the video feature vector or the game frame rate requirement, a first weight coefficient, a second weight coefficient, a third weight coefficient, and a fourth weight coefficient are determined. Each candidate protocol in the protocol list supported by the mobile terminal device is traversed. The latency score, bandwidth score, compatibility score, and quality score of each candidate protocol are multiplied by the first weight coefficient, the second weight coefficient, the third weight coefficient, and the fourth weight coefficient, and then summed to obtain the protocol score of each candidate protocol. The optimal transport protocol is selected based on the protocol scores of each candidate protocol.

[0010] Optionally, in the fifth implementation of the first aspect of the present invention, the baseline round-trip delay and network jitter value are calculated by sending probe data packets through the optimal transmission protocol connection, and video data within a preset time range is requested from the video server to be written into the local buffer according to the current playback progress timestamp of the mobile terminal device. When the buffer time reaches a preset value, a buffer ready signal is sent, including: Multiple probe data packets are continuously sent to the TV terminal through the optimal transmission protocol connection and acknowledgment data packets are received. The baseline round-trip delay and network jitter value are calculated. The system receives playback status data packets sent by mobile terminal devices and extracts the current playback progress timestamp. Based on the current playback progress timestamp, it subtracts the advance time margin and adds the subsequent time margin to determine the request range and sends a range request to the video server. It writes the received video stream data into the local buffer and establishes a mapping index from timestamp to buffer address. When the buffer time reaches a preset value, it sends a buffer ready signal.

[0011] Optionally, in a sixth implementation of the first aspect of the present invention, generating a smooth transition image between the mobile terminal device and the television terminal based on the buffer ready signal includes: After receiving the buffer ready signal, the mobile terminal device reads the current playback timestamp and the corresponding frame number to construct a switching control data packet and sends it to the TV. The TV calculates the transmission delay by subtracting the clock deviation from the difference between the receiving time and the sending time, calculates the estimated value of the current playback progress based on the transmission delay and the playback timestamp, and finds the target frame in the local buffer whose timestamp is closest to the estimated value of the playback progress and locates the decoder read pointer. The mobile terminal device continuously captures the current display frame and calculates the pixel difference with the previous frame to obtain frame differential data, which is then transmitted to the TV. The TV reconstructs the mobile terminal device frame and decodes the local buffer frame to generate a smooth transition image.

[0012] Optionally, in a seventh implementation of the first aspect of the present invention, the home multi-screen interactive display method further includes: The mobile terminal device transmits a first playback timestamp attached to a frame and a second playback timestamp of a locally decoded TV frame, and calculates a timestamp offset value based on the first playback timestamp and the second playback timestamp; When the timestamp deviation value is positive and exceeds the first synchronization threshold, the timestamp deviation value is divided by the single frame duration, rounded down, and then incremented by one to obtain the fast step amount. The decoder skips intermediate frames and reads subsequent frames according to the fast step amount. When the timestamp deviation value is negative and exceeds the second synchronization threshold, the absolute value of the timestamp deviation value is taken as the waiting delay duration, and the decoder pauses reading the waiting delay duration and continues to read the next frame.

[0013] Optionally, in an eighth implementation of the first aspect of the present invention, the home multi-screen interactive display method further includes: Find two consecutive frames that satisfy the condition that the timestamp of the nth frame is less than or equal to the refresh time of the TV screen and that the refresh time is less than the timestamp of the (n+1)th frame. Calculate the difference between the refresh time and the timestamp of the nth frame and divide it by the difference between the timestamp of the (n+1)th frame and the timestamp of the nth frame to obtain the time position ratio. For each pixel coordinate of two consecutive frames, the pixel color value of the nth frame is multiplied by the result of subtracting the time position ratio to obtain a first target value. The pixel color value of the (n+1)th frame is multiplied by the time position ratio to obtain a second target value. The first target value and the second target value are summed to obtain an interpolated frame.

[0014] The present invention also provides a home multi-screen interactive display system, comprising: The acquisition module is used to acquire the user's location coordinate sequence and scan the list of available devices when the TV is in standby mode; The device connection module is used to count the number of times the gaze point is located in the TV screen area. When the number of times exceeds the preset gaze threshold, the screen casting confirmation interface is displayed. After receiving user confirmation, the optimal transmission protocol is selected and a device connection is established with the mobile terminal device in the list of available devices. The buffer ready module is used to connect and send probe data packets through the optimal transmission protocol to calculate the baseline round-trip delay and network jitter value, request video data within a preset time range from the video server according to the current playback progress timestamp of the mobile terminal device and write it into the local buffer, and send a buffer ready signal when the buffer time length reaches the preset value. A smooth transition module is used to generate a smooth transition image between the mobile terminal device and the TV based on the buffer ready signal.

[0015] In summary, this invention achieves an intelligent switching trigger mechanism that requires no manual operation by combining the calculation of movement direction vectors from the user's position coordinate sequence using an infrared thermal imaging sensor array with gaze statistical analysis from a gaze tracking module. This solves the problem of existing technologies relying on user-initiated clicks. Furthermore, by using machine learning to classify the content type of mobile terminal devices to obtain video feature vectors or game frame rate requirements, and calculating a protocol scoring function based on latency, bandwidth, compatibility, and quality scores to select the optimal transmission protocol, this invention achieves intelligent multi-protocol adaptation across brands and systems, solving the problem of single and incompatible protocols in existing technologies. Finally, by controlling the frame difference transmitted to mobile terminal devices within a preset transition time period… The reconstructed data frames and locally decoded frames are pixel-level mixed according to a dynamic weighting function, achieving a smooth transition between dual-source images and eliminating the visual interruption of black screen waiting during switching in existing technologies. By dynamically adjusting the decoder frame reading step or waiting delay by calculating the timestamp deviation between the mobile terminal device's transmitted frames and the TV's local decoded frames, the playback progress synchronization accuracy is controlled within the frame level, solving the playback progress misalignment problem in existing technologies. By linearly interpolating the pixel color values ​​between two consecutive frames on the mobile terminal device at the time of TV screen refresh to obtain interpolated frames with matching frame rates, a smooth image transition between devices with different frame rates is achieved, solving the stuttering problem caused by frame rate mismatch in existing technologies. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the steps of a family multi-screen interactive display method in one embodiment of the present invention; Figure 2 This is a structural block diagram of a home multi-screen interactive display system in an embodiment of the present invention.

[0017] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] Reference Figure 1 This embodiment provides a method for multi-screen interactive display in the home, including the following steps: S1, while the TV is in standby mode, collect the user's location coordinate sequence and scan the list of available devices; S2, count the number of times the gaze falls on the TV screen area. When the number exceeds the preset gaze threshold, display the screen casting confirmation interface. After receiving user confirmation, select the optimal transmission protocol and establish a device connection with the mobile terminal device in the list of available devices. S3 connects and sends probe data packets through the optimal transmission protocol to calculate the baseline round-trip delay and network jitter value. Based on the current playback progress timestamp of the mobile terminal device, it requests video data within a preset time range from the video server to write to the local buffer. When the buffer time reaches the preset value, it sends a buffer ready signal. S4 generates a smooth transition between the mobile terminal device and the TV based on the buffer ready signal.

[0020] In one example, while the TV is in standby mode, the user's location coordinate sequence is collected and a list of available devices is scanned, including: While the TV is in standby mode, the user's location coordinate sequence is collected; The movement direction vector is calculated based on the user's location coordinate sequence, and the angle between the movement direction vector and the normal direction of the TV screen is calculated. Based on the included angle and the vertical distance between the user and the screen, the wireless network scanning module is triggered to send a device discovery broadcast packet; Receive response packets from surrounding mobile terminal devices and parse the list of available devices.

[0021] In this example, when the TV is in standby mode and the main display screen is off, the system control chip maintains a low-power operation and continuously scans the environment at a low frequency using an infrared thermal imaging sensor array mounted on top of the TV. This array, composed of multiple infrared detection units, covers a large area in front of the TV. It acquires a sequence of three-dimensional spatial coordinates of the heat source target from the sensor array at fixed time intervals, representing each target position as a three-dimensional vector coordinate P(x, y, z), where x is the horizontal displacement of the target relative to the center of the TV screen, y is the vertical displacement, and z is the vertical distance between the user and the screen. After continuously recording the position coordinates at multiple time points, at least three consecutive time points t1, t2, and t3 are selected, and the corresponding position coordinates P1, P2, and P3 are extracted. The average movement direction vector V from P1 to P3 is then calculated. m = (P3 - P1) / (t3 - t1), the average movement direction vector reflects the user's movement trend within the current time window. It is calculated using the normal direction N of the TV screen. s As a reference direction, N s Defined as a unit vector perpendicular to the screen plane and pointing in the user's direction, then using the vector angle formula θ = arccos((V m · N s ) / (|V m | × |N sThe system calculates the angle between the user's movement direction and the area directly in front of the screen. When the angle is less than a preset threshold (e.g., 45 degrees), meaning the user is moving towards the TV, and the user's z-axis coordinate (vertical distance from the screen) continuously decreases from an initial value (e.g., 5 meters) to a preset range (e.g., within 2.5 meters), and the entire approach process lasts for more than a preset time window (e.g., 2 seconds) to exclude temporary passing by, the system recognizes active approach behavior and immediately triggers the wireless network scanning module. At this time, the scanning module activates the TV's Wi-Fi adapter and Bluetooth communication module and broadcasts a device discovery broadcast packet to the surrounding area. The device discovery broadcast packet contains the TV's unique identifier, current timestamp, and a list of supported transmission protocol types, including Miracast, AirPlay, DLNA, and Wi-Fi Direct. Simultaneously, within the set response time window, all received response packets are continuously monitored. Each response packet is actively sent by surrounding mobile terminal devices and contains a unique identifier for the responding device, device type information (e.g., iPhone, Android), supported protocol types, currently playing content identifier, content type identifier (e.g., video, audio, webpage), and wireless signal strength indicators. After receiving all response packets, they are parsed to extract the various attributes of the responding devices and construct a list of available devices. Each item in the list corresponds to a recognizable candidate terminal device for screen mirroring that has the currently playing content. This device list is temporarily stored in memory.

[0022] In one example, the movement direction vector is calculated based on the user's position coordinate sequence, and the angle between the movement direction vector and the normal direction of the TV screen is calculated, including: Select the position coordinates at the first moment, the second moment, and the third moment from the position coordinate sequence, and calculate the movement direction vector based on the position coordinates at the third moment and the position coordinates at the first moment; Calculate the dot product of the movement direction vector and the normal direction of the TV screen. Calculate the first magnitude of the movement direction vector and the second magnitude of the normal direction of the TV screen, respectively. Divide the dot product by the product of the first and second magnitudes and take the inverse cosine value to obtain the included angle.

[0023] In this example, while the television is in standby mode, the system control module continuously acquires a sequence of the user's spatial position coordinates from the infrared thermal imaging sensor array, and represents the position coordinates corresponding to each sampling moment as a three-dimensional vector P(t). i ) = (x i , y i , z i ), where t i For timestamps, x i y i z iThese represent the user's spatial coordinates in the horizontal, vertical, and depth directions, respectively. Three sampling points that satisfy a time-increasing relationship are selected sequentially from the position coordinate sequence: position coordinates at time 1 P1 = (x1, y1, z1), position coordinates at time 2 P2 = (x2, y2, z2), and position coordinates at time 3 P3 = (x3, y3, z3). The vector difference between P3 and P1 is taken as the user's movement direction vector V. m V m = P3 - P1 = (x3 - x1, y3 - y1, z3 - z1), constructing a direction vector representing the user's overall movement trend over a time interval. To determine whether this direction is towards the TV screen, the normal direction vector N of the TV screen is predefined. s = (x n ,y n , z n The normal vector is perpendicular to the plane containing the center point of the television screen and points directly forward into external space. For V... m With N s Perform the three-dimensional vector dot product calculation, that is, calculate V. m · N s = (x3 - x1)×x n + (y3 - y1)×y n + (z3 -z1)×z n Obtain the projected inner product value between these two vectors. Calculate V respectively. m The first modulus length, i.e., ||V m || = √[(x3 -x1) 2 + (y3 - y1) 2 + (z3 - z1) 2 ], and N s The second modulus, i.e., ||N s || = √(x n 2 + y n 2 + z n 2 Divide the above dot product result by the product of these two moduli to calculate the cosine value cosθ = (V m · N s ) / (‖V m || × || N s Then, take the inverse cosine function of cosθ, i.e., θ = arccos(cosθ), to obtain the angle θ between the user's movement direction and the normal direction of the TV screen.

[0024] In one example, the system counts the number of times the user's gaze falls on the TV screen area. When this number exceeds a preset gaze threshold, a screen mirroring confirmation interface is displayed. After receiving user confirmation, the system selects the optimal transmission protocol and establishes a device connection with a mobile terminal device from the available device list, including: The system acquires images of the user's eyeballs and extracts the coordinates of the pupil center. Based on the coordinates of the pupil center and the relative position of the user's head and the screen, it calculates the intersection of the gaze direction vector and the TV screen plane to obtain the coordinates of the gaze point. The system counts the number of times the gaze point coordinates are located in the TV screen area within a preset time window. When the number exceeds a preset gaze threshold, a screen casting confirmation interface is displayed on the TV screen. After receiving user confirmation, the system selects the optimal transmission protocol and determines the mobile terminal device from the list of available devices. The corresponding protocol stack is started according to the optimal transmission protocol, and a connection invitation is sent to the mobile terminal device to establish a device connection.

[0025] In this example, after the TV system enters the screen casting detection phase, the infrared eye-tracking module installed on top of the TV captures images of the user's face at a set frame rate. Each acquired frame undergoes face detection processing. After accurately locating the facial region using a Haar cascade classifier, the image regions of the left and right eyes are extracted from the facial region. An edge detection algorithm (such as Canny) combined with a circular fitting method is then applied to extract the boundaries of both pupils, determining the center coordinates of the left and right pupils respectively. The average of these two pupil center coordinates yields the average gaze point coordinates (x, y) of the user's eye position at the current moment. p , y p (and combined with the three-dimensional spatial position P of the user's head relative to the center of the TV screen obtained from the infrared thermal imaging sensor) 用户 = (x u , y u , z uBased on a pinhole camera model, a user's gaze direction vector is constructed. This vector originates from the center of the user's eyeball along the optical axis. The intersection of the gaze direction vector with the screen plane is calculated based on the spatial geometric relationship between the vector and the screen plane, yielding the coordinates of the gaze point at the current moment. The gaze points, sampled every 100 milliseconds, are continuously recorded within a time window to form a sequence. The number of times the gaze point coordinates fall within the physical boundary area of ​​the TV screen within this time window is counted. When the statistical result exceeds a preset gaze threshold (e.g., more than 20 out of 30 samples fall within the screen area), indicating that the user has clearly watched TV, the TV system lights up the main screen and displays an interactive screen mirroring confirmation interface. This interface shows a list of available devices obtained through a previous wireless scan, including the device name, device type icon, thumbnail of the currently playing content, and signal quality information. Users can confirm the connection via a remote control or a pop-up window on their mobile phone. Upon receiving the confirmation, the system selects the user-specified mobile device from the device list as the target connection. Based on the supported screen mirroring protocols and the current content type, the system evaluates the performance of all available protocols using a protocol scoring function, selecting the highest-scoring optimal transmission protocol. The system then starts the protocol stack corresponding to the selected optimal protocol. For Wi-Fi Direct, it configures P2P group owner mode and sends a P2P connection invitation; for AirPlay, it starts the Bonjour service and listens on port 7000; for Miracast, it establishes an RTSP session and negotiates WFD parameters; and for DLNA, it starts the UPnP media renderer service. Finally, it sends a connection invitation signal to the target terminal device. Upon receiving the signal, the terminal device automatically starts the corresponding client protocol module, and both sides enter a handshake process to complete the connection establishment.

[0026] In one example, after receiving user confirmation, the optimal transport protocol is selected and the mobile terminal device is determined from the list of available devices, including: Extract mobile terminal devices and their corresponding content types from the list of available devices; When the content type is video, extract the encoding format, resolution, frame rate and bit rate to form a video feature vector; when the content type is game, extract the game frame rate requirement. The first weight coefficient, the second weight coefficient, the third weight coefficient, and the fourth weight coefficient are determined based on the video feature vector or game frame rate requirements. The candidate protocols in the protocol list supported by the mobile terminal device are traversed. The latency score, bandwidth score, compatibility score, and quality score of each candidate protocol are multiplied by the first weight coefficient, the second weight coefficient, the third weight coefficient, and the fourth weight coefficient, respectively, and then summed to obtain the protocol score of each candidate protocol. The optimal transport protocol is selected based on the protocol scores of each candidate protocol.

[0027] In this example, the device identifier of the user-selected mobile terminal device and the content type field currently being played on that device are extracted from the list of available devices. Content types include categories such as video, game, document, and audio. Content feature analysis is performed based on the extracted content type field. When the content type is identified as video, the encoding format (e.g., H.264, H.265), resolution parameters (e.g., 1920×1080), frame rate parameters (e.g., 30fps), and bitrate parameters (e.g., 5000kbps) of the video content are extracted from the media metadata returned by the terminal device. These four parameters are then combined in a fixed order to form a video feature vector = [encoding format, resolution, frame rate, bitrate], which serves as input for the protocol adaptation process. When the content type is game, the game's frame rate requirements for video output are extracted first. For example, action or racing games require a frame rate of at least 60fps, and this frame rate requirement is used as input for protocol scoring. Based on content feature vectors or frame rate requirements, the application scenario's different needs for transmission latency, bandwidth utilization, protocol compatibility, and picture quality are determined. Four weighted coefficients are then set for the protocol scoring function: the first weight coefficient W1 represents latency sensitivity; the second weight coefficient W2 represents bandwidth requirements; the third weight coefficient W3 represents the requirements for terminal and TV protocol compatibility; and the fourth weight coefficient W4 represents the priority of transmission picture quality fidelity. For example, W2 has a higher weight when the content is high-bitrate high-definition video, while W1 has a higher priority when the content is high-frame-rate low-latency games. All candidate protocols in the list of protocols supported by the mobile terminal device are traversed, and for each protocol, its latency score L is evaluated in the current network environment by looking up a table or using a model. p Bandwidth rating B p Compatibility rating: C p And quality rating Q p The protocol is then comprehensively scored based on a weighted scoring function, with the specific calculation formula being S. p = W1×L p + W2×B p + W3×C p + W4×Q p S p This represents the protocol score for each candidate protocol. The scores are calculated for each candidate protocol, compared, and the protocol with the highest score is selected as the optimal transmission protocol for the current scenario.

[0028] In one example, probe packets are sent via an optimal transport protocol connection to calculate baseline round-trip latency and network jitter values. Based on the mobile terminal device's current playback progress timestamp, video data within a preset time range is requested from the video server and written to the local buffer. When the buffering time reaches a preset value, a buffer-ready signal is sent, including: Multiple probe data packets are continuously sent to the TV terminal through the optimal transmission protocol connection and acknowledgment data packets are received. The baseline round-trip delay and network jitter value are calculated. The system receives playback status data packets sent by mobile terminal devices and extracts the current playback progress timestamp. Based on the current playback progress timestamp, it subtracts the advance time margin and adds the subsequent time margin to determine the request range and sends a range request to the video server. It writes the received video stream data into the local buffer and establishes a mapping index from the timestamp to the buffer address. When the buffer time reaches the preset value, it sends a buffer ready signal.

[0029] In this example, after the mobile terminal device and the TV establish a connection using the selected optimal transmission protocol, the mobile terminal device initiates a network performance measurement module. It continuously sends multiple probe packets to the TV at fixed time intervals through the protocol connection channel. Each probe packet contains a unique sequence number field, a sending timestamp field, and a verification field for integrity checks. Upon receiving each probe packet, the TV constructs an acknowledgment packet and returns it along the same path. The acknowledgment packet contains the corresponding sequence number, a receiving timestamp, and an acknowledgment sending timestamp. After receiving each acknowledgment packet, the mobile terminal device calculates the round-trip delay for each probe packet based on the sending and receiving timestamps, and summarizes the round-trip times of all probe packets into a delay time series. From the delay time series, the median is extracted as the baseline round-trip delay of the current network channel. Simultaneously, the standard deviation of the time series is derived using variance calculation to measure the intensity of current network transmission jitter, i.e., the network jitter value. Both together constitute the quality assessment parameters for the transmission channel. The mobile terminal device sends a playback status packet to the TV. This packet contains multiple fields, including the playback progress timestamp, total content duration, encoding format, resolution, frame rate, bitrate, and media resource address. After receiving the playback status data packet, the TV extracts the current playback progress timestamp and subtracts a preset advance time margin (e.g., 2 seconds) and adds a subsequent time margin (e.g., 5 seconds) from it to obtain a target time range. Based on this target time range, the TV sends a video data range request to the original video server, using the Range header field in the HTTP protocol or the PLAY directive in the RTSP protocol with the time range parameter. The server returns video stream data segments within the corresponding time range according to the request. The TV writes the received data into a local buffer in blocks. The local buffer is organized in a circular structure, with each data block accompanied by its corresponding video timestamp. The system also maintains a mapping index structure with timestamps as keys and buffer address offsets as values, such as using a B-tree or hash mapping to build the index table. As video data continues to be received, the length of time covered by the buffered video data is continuously counted. When the accumulated buffer time reaches a preset threshold (e.g., 4 seconds) and covers the time period near the playback progress timestamp, the buffer is considered ready, and a buffer ready signal is constructed and sent to the mobile terminal device through an optimal protocol connection.

[0030] In one example, a smooth transition between the mobile terminal device and the TV is generated based on a buffer ready signal, including: After receiving the buffer ready signal, the mobile terminal device reads the current playback timestamp and the corresponding frame number to construct a switching control data packet and send it to the TV. The TV calculates the transmission delay by subtracting the clock deviation from the difference between the receiving time and the sending time. It calculates the estimated current playback progress based on the transmission delay and the playback timestamp, and searches for the target frame in the local buffer whose timestamp is closest to the estimated playback progress and locates the decoder reading pointer. The mobile terminal device continuously captures the current display frame and calculates the pixel difference with the previous frame to obtain frame differential data, which is then transmitted to the TV. The TV reconstructs the mobile terminal device frame and decodes the local buffer frame to generate a smooth transition image.

[0031] In this example, after the mobile terminal device receives the buffer ready signal from the TV, it reads the timestamp T from the current playback state from the local player system. s and the absolute frame number F corresponding to the timestamp s The frame number is calculated by dividing the playback time by the duration of a single frame and is used to identify the position of the currently displayed video frame. Mobile terminal devices use a playback timestamp T. s Frame number F s and the current sending time T of the local system e The handover control data packet is jointly constructed and sent to the television terminal through the established optimal transmission protocol channel. The handover control data packet also includes a checksum field for data integrity verification. Upon receiving the handover control data packet, the television terminal records the reception time T. r And based on the pre-established clock synchronization deviation ΔT between the local terminal and the mobile terminal, the actual network transmission delay D is calculated. t = T r - T e - ΔT, the network transmission delay value, represents the actual time-domain propagation overhead from the terminal issuing control commands to the television receiving them. The television uses this transmission delay D. t With switching timestamp T s Add them together to get an estimated value T of the current playback progress. p That is, T p = T s + D t This estimate reflects the theoretical time position of the content actually being played by the terminal device at the current moment. To achieve decoder alignment, the TV terminal uses a timestamp index to look up the distance from the estimated time T in its locally built video data buffer. pThe most recent data block determines the location of the target video frame and positions the decoder's read pointer to the corresponding starting offset position, ensuring that decoding begins from the correct frame sequence point. This eliminates the need to reload the entire video sequence, shortening decoding preparation time and improving response speed. Simultaneously, the mobile terminal device enters frame differential data capture mode, which continuously captures the currently playing video frame image within each video frame cycle. n and the frame image from the previous cycle. n-1 Pixel-by-pixel comparison is performed to calculate the pixel difference ΔFrame = Frame between the current frame and the previous frame. n - Frame n-1 Frame differencing is performed at the pixel level, calculating the channel-by-channel difference of the RGB color channels at each position (x, y) in the image, resulting in a new differencing image. The frame differencing data is then processed by lightweight compression algorithms such as run-length encoding (RLE) to reduce the transmitted data size and is transmitted to the television in real time via the transmission protocol channel. Upon receiving the differencing data, the television performs decoding to reconstruct the complete frame image and then reconstructs the current frame image pixel-by-pixel by overlaying the reconstructed previous frame image. n =Frame n-1 + ΔFrame, forming a video frame consistent with the current display on the terminal device. Simultaneously, the local video decoding module on the TV starts the buffer to read and decode the corresponding frame based on the previous playback progress estimate, forming the local output frame. The TV's image rendering system performs pixel-level linear blending of the frames transmitted from the terminal device and the locally decoded frames within the current display cycle. Using a dynamic weighting function, it controls the transparency ratio of the terminal frame and the local frame within the transition window according to the time progress, thereby generating a smoothly transitioning blended image on the screen.

[0032] In one example, the family multi-screen interactive display method also includes: The mobile terminal device transmits a first playback timestamp attached to a frame and a second playback timestamp of a locally decoded TV frame, and calculates a timestamp offset value based on the first playback timestamp and the second playback timestamp. When the timestamp deviation value is positive and exceeds the first synchronization threshold, the timestamp deviation value is divided by the single frame duration, rounded down, and then incremented by one to obtain the fast step size. The decoder skips intermediate frames and reads subsequent frames according to the fast step size. When the timestamp deviation value is negative and exceeds the second synchronization threshold, the absolute value of the timestamp deviation value is taken as the waiting delay duration. The decoder pauses reading and continues reading the next frame after waiting for the delay duration.

[0033] In this example, when the mobile terminal device enters the frame data transmission phase, each frame differential data packet includes the first playback timestamp of that frame on the terminal device. Meanwhile, the local video decoder on the TV continuously maintains the second playback timestamp of the current output frame during its internal decoding process. At any given time point, these two timestamps are compared; that is, the timestamp deviation value Δt = T1 - T2 is obtained by subtracting the second playback timestamp T2 from the first playback timestamp T1. The timestamp deviation value reflects whether the playback progress of the terminal device is ahead or behind the TV. If the timestamp deviation value Δt is positive and greater than the first synchronization threshold, for example, greater than 50 milliseconds, it indicates that the playback of the terminal device is significantly ahead of the TV. In this case, the TV needs to adopt an acceleration strategy to catch up. The positive deviation value is divided by the single frame duration (e.g., 33.33 milliseconds), and the result is rounded down and then incremented by one to obtain the fast step size N. 快速 The fast step size indicates the number of frames that the TV needs to skip to reduce decoding latency. Instead of decoding the next frame sequentially, the TV decoder directly skips N frames forward from the decoder read pointer. 快速 - The offset position of frame 1, extract the Nth frame from the buffer. 快速 Frames are decoded, and skipped frame data is discarded, significantly shortening the distance between playback timelines and achieving rapid progress alignment. When the timestamp deviation value Δt is negative and its absolute value exceeds the second synchronization threshold (e.g., less than -50 milliseconds), it indicates that the playback progress on the TV is significantly ahead of the terminal device. In this case, the system does not perform frame skipping but instead enters a waiting strategy. The absolute value of Δt is used to obtain the waiting delay time T. 等 = |Δt|, then pause the decoder's frame reading operation, and maintain the delay time T by calling a timed delay mechanism (such as sleep or a high-precision timer) through the delay control module. 等 During the waiting period, the decoder does not perform any new frame decoding operations, maintaining the current frame output unchanged, thus delaying the display of the next frame and allowing the playback progress to gradually converge with the terminal device. After the waiting period ends, the system resumes the decoder's normal decoding path and continues to sequentially read the next frame from the buffer queue for decoding. The entire control process forms a closed-loop feedback control mechanism by periodically measuring the deviation between the first and second playback timestamps and executing synchronization judgment logic and response strategy selection within a 100-millisecond cycle, ensuring that the playback progress deviation stably converges to within ±10 milliseconds within a short period of time.

[0034] In one example, the family multi-screen interactive display method also includes: Find two consecutive frames that satisfy the condition that the timestamp of the nth frame is less than or equal to the refresh time of the TV screen and the refresh time is less than the timestamp of the (n+1)th frame. Calculate the difference between the refresh time and the timestamp of the nth frame and divide it by the difference between the timestamp of the (n+1)th frame and the timestamp of the nth frame to obtain the time position ratio. For each pixel coordinate in two consecutive frames, the pixel color value of the nth frame is multiplied by the result of subtracting the time position ratio to obtain the first target value. The pixel color value of the (n+1)th frame is multiplied by the time position ratio to obtain the second target value. The first target value and the second target value are summed to obtain the interpolated frame.

[0035] In this example, based on the playback timestamp information attached to the transmission frames from the terminal device and the decoding frames from the television, the data of two adjacent frames before and after the current moment are obtained, that is, the nth frame and the (n+1)th frame that meet the conditions are found, where the timestamp T of the nth frame is... n Less than or equal to the current TV screen refresh time T r The timestamp T of the (n+1)th frame n+1 Then greater than T r This ensures that the current time falls within the time interval of these two frames. After confirming these two frames, calculate the time difference Δ1 = T between the refresh time and the timestamp of the nth frame. r - T n Simultaneously calculate the time difference Δ2 = T between the (n+1)th frame and the nth frame. n+1 - T n The normalized time position ratio (ratio ∈ [0,1)) of the refresh time is obtained by dividing the two time differences by Δ1 / Δ2. This ratio reflects the progress of time interpolation between the two keyframes at the current time. Pixel-level linear interpolation is performed on all pixels in these two consecutive frames in the GPU graphics processing module. For each screen pixel coordinate (x, y), the pixel color value Color of the nth frame is extracted. n (x,y) = [R n G n B n ] and the pixel color value Color of the (n+1)th frame n+1 (x,y) = [R n+1 G n+1 B n+1 For each color channel, interpolation is performed, where the pixel value of the nth frame is multiplied by 1 and subtracted from the time-position ratio to obtain the first target value: R1 = R n × (1 - ratio), G1 = G n ×(1 - ratio), B1 = B n × (1 - ratio), and the pixel value of the (n+1)th frame multiplied by ratio gives the second target value: R2 = R n+1 × ratio, G2 = G n+1 × ratio, B2 = Bn+1 × ratio. Add the first target value and the second target value channel by channel, i.e., R = R1 + R2, G = G1 + G2, B = B1 + B2, to obtain the value at refresh time T. r The interpolated frame pixel color values ​​to be presented. This process is performed in parallel for each pixel, and the entire interpolated frame is composed of the pixel-level fusion results. The picture has temporal continuity and visual smoothness when transitioning from frame n to frame n+1. The interpolation mechanism is executed in conjunction with the bilinear interpolation hardware unit in the GPU texture sampler, which greatly reduces the computation latency. It can also work in conjunction with the screen's vertical synchronization signal (VSync) to ensure that there is an interpolated frame result for each refresh, so that the original 30fps video source content is smoothly presented on the TV screen at a 60Hz refresh rate, avoiding frame repetition, picture jumps, or discontinuous motion.

[0036] Reference Figure 2 This embodiment provides a home multi-screen interactive display system, including: Acquisition module 1 is used to acquire the user's location coordinate sequence and scan the list of available devices when the TV is in standby mode; Device connection module 2 is used to count the number of times the gaze falls on the TV screen area. When the number exceeds the preset gaze threshold, the screen casting confirmation interface is displayed. After receiving user confirmation, the optimal transmission protocol is selected and a device connection is established with the mobile terminal device in the list of available devices. The buffer ready module 3 is used to send probe data packets through the optimal transmission protocol to calculate the baseline round-trip delay and network jitter value, request video data within a preset time range from the video server based on the current playback progress timestamp of the mobile terminal device and write it into the local buffer, and send a buffer ready signal when the buffer time reaches the preset value. The smooth transition module 4 is used to generate a smooth transition image between the mobile terminal device and the TV based on the buffer ready signal.

[0037] In this embodiment, the specific implementation of each unit in the above system embodiment is described in the above method embodiment, and will not be repeated here.

[0038] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, system, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, system, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, system, article, or method that includes that element.

[0039] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for multi-screen interactive display in a home, characterized in that, include: While the TV is in standby mode, the system collects the user's location coordinate sequence and scans the list of available devices. The system counts the number of times the user's gaze falls on the TV screen area. When the number of times exceeds a preset gaze threshold, a screen casting confirmation interface is displayed. After receiving confirmation from the user, the system selects the optimal transmission protocol and establishes a device connection with the mobile terminal device in the list of available devices. The optimal transmission protocol is used to connect and send probe data packets to calculate the baseline round-trip delay and network jitter value. Based on the current playback progress timestamp of the mobile terminal device, the video server is requested to write video data within a preset time range into the local buffer. When the buffer time reaches the preset value, a buffer ready signal is sent. A smooth transition between the mobile terminal device and the TV is generated based on the buffer ready signal.

2. The home multi-screen interactive display method according to claim 1, characterized in that, In TV standby mode, the system collects the user's location coordinate sequence and scans the list of available devices, including: While the TV is in standby mode, the user's location coordinate sequence is collected; The movement direction vector is calculated based on the user's location coordinate sequence, and the angle between the movement direction vector and the normal direction of the TV screen is calculated. Based on the included angle and the vertical distance between the user and the screen, the wireless network scanning module is triggered to send a device discovery broadcast packet; Receive response packets from surrounding mobile terminal devices and parse the list of available devices.

3. The home multi-screen interactive display method according to claim 2, characterized in that, Calculate the movement direction vector based on the user's location coordinate sequence, and calculate the angle between the movement direction vector and the normal direction of the TV screen, including: Select the position coordinates at the first moment, the position coordinates at the second moment, and the position coordinates at the third moment from the position coordinate sequence, and calculate the movement direction vector based on the position coordinates at the third moment and the position coordinates at the first moment; Calculate the dot product of the movement direction vector and the normal direction of the TV screen. Calculate the first magnitude of the movement direction vector and the second magnitude of the normal direction of the TV screen, respectively. Divide the dot product by the product of the first magnitude and the second magnitude, and take the inverse cosine value to obtain the included angle.

4. The home multi-screen interactive display method according to claim 1, characterized in that, The system counts the number of times the user's gaze falls on the TV screen area. When this number exceeds a preset gaze threshold, a screen casting confirmation interface is displayed. After receiving user confirmation, the system selects the optimal transmission protocol and establishes a device connection with the mobile terminal device in the available device list, including: The system acquires images of the user's eyeballs and extracts the coordinates of the pupil center. Based on the pupil center coordinates and the relative position of the user's head and the screen, it calculates the intersection of the gaze direction vector and the TV screen plane to obtain the coordinates of the gaze point. The number of times the coordinates of the gaze point fall on the TV screen area within a preset time window is counted. When the number of times exceeds a preset gaze threshold, a screen casting confirmation interface is displayed on the TV screen. After receiving user confirmation, the optimal transmission protocol is selected and the mobile terminal device is determined from the list of available devices; The corresponding protocol stack is started according to the optimal transmission protocol, and a connection invitation is sent to the mobile terminal device to establish a device connection.

5. The home multi-screen interactive display method according to claim 4, characterized in that, After receiving user confirmation, the system selects the optimal transmission protocol and determines the mobile terminal device from the list of available devices, including: Extract the mobile terminal devices and their corresponding content types from the list of available devices; When the content type is video, the encoding format, resolution, frame rate, and bit rate are extracted to form a video feature vector; when the content type is game, the game frame rate requirement is extracted. Based on the video feature vector or the game frame rate requirement, a first weight coefficient, a second weight coefficient, a third weight coefficient, and a fourth weight coefficient are determined. Each candidate protocol in the protocol list supported by the mobile terminal device is traversed. The latency score, bandwidth score, compatibility score, and quality score of each candidate protocol are multiplied by the first weight coefficient, the second weight coefficient, the third weight coefficient, and the fourth weight coefficient, and then summed to obtain the protocol score of each candidate protocol. The optimal transport protocol is selected based on the protocol scores of each candidate protocol.

6. The home multi-screen interactive display method according to claim 1, characterized in that, The optimal transmission protocol is used to connect and send probe data packets to calculate the baseline round-trip delay and network jitter value. Based on the current playback progress timestamp of the mobile terminal device, video data within a preset time range is requested from the video server to be written to the local buffer. When the buffering time reaches a preset value, a buffer ready signal is sent, including: Multiple probe data packets are continuously sent to the TV terminal through the optimal transmission protocol connection and acknowledgment data packets are received. The baseline round-trip delay and network jitter value are calculated. The system receives playback status data packets sent by mobile terminal devices and extracts the current playback progress timestamp. Based on the current playback progress timestamp, it subtracts the advance time margin and adds the subsequent time margin to determine the request range and sends a range request to the video server. It writes the received video stream data into the local buffer and establishes a mapping index from timestamp to buffer address. When the buffer time reaches a preset value, it sends a buffer ready signal.

7. The home multi-screen interactive display method according to claim 6, characterized in that, Generating a smooth transition between the mobile terminal device and the television based on the buffer ready signal includes: After receiving the buffer ready signal, the mobile terminal device reads the current playback timestamp and the corresponding frame number to construct a switching control data packet and sends it to the TV. The TV calculates the transmission delay by subtracting the clock deviation from the difference between the receiving time and the sending time, calculates the estimated value of the current playback progress based on the transmission delay and the playback timestamp, and finds the target frame in the local buffer whose timestamp is closest to the estimated value of the playback progress and locates the decoder read pointer. The mobile terminal device continuously captures the current display frame and calculates the pixel difference with the previous frame to obtain frame differential data, which is then transmitted to the TV. The TV reconstructs the mobile terminal device frame and decodes the local buffer frame to generate a smooth transition image.

8. The home multi-screen interactive display method according to claim 1, characterized in that, The family multi-screen interactive display method also includes: The mobile terminal device transmits a first playback timestamp attached to a frame and a second playback timestamp of a locally decoded TV frame, and calculates a timestamp offset value based on the first playback timestamp and the second playback timestamp; When the timestamp deviation value is positive and exceeds the first synchronization threshold, the timestamp deviation value is divided by the single frame duration, rounded down, and then incremented by one to obtain the fast step amount. The decoder skips intermediate frames and reads subsequent frames according to the fast step amount. When the timestamp deviation value is negative and exceeds the second synchronization threshold, the absolute value of the timestamp deviation value is taken as the waiting delay duration, and the decoder pauses reading the waiting delay duration and continues to read the next frame.

9. The home multi-screen interactive display method according to claim 8, characterized in that, The family multi-screen interactive display method also includes: Find two consecutive frames that satisfy the condition that the timestamp of the nth frame is less than or equal to the refresh time of the TV screen and that the refresh time is less than the timestamp of the (n+1)th frame. Calculate the difference between the refresh time and the timestamp of the nth frame and divide it by the difference between the timestamp of the (n+1)th frame and the timestamp of the nth frame to obtain the time position ratio. For each pixel coordinate of two consecutive frames, the pixel color value of the nth frame is multiplied by the result of subtracting the time position ratio to obtain a first target value. The pixel color value of the (n+1)th frame is multiplied by the time position ratio to obtain a second target value. The first target value and the second target value are summed to obtain an interpolated frame.

10. A home multi-screen interactive display system, characterized in that, The steps for implementing the home multi-screen interactive display method according to any one of claims 1 to 9 include: The acquisition module is used to acquire the user's location coordinate sequence and scan the list of available devices when the TV is in standby mode; The device connection module is used to count the number of times the gaze point is located in the TV screen area. When the number of times exceeds the preset gaze threshold, the screen casting confirmation interface is displayed. After receiving user confirmation, the optimal transmission protocol is selected and a device connection is established with the mobile terminal device in the list of available devices. The buffer ready module is used to connect and send probe data packets through the optimal transmission protocol to calculate the baseline round-trip delay and network jitter value, request video data within a preset time range from the video server according to the current playback progress timestamp of the mobile terminal device and write it into the local buffer, and send a buffer ready signal when the buffer time length reaches the preset value. A smooth transition module is used to generate a smooth transition image between the mobile terminal device and the TV based on the buffer ready signal.

Images