Synchronization time delay measurement method, content synchronization method, terminal device and storage medium

Abstract

This application relates to the field of terminal technology, and particularly to a synchronization latency measurement method, a content synchronization method, a terminal device, and a storage medium. This method can accurately measure the synchronization latency of screen synchronization, video synchronization, and audio synchronization between a first terminal device and a second terminal device. This allows the first terminal device to determine the connection information between the two devices when sharing content with the second terminal device for synchronized playback. Based on the shared content and the connection information, the synchronization latency for synchronized playback is determined, thereby controlling the content playback on the first and / or second terminal devices according to the synchronization latency. This ensures the synchronization of content playback in multi-screen interaction, improves playback quality, and enhances user experience.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of terminals, and particularly relates to a synchronization time delay measurement method, a content synchronization method, a terminal device, and a computer readable storage medium. BACKGROUND

[0002] With the development of terminal technology, more and more terminal devices have a multi-screen interaction function to synchronize playing of content on different terminal devices. For example, a certain terminal device can project a screen of the terminal device to other terminal devices for synchronized playing, or can project audio of the terminal device to other terminal devices for synchronized playing, to facilitate user browsing or improve the playing effect of content. However, when multi-screen interaction is performed, the content often needs to be encoded, transmitted, or decoded, and the like. Since time delay is likely to occur in the encoding, transmission, or decoding, and the like, the content playing is not synchronized, which affects user experience. SUMMARY

[0003] Embodiments of the present application provide a synchronization time delay measurement method, a content synchronization method, a terminal device, and a computer readable storage medium, which can accurately measure the synchronization time delay of synchronized playing, to ensure the synchronization of content playing in multi-screen interaction according to the synchronization time delay, and improve user experience.

[0004] In a first aspect, embodiments of the present application provide a synchronization time delay measurement method applied to a first terminal device, for measuring a synchronization time delay in screen synchronization. The method can include:

[0005] The first terminal device establishes a network connection between the first terminal device and a second terminal device according to first connection information;

[0006] The first terminal device plays a first video and projects a video frame in the first video to the second terminal device;

[0007] The first terminal device acquires a photographed image, the photographed image including a second video frame currently played by the second terminal device, the second video frame being a video frame in the first video;

[0008] The first terminal device determines a synchronization time delay of screen synchronization between the first terminal device and the second terminal device according to the photographed image.

[0009] Through the above synchronization time delay measurement method, in screen synchronization, the first terminal device can play a first video and project a video frame in the played first video to a second terminal device. Then, a display interface of the second terminal device can be photographed to obtain a photographed image, so that the synchronization time delay of screen synchronization between the first terminal device and the second terminal device can be accurately determined according to the photographed image.

[0010] In a possible implementation manner of the first aspect, the first terminal device obtaining the photographed image can include:

[0011] The first terminal device obtains the photographed image of the second terminal device through a camera of the first terminal device.

[0012] In the synchronization delay measurement provided in this solution, the first terminal device can obtain the photographed image through the camera of the first terminal device, that is, the photographed image only includes the display interface of the second terminal device, so as to facilitate the obtaining of the photographed image, thereby the synchronization delay can be determined according to the photographed image, and the user experience is improved.

[0013] For example, the first terminal device determines the synchronization delay of the first terminal device and the second terminal device for picture synchronization according to the photographed image, which includes:

[0014] The first terminal device determines the second playing time of the second video frame according to the photographed image.

[0015] The first terminal device obtains a first playing time of a first video frame played by the first terminal device at the same time, the first playing time and the second playing time are playing times of the video frame in the first video, and the same time is a time when the second terminal device plays the second video frame.

[0016] The first terminal device determines the synchronization delay of the first terminal device and the second terminal device for picture synchronization according to the first playing time and the second playing time.

[0017] In the synchronization delay measurement method provided in this solution, the first terminal device can obtain the video playing progress (that is, the second playing time) of the second terminal device according to the photographed image, and can obtain the video playing progress (that is, the first playing time) of the first terminal device at the same time in the background, so as to determine the synchronization delay of the first terminal device and the second terminal device for picture synchronization according to the time difference between the video playing progress of the first terminal device and the video playing progress of the second terminal device at the same time.

[0018] For example, each video frame of the first video is provided with a corresponding video identifier, and the video identifier includes a frame serial number.

[0019] The first terminal device determines the synchronization delay of the first terminal device and the second terminal device for picture synchronization according to the photographed image, which can include:

[0020] The first terminal device determines a second playing time of the second video frame according to the photographed image, and determines a second frame serial number of the second video frame according to the video identifier in the photographed image;

[0021] The first terminal device acquires a first playing time at which the first terminal device plays the video frame with the second frame serial number;

[0022] The first terminal device determines a synchronization time delay for the first terminal device to perform picture synchronization with the second terminal device according to the first playing time and the second playing time.

[0023] The first playing time and the second playing time are system times of the first terminal device or the second terminal device.

[0024] In the synchronization time delay measurement method provided in the scheme, the first terminal device can acquire the system time (i.e., the second playing time) and the frame serial number (i.e., the second frame serial number) of the video frame currently played by the second terminal device according to the photographed image, and can acquire the system time (i.e., the first playing time) at which the first terminal device plays the video frame with the same frame serial number in the background, so as to determine the synchronization time delay for the first terminal device to perform picture synchronization with the second terminal device according to the time difference between the system times at which the first terminal device and the second terminal device play the same video frame.

[0025] For example, each video frame of the first video is provided with a corresponding video identifier, and the video identifier includes a frame serial number and a frame rate.

[0026] The first terminal device determines a synchronization time delay for the first terminal device to perform picture synchronization with the second terminal device according to the photographed image, which can include:

[0027] The first terminal device acquires the second frame serial number and the frame rate of the second video frame according to the video identifier in the photographed image, and acquires a first frame serial number of a first video frame played by the first terminal device at the same time, the same time being a time at which the second terminal device plays the second video frame.

[0028] The first terminal device determines a synchronization time delay for the first terminal device to perform picture synchronization with the second terminal device according to the first frame serial number, the second frame serial number, and the frame rate.

[0029] Specifically, the first terminal device acquires a first frame serial number of a first video frame played by the first terminal device at the same time, which can include:

[0030] The first terminal device acquires a first frame serial number of a first video frame played by the first terminal device at the same time according to a video identifier in the first video frame.

[0031] In the synchronization delay measurement method provided in the scheme, the first terminal device can obtain a frame sequence number (i.e., a second frame sequence number) of a video frame currently played by the second terminal device according to the captured image, and can obtain a first frame sequence number of a video frame currently played by the first terminal device in the background, so as to determine a frame difference between the video frame currently played by the first terminal device and the video frame currently played by the second terminal device according to the first frame sequence number and the second frame sequence number, and determine a synchronization delay of the first terminal device and the second terminal device for picture synchronization according to the frame difference and a frame rate corresponding to the first video (e.g., frame number*frame rate).

[0032] In a possible implementation form of the first aspect, the first terminal device obtaining the captured image can include:

[0033] The first terminal device obtains the captured image through a camera of a third terminal device, the third terminal device being a terminal device other than the first terminal device and the second terminal device, and the captured image further including a first video frame currently played by the first terminal device.

[0034] In the synchronization delay measurement method provided in the scheme, the third terminal device can capture pictures of the display interfaces of the first terminal device and the second terminal device, to obtain the captured image. That is, the captured image includes the display interface of the first terminal device and the display interface of the second terminal device, so that the captured image can be directly analyzed to determine the synchronization delay of the first terminal device and the second terminal device for picture synchronization.

[0035] For example, the first terminal device determining the synchronization delay of the first terminal device and the second terminal device for picture synchronization according to the captured image can include:

[0036] The first terminal device determines a first playing time of the first video frame and a second playing time of the second video frame according to the captured image, the first playing time and the second playing time being playing times of the video frames in the first video.

[0037] The first terminal device determines the synchronization delay of the first terminal device and the second terminal device for picture synchronization according to the first playing time and the second playing time.

[0038] In the synchronization delay measurement method provided in the scheme, the first terminal device can directly obtain the video playing progress of the first terminal device and the video playing progress of the second terminal device according to the captured image, so as to determine the synchronization delay of the first terminal device and the second terminal device for picture synchronization according to the time difference between the video playing progress of the first terminal device and the video playing progress of the second terminal device at the same time.

[0039] For example, the first video frame of the first video is provided with a corresponding video identifier, and the video identifier includes a frame sequence number and a frame rate.

[0040] The first terminal device determines the synchronization delay of the first terminal device and the second terminal device for picture synchronization according to the captured image, which can include:

[0041] The first terminal device determines the first frame sequence number and the frame rate of the first video frame according to the first video identifier in the captured image, and determines the second frame sequence number of the second video frame according to the second video identifier in the captured image.

[0042] The first terminal device determines the synchronization delay of the first terminal device and the second terminal device for picture synchronization according to the first frame sequence number, the second frame sequence number and the frame rate.

[0043] In the synchronization delay measurement method provided in the scheme, the first terminal device can directly obtain the first frame sequence number of the video frame played by the first terminal device and the second frame sequence number of the video frame played by the second terminal device according to the captured image, so as to determine the number of frames between the video frame played by the first terminal device and the video frame played by the second terminal device at the same time according to the first frame sequence number and the second frame sequence number, and determine the synchronization delay of the first terminal device and the second terminal device for picture synchronization according to the number of frames and the frame rate corresponding to the first video.

[0044] For example, the captured image includes multiple images, and the first terminal device determines the synchronization delay of the first terminal device and the second terminal device for picture synchronization according to the captured image, which can include:

[0045] The first terminal device determines the second playing time of the second video frame according to the first captured image, and determines the second frame sequence number of the second video frame according to the video identifier in the first captured image, wherein the first captured image is any one of the captured images.

[0046] The first terminal device determines the second captured image according to the second frame sequence number, and the second captured image includes the video frame of the second frame sequence number played by the first terminal device.

[0047] The first terminal device determines a first playing time of the first terminal device playing the video frame of the second frame number according to the second photographed image;

[0048] The first terminal device determines a synchronization delay of the first terminal device and the second terminal device for picture synchronization according to the first playing time and the second playing time;

[0049] The first playing time and the second playing time are system times of the first terminal device or the second terminal device.

[0050] In the synchronization delay measurement method provided in the scheme, a plurality of photographed images can be obtained by recording the pictures currently displayed by the first terminal device and the second terminal device through the third terminal device, and the system times when the first terminal device and the second terminal device play the same video frame at the same time can be obtained according to the plurality of photographed images, so as to determine the synchronization delay according to the time difference between the system times.

[0051] It can be understood that the first connection information can include a network type and a network topology structure. The video identifier can be a two-dimensional code.

[0052] In a second aspect, an embodiment of the present application provides a synchronization delay measurement method applied to a first terminal device, for measuring a synchronization delay in video synchronization. The method can include:

[0053] The first terminal device establishes a network connection between the first terminal device and a second terminal device according to first connection information;

[0054] The first terminal device plays a first video and projects an audio and / or a video frame in the first video to the second terminal device;

[0055] The first terminal device obtains a target audio and / or a photographed image, and determines a synchronization delay in video synchronization according to the target audio and / or the photographed image.

[0056] Through the above synchronization delay measurement method, in video synchronization, the first terminal device can play a first video, and project a video frame and / or an audio in the played first video to a second terminal device. Then, a display picture of the second terminal device can be photographed, and / or an audio of the second terminal device can be collected, so as to determine a synchronization delay in video synchronization according to a photographed image and / or a target audio.

[0057] In a possible implementation manner of the second aspect, each video frame of the first video is provided with a corresponding audio, and frequencies of the corresponding audios of the video frames are different from each other;

[0058] The target audio is audio currently played by the second terminal device when the video synchronization is synchronization of video frames played by the first terminal device and audio played by the second terminal device;

[0059] The first terminal device determines a synchronization time delay in the video synchronization according to the target audio and / or the photographed image, including:

[0060] The first terminal device obtains a frequency of the target audio, and determines a second frame number of a second video frame corresponding to the target audio according to the frequency of the target audio;

[0061] The first terminal device obtains a first frame number of a first video frame played by the first terminal device at the same time, the same time being a time when the second terminal device plays the target audio;

[0062] The first terminal device determines a synchronization time delay of the first terminal device and the second terminal device in the video synchronization according to the first frame number, the second frame number, and a frame rate of the first video.

[0063] In the synchronization time delay measurement method provided in the scheme, when the video synchronization is synchronization of video frames played by the first terminal device and audio played by the second terminal device, the audio played by the second terminal device can be recorded to obtain target audio. Then, a video frame corresponding to the target audio is determined according to a frequency of the target audio, and a video frame played by the first terminal device at the same time is obtained, so as to determine a synchronization time delay between the video frame played by the first terminal device and the audio played by the second terminal device according to a frame number difference between the two video frames and a frame rate corresponding to the first video.

[0064] In a possible implementation form of the second aspect, each video frame of the first video is provided with corresponding audio, and frequencies of the audio corresponding to the video frames are different from each other.

[0065] The target audio is audio currently played by the second terminal device when the video synchronization is synchronization of video frames played by the first terminal device and audio played by the second terminal device;

[0066] The first terminal device determines a synchronization time delay in the video synchronization according to the target audio and / or the photographed image, including:

[0067] The first terminal device obtains a frequency of the target audio and a second playing time of the target audio played by the second terminal device;

[0068] The first terminal device determines the second frame number of the second video frame corresponding to the target audio based on the frequency of the target audio, and obtains the first playback time of the video frame with the second frame number played by the first terminal device;

[0069] The first terminal device determines the synchronization delay between the first terminal device and the second terminal device for video synchronization based on the first playback time and the second playback time;

[0070] Wherein, the first playback time and the second playback time are the system time of the first terminal device.

[0071] In the synchronization delay measurement method provided by this scheme, when video synchronization is achieved by synchronizing the video frame played by the first terminal device with the audio played by the second terminal device, the audio played by the second terminal device can be recorded to obtain the target audio, and the playback time corresponding to the target audio can be acquired. Then, the video frame corresponding to the target audio can be determined based on the frequency of the target audio, and the playback time of the video frame played by the first terminal device can be acquired, so as to determine the synchronization delay based on the time difference between the two playback times.

[0072] In one possible implementation of the second aspect, each video frame of the first video is provided with a corresponding video identifier and audio, wherein the video identifier includes a frame number and a frame rate, and the frequency of the audio corresponding to each video frame is different from each other.

[0073] When the video synchronization is synchronized between the audio played by the first terminal device and the video frame played by the second terminal device, the captured image is the image corresponding to the second terminal device, and the captured image includes the second video frame currently being played by the second terminal device.

[0074] The first terminal device determines the synchronization delay in video synchronization based on the target audio and / or the captured image, which may include:

[0075] The first terminal device determines the second frame number and frame rate of the second video frame based on the video identifier in the captured image;

[0076] The first terminal device obtains the frequency of the first audio played by the first terminal device at the same time, and determines the first frame number of the first video frame corresponding to the frequency of the first audio.

[0077] The first terminal device determines the synchronization delay for video synchronization between the first terminal device and the second terminal device based on the first frame sequence number, the second frame sequence number, and the frame rate.

[0078] In the synchronization delay measurement method provided by this scheme, when video synchronization is achieved by synchronizing the audio played by the first terminal device with the video frame played by the second terminal device, the currently played video frame of the second terminal device can be captured to obtain a captured image. The frame number (i.e., the second frame number) of the video frame played by the second terminal device can then be obtained based on the captured image. Then, the target audio played by the first terminal device at the same time can be acquired, and the frame number (i.e., the first frame number) of the corresponding video frame can be determined based on the frequency of the target audio. The synchronization delay can then be determined based on the difference in frame count between the two video frames and the frame rate of the first video.

[0079] In another possible implementation of the second aspect, each video frame of the first video is provided with a corresponding video identifier and audio, wherein the video identifier includes a frame number and a frame rate, and the audio frequencies corresponding to each video frame are different.

[0080] When the video synchronization is synchronized between the audio played by the first terminal device and the video frame played by the second terminal device, the captured image is the image corresponding to the second terminal device, and the captured image includes the second video frame currently being played by the second terminal device.

[0081] The first terminal device determines the synchronization delay in video synchronization based on the target audio and / or the captured image, which may include:

[0082] The first terminal device acquires the second playback time of the second video frame and determines the second frame number of the second video frame based on the video identifier in the captured image;

[0083] The first terminal device determines the second audio corresponding to the second frame sequence number and obtains the first playback time of the second audio played by the first terminal device;

[0084] The first terminal device determines the synchronization delay for video synchronization between the first terminal device and the second terminal device based on the first playback time and the second playback time.

[0085] In the synchronization delay measurement method provided by this scheme, when video synchronization is achieved by synchronizing the audio played by the first terminal device with the video frame played by the second terminal device, the currently played video frame of the second terminal device can be captured to obtain an image, and the playback time of the currently played video frame of the second terminal device can be obtained. Then, the second frame sequence number of the video frame played by the second terminal device can be obtained based on the captured image, and the playback time of the video frame with the second frame sequence number played by the first terminal device can be obtained, so as to determine the synchronization delay based on the time difference between the two playback times.

[0086] In one possible implementation of the second aspect, each video frame of the first video is provided with a corresponding video identifier and audio, wherein the video identifier includes a frame number and a frame rate, and the frequency of the audio corresponding to each video frame is different from each other.

[0087] When the video synchronization is synchronized with the audio and video frames played by the second terminal device, the target audio is the audio currently played by the second terminal device, the captured image is the image corresponding to the second terminal device, and the captured image includes the second video frame currently played by the second terminal device;

[0088] The first terminal device determines the synchronization delay in video synchronization based on the target audio and / or the captured image, which may include:

[0089] The first terminal device determines the second frame number and frame rate of the second video frame played by the second terminal device based on the video identifier in the captured image;

[0090] The first terminal device determines the first frame number of the first video frame corresponding to the target audio based on the frequency of the target audio.

[0091] The first terminal device determines the synchronization delay for the second terminal device to perform video synchronization based on the first frame sequence number, the second frame sequence number, and the frame rate.

[0092] In the synchronization delay measurement method provided by this scheme, when video synchronization is achieved by synchronizing the audio and video frames played by the second terminal device, the currently played video frame of the second terminal device can be captured to obtain a captured image, and the currently played audio of the second terminal device can be recorded to obtain the target audio. Then, the second video frame currently played by the second terminal device can be determined based on the video identifier of the captured image, and the first video frame corresponding to the target audio can be determined based on the frequency of the target audio. The number of frames between the first and second video frames can be determined based on the first frame number and the second frame number, thereby determining the synchronization delay based on the number of frames and the frame rate corresponding to the first video.

[0093] Thirdly, embodiments of this application provide a synchronization delay measurement method, applied to measuring synchronization delay in audio synchronization, the method including:

[0094] Establish network connections between the first terminal device and each of the second terminal devices based on the first connection information;

[0095] Each first audio is transmitted from the first terminal device to each of the second terminal devices. Each first audio is the same and is time-calibrated in the same time interval using a preset method.

[0096] Acquire a second audio, which includes the audio played by each of the second terminal devices;

[0097] The playback time of the same time interval in each of the second terminal devices is determined based on the second audio, and the synchronization delay in audio synchronization is determined based on each playback time.

[0098] Specifically, each of the first audio frequencies is time-calibrated within the same time interval by means of frequency or frequency combination.

[0099] Using the aforementioned synchronization delay measurement method, in audio synchronization, the first terminal device can project each first audio signal to a corresponding second terminal device, and record the audio played on each second terminal device to obtain the second audio signal. Then, based on the second audio signal, the playback time of the time interval for time calibration on each second terminal device can be determined, thereby determining the synchronization delay of the audio synchronization.

[0100] Fourthly, embodiments of this application provide a content synchronization method applied to a first terminal device, the method including:

[0101] The first terminal device determines the connection information between the first terminal device and the second terminal device;

[0102] The first terminal device acquires shared content and determines the synchronization delay for synchronous playback based on the shared content and the connection information. The synchronization delay is obtained by any one of the methods described in the first, second, and third aspects above.

[0103] The first terminal device sends the shared content to the second terminal device, and controls the playback of the content on the first terminal device according to the synchronization delay, and / or controls the playback of the shared content on the second terminal device.

[0104] Using the content synchronization method described above, when a first terminal device shares content from its own device to a second terminal device for synchronized playback, the first terminal device can obtain the shared content and the corresponding second terminal device, and determine the connection information between the two devices. Then, based on the shared content and the connection information, the synchronization delay for synchronized playback can be determined. This delay allows for control of content playback on the first and / or second terminal devices, ensuring synchronization in multi-screen interaction, improving playback quality, and enhancing user experience. This method is highly user-friendly and practical.

[0105] For example, the first terminal device sending the shared content to the second terminal device, and controlling the playback of the content on the first terminal device according to the synchronization delay, and / or controlling the playback of the shared content on the second terminal device, may include:

[0106] The first terminal device sends the shared content, the synchronization delay, and a control instruction to the second terminal device. The control instruction is used to instruct the second terminal device to control the playback of the shared content according to the synchronization delay.

[0107] For example, the first terminal device sending the shared content to the second terminal device, and controlling the playback of the content on the first terminal device according to the synchronization delay, and / or controlling the playback of the shared content on the second terminal device, may include:

[0108] The first terminal device determines the playback time of the second terminal device based on the synchronization delay, and sends the shared content to the second terminal device during the playback time.

[0109] Fifthly, embodiments of this application provide a synchronization delay measurement device, applied to a first terminal device, for measuring synchronization delay during screen synchronization. The device may include:

[0110] A network connection establishment module is used to establish a network connection between the first terminal device and the second terminal device based on the first connection information.

[0111] The screen mirroring module is used to play a first video and project video frames from the first video to the second terminal device.

[0112] The image acquisition module is used to acquire captured images, wherein the captured images include a second video frame currently being played by the second terminal device, and the second video frame is a video frame in the first video;

[0113] The synchronization delay determination module is used to determine the synchronization delay for the first terminal device and the second terminal device to synchronize their images based on the captured images.

[0114] In one possible implementation of the fifth aspect, the image acquisition module is used to acquire a captured image corresponding to the second terminal device through the camera of the first terminal device.

[0115] For example, the synchronization delay determination module is used to determine the second playback time of the second video frame based on the captured image; obtain the first playback time of the first video frame played by the first terminal device at the same time, wherein the first playback time and the second playback time are the playback times of the video frame in the first video, and the same time is the moment when the second terminal device plays the second video frame; and determine the synchronization delay for the first terminal device and the second terminal device to synchronize the images based on the first playback time and the second playback time.

[0116] For example, each video frame of the first video is provided with a corresponding video identifier, and the video identifier includes a frame number;

[0117] The synchronization delay determination module is further configured to determine the second playback time of the second video frame based on the captured image, and determine the second frame number of the second video frame based on the video identifier in the captured image; obtain the first playback time of the video frame with the second frame number played by the first terminal device; and determine the synchronization delay for the first terminal device and the second terminal device to synchronize the images based on the first playback time and the second playback time.

[0118] Wherein, the first playback time and the second playback time are the system time of the first terminal device or the second terminal device.

[0119] For example, each video frame of the first video is provided with a corresponding video identifier, which includes a frame number and a frame rate;

[0120] The synchronization delay determination module is further configured to obtain the second frame number and frame rate of the second video frame based on the video identifier in the captured image, and obtain the first frame number of the first video frame played by the first terminal device at the same time, wherein the same time is the moment when the second terminal device plays the second video frame; and determine the synchronization delay for the first terminal device and the second terminal device to synchronize the images based on the first frame number, the second frame number and the frame rate.

[0121] Specifically, the synchronization delay determination module obtains the first frame sequence number of the first video frame played by the first terminal device at the same time based on the video identifier in the first video frame.

[0122] In another possible implementation of the fifth aspect, the image acquisition module is further configured to acquire the captured image through the camera of a third terminal device, wherein the third terminal device is a terminal device other than the first terminal device and the second terminal device, and the captured image also includes a first video frame currently being played by the first terminal device.

[0123] For example, the synchronization delay determination module is further configured to determine the first playback time of the first video frame and the second playback time of the second video frame based on the captured image, wherein the first playback time and the second playback time are the playback times of the video frames in the first video; and to determine the synchronization delay for the first terminal device and the second terminal device to synchronize their images based on the first playback time and the second playback time.

[0124] For example, each video frame of the first video is provided with a corresponding video identifier, which includes a frame number and a frame rate;

[0125] The synchronization delay determination module is further configured to determine the first frame number and frame rate of the first video frame based on the first video identifier in the captured image, and determine the second frame number of the second video frame based on the second video identifier in the captured image; and determine the synchronization delay for the first terminal device and the second terminal device to synchronize the images based on the first frame number, the second frame number and the frame rate.

[0126] For example, the captured images include multiple images. The synchronization delay determination module is further configured to determine the second playback time of the second video frame based on the first captured image, and determine the second frame number of the second video frame based on the video identifier in the first captured image, wherein the first captured image is any one of the captured images; determine the second captured image based on the second frame number, wherein the second captured image includes the video frame with the second frame number played by the first terminal device; determine the first playback time of the video frame with the second frame number played by the first terminal device based on the second captured image; and determine the synchronization delay for screen synchronization between the first terminal device and the second terminal device based on the first playback time and the second playback time.

[0127] Wherein, the first playback time and the second playback time are the system time of the first terminal device or the second terminal device.

[0128] It is understood that the first connection information may include network type and network topology. The video identifier may be a QR code.

[0129] Sixthly, embodiments of this application provide a synchronization delay measurement device, applied to a first terminal device, for measuring synchronization delay in video synchronization. The device may include:

[0130] A network connection establishment module is used to establish a network connection between the first terminal device and the second terminal device based on the first connection information.

[0131] The screen mirroring module is used to play a first video and project the audio and / or video frames from the first video to the second terminal device.

[0132] The synchronization delay determination module is used to acquire target audio and / or captured images, and determine the synchronization delay in video synchronization based on the target audio and / or captured images.

[0133] In one possible implementation of the sixth aspect, each video frame of the first video is provided with a corresponding audio, and the frequencies of the audio corresponding to each video frame are different from each other.

[0134] When video synchronization is achieved by synchronizing the video frames played by the first terminal device with the audio played by the second terminal device, the target audio is the audio currently played by the second terminal device.

[0135] The synchronization delay determination module is used to obtain the frequency of the target audio and determine the second frame number of the second video frame corresponding to the target audio based on the frequency of the target audio; obtain the first frame number of the first video frame played by the first terminal device at the same time, wherein the same time is the time when the second terminal device plays the target audio; and determine the synchronization delay for video synchronization between the first terminal device and the second terminal device based on the first frame number, the second frame number and the frame rate of the first video.

[0136] In another possible implementation of the sixth aspect, each video frame of the first video is provided with a corresponding audio, and the frequencies of the audio corresponding to each video frame are different from each other.

[0137] When video synchronization is achieved by synchronizing the video frames played by the first terminal device with the audio played by the second terminal device, the target audio is the audio currently played by the second terminal device.

[0138] The synchronization delay determination module is used to obtain the frequency of the target audio and the second playback time of the target audio played by the second terminal device; determine the second frame number of the second video frame corresponding to the target audio according to the frequency of the target audio, and obtain the first playback time of the video frame with the second frame number played by the first terminal device; and determine the synchronization delay for video synchronization between the first terminal device and the second terminal device according to the first playback time and the second playback time.

[0139] Wherein, the first playback time and the second playback time are the system time of the first terminal device.

[0140] In one possible implementation of the sixth aspect, each video frame of the first video is provided with a corresponding video identifier and audio, wherein the video identifier includes a frame number and a frame rate, and the frequency of the audio corresponding to each video frame is different from each other.

[0141] When the video synchronization is synchronized between the audio played by the first terminal device and the video frame played by the second terminal device, the captured image is the image corresponding to the second terminal device, and the captured image includes the second video frame currently being played by the second terminal device.

[0142] The synchronization delay determination module is used to determine the second frame number and frame rate of the second video frame based on the video identifier in the captured image; obtain the frequency of the first audio played by the first terminal device at the same time, and determine the first frame number of the first video frame corresponding to the frequency of the first audio; and determine the synchronization delay for video synchronization between the first terminal device and the second terminal device based on the first frame number, the second frame number and the frame rate.

[0143] In another possible implementation of the sixth aspect, each video frame of the first video is provided with a corresponding video identifier and audio, wherein the video identifier includes a frame number and a frame rate, and the frequency of the audio corresponding to each video frame is different from each other.

[0144] When the video synchronization is synchronized between the audio played by the first terminal device and the video frame played by the second terminal device, the captured image is the image corresponding to the second terminal device, and the captured image includes the second video frame currently being played by the second terminal device.

[0145] The synchronization delay determination module is used to obtain the second playback time of the second video frame and determine the second frame number of the second video frame according to the video identifier in the captured image; determine the second audio corresponding to the second frame number and obtain the first playback time of the first terminal device playing the second audio; and determine the synchronization delay for video synchronization between the first terminal device and the second terminal device according to the first playback time and the second playback time.

[0146] In one possible implementation of the sixth aspect, each video frame of the first video is provided with a corresponding video identifier and audio, wherein the video identifier includes a frame number and a frame rate, and the frequency of the audio corresponding to each video frame is different from each other.

[0147] When the video synchronization is synchronized with the audio and video frames played by the second terminal device, the target audio is the audio currently played by the second terminal device, the captured image is the image corresponding to the second terminal device, and the captured image includes the second video frame currently played by the second terminal device;

[0148] The synchronization delay determination module is used to determine the second frame number and frame rate of the second video frame played by the second terminal device based on the video identifier in the captured image; determine the first frame number of the first video frame corresponding to the target audio based on the frequency of the target audio; and determine the synchronization delay of the second terminal device for video synchronization based on the first frame number, the second frame number, and the frame rate.

[0149] Seventhly, embodiments of this application provide a synchronization delay measurement device for measuring synchronization delay in audio synchronization. The device may include:

[0150] A network connection establishment module is used to establish a network connection between the first terminal device and each of the second terminal devices based on the first connection information.

[0151] The delivery module is used to deliver each first audio to each of the second terminal devices through the first terminal device. Each first audio is the same and each first audio is time-calibrated in the same time interval by a preset method.

[0152] The second audio acquisition module is used to acquire second audio, which includes the audio played by each of the second terminal devices;

[0153] The synchronization delay determination module is used to determine the playback time of the same time interval in each of the second terminal devices based on the second audio, and to determine the synchronization delay in audio synchronization based on each playback time.

[0154] Specifically, each of the first audio frequencies is time-calibrated within the same time interval by means of frequency or frequency combination.

[0155] Eighthly, embodiments of this application provide a content synchronization device applied to a first terminal device, the device including:

[0156] A connection information determination module is used to determine the connection information between the first terminal device and the second terminal device;

[0157] A synchronization delay determination module is used to acquire shared content and determine the synchronization delay for synchronous playback based on the shared content and the connection information, wherein the synchronization delay is obtained by any one of the methods in the first aspect, the second aspect and the third aspect described above;

[0158] The playback control module is used to send the shared content to the second terminal device, control the playback of the content on the first terminal device according to the synchronization delay, and / or control the playback of the shared content on the second terminal device.

[0159] For example, the playback control module is used to send the shared content, the synchronization delay, and control instructions to the second terminal device, wherein the control instructions are used to instruct the second terminal device to control the playback of the shared content according to the synchronization delay.

[0160] For example, the playback control module is used to determine the playback time of the second terminal device based on the synchronization delay, and send the shared content to the second terminal device during the playback time.

[0161] Ninthly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it causes the terminal device to implement the method described in any one of the first, second, third, and fourth aspects above.

[0162] In a tenth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a computer, causes the computer to implement the method described in any one of the first, second, third, and fourth aspects above.

[0163] In one aspect, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the method described in any one of the first, second, third, and fourth aspects. Attached Figure Description

[0164] Figure 1 This is a schematic diagram of the structure of a terminal device to which the synchronization delay measurement method or content synchronization method provided in an embodiment of this application is applicable;

[0165] Figure 2 This is a schematic diagram of the software architecture to which the synchronization delay measurement method or content synchronization method provided in an embodiment of this application is applicable;

[0166] Figure 3 This is a schematic diagram of a communication system provided in an embodiment of this application;

[0167] Figure 4 This is a schematic diagram of an application scenario for screen projection provided in an embodiment of this application;

[0168] Figure 5 This is a schematic diagram of an application scenario for simultaneously projecting video and audio from a video, provided by an embodiment of this application.

[0169] Figure 6 This is a schematic diagram of an application scenario for audio delivery provided in an embodiment of this application;

[0170] Figure 7 and Figure 8 This is an example diagram of the content synchronization function provided in one embodiment of this application;

[0171] Figure 9 and Figure 10 This is a schematic diagram of an application scenario for performing synchronization delay measurement provided in an embodiment of this application;

[0172] Figure 11 This is a schematic diagram of an application scenario for performing synchronization delay measurement provided in another embodiment of this application;

[0173] Figures 12 to 14 This is a schematic flowchart of a synchronization delay measurement method provided in an embodiment of this application;

[0174] Figure 15 This is a flowchart illustrating a content synchronization method provided in an embodiment of this application. Detailed Implementation

[0175] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0176] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0177] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0178] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0179] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0180] Furthermore, the term "multiple" mentioned in the embodiments of this application should be interpreted as two or more.

[0181] The steps involved in the synchronization delay measurement method and content synchronization method provided in the embodiments of this application are merely examples. Not all steps are mandatory, nor are all contents in each piece of information or message required. They can be added or removed as needed during use. The same step or step or message with the same function in the embodiments of this application can be referenced and learned from each other in different embodiments.

[0182] The business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0183] Currently, terminal devices can share content through multi-screen interaction, enabling synchronized playback to facilitate user browsing or improve playback quality. Multi-screen interaction can take several forms: screen mirroring (e.g., a mobile phone) can mirror its screen to one or more second-terminal devices (e.g., tablets), sharing the screen between them for easier browsing; audio and video mirroring (e.g., video mirroring) can mirror both audio and video from a video to a second-terminal device (e.g., a smart TV), enhancing playback; audio-only mirroring (e.g., a mobile phone can mirror audio from a video to a second-terminal device (e.g., a smart speaker), coordinating playback for improved video quality; or audio mirroring (e.g., audio-only mirroring) can mirror audio from a video to a second-terminal device (e.g., a smart speaker), using the phone and speaker to enhance playback; or audio mirroring can mirror audio to multiple second-terminal devices (e.g., smart speakers), creating stereo sound and improving overall audio quality.

[0184] When engaging in multi-screen interaction, the first terminal device needs to transmit shared content to the second terminal device via communication networks such as Bluetooth or Wireless Fidelity (WiFi). Alternatively, the first terminal device may need to encode the shared content before transmitting the encoded content to the second terminal device via Bluetooth or WiFi. Upon receiving the shared content, the second terminal device needs to decode it before playback. However, because encoding, transmission, and decoding operations can introduce latency, when the shared content is dynamic (i.e., changes over time), it can easily cause asynchrony between content sharing and collaborative playback, impacting the user experience.

[0185] For example, when a mobile phone projects a video onto one or more tablets for synchronized display using screen mirroring technologies like Miracast, the phone needs to encode the video, for instance, into H.264 format data, and then send this H.264 data to the tablets via Bluetooth or Wi-Fi. The tablet, upon receiving the H.264 data, needs to decode it and then display the decoded image. Because encoding, transmission, and decoding operations can introduce latency, there will be a time difference between the video displayed on the tablet and the video displayed on the phone, causing them to be out of sync and negatively impacting the user experience.

[0186] For example, when a mobile phone uses screen mirroring technology like Miracast to project the audio from a video played on the phone to a smart speaker for collaborative video playback, the phone needs to encode the audio in the video, for example, into AAC format data, and then transmit the AAC format data to the smart speaker via a communication network such as Bluetooth or Wi-Fi. After receiving the AAC format data from the phone, the smart speaker needs to decode the AAC format data and then play the decoded audio. Because encoding, transmission, and decoding operations can introduce latency, there will be a time difference between the audio played by the smart speaker and the video played on the phone. This causes the audio played by the smart speaker to lag behind the video played on the phone, affecting the collaborative video playback effect and impacting the user experience.

[0187] For example, when a mobile phone casts a video to a smart TV using screen mirroring technologies like Miracast to improve playback on a larger screen, the phone needs to encode the audio and video separately. For instance, the video is encoded into H.264 format data, and the audio into AAC format data. These are then sent to the smart TV via Bluetooth or Wi-Fi. Upon receiving the H.264 and AAC data, the smart TV needs to decode them separately before playing the resulting audio and video. Because the audio decoding speed and video decoding speed may differ, a time lag exists between the audio and video playback, causing desynchronization and affecting the video playback quality and user experience.

[0188] For example, when a mobile phone uses screen mirroring technology like Miracast to project a video onto a smart TV and simultaneously projects the audio onto a smart speaker for playback, enhancing the video's viewing experience through coordinated playback on a large-screen device and a smart speaker, the phone needs to encode the audio and video separately. For instance, the video is encoded into H.264 format data, and the audio into AAC format data. The H.264 data is then sent to the smart TV via Bluetooth or Wi-Fi, and the AAC data is sent to the smart speaker via the same network. The smart TV, upon receiving the H.264 data, decodes it and displays the resulting video. Similarly, the smart speaker, upon receiving the AAC data, decodes it and plays the resulting audio. Because the audio decoding speed of a smart speaker may differ from the video decoding speed of a smart TV, and / or because the transmission speed between a mobile phone and a smart TV may differ from the transmission speed between a mobile phone and a smart speaker, there will be a time difference between the video played on the smart TV and the audio played on the smart speaker. This will cause the audio and video to be out of sync when the smart TV and smart speaker play videos together, affecting the user experience.

[0189] For example, when a mobile phone transmits audio to multiple smart speakers via Bluetooth or WiFi to create stereo sound through synchronized audio playback, the network connections between the phone and the different smart speakers may vary, resulting in different latency for each smart speaker. Consequently, there will be a time difference between the audio played by each smart speaker, affecting the stereo effect of synchronized audio playback and impacting the user experience.

[0190] To address the aforementioned issues, this application provides a synchronization latency measurement method and a content synchronization method. These methods can accurately measure the synchronization latency in screen synchronization, video synchronization, and audio synchronization when a first terminal device and a second terminal device are connected to a network. This allows the first terminal device to obtain the shared content and the corresponding second terminal device when sharing content to the second terminal device for synchronized playback. It also allows for the determination of the connection information between the first and second terminal devices. Based on the shared content and the connection information, the synchronization latency can be determined, thereby controlling the content playback on the first and / or second terminal devices. This ensures synchronization of content playback in multi-screen interaction, improves playback quality, and enhances user experience. The method is highly user-friendly and practical.

[0191] The synchronization delay measurement method and content synchronization method provided in this application embodiment can be applied to terminal devices, which can be mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), desktop computers, smart TVs, smart speakers, etc. This application embodiment does not impose any restrictions on the specific type of terminal device.

[0192] For example, Figure 1A schematic diagram of a terminal device 100 is shown. The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0193] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0194] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0195] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0196] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0197] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0198] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thereby realizing the touch function of the terminal device 100.

[0199] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface to enable the function of answering phone calls through a Bluetooth headset.

[0200] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0201] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to enable music playback through Bluetooth headphones.

[0202] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to enable the shooting function of the terminal device 100. The processor 110 and the display screen 194 communicate via the DSI interface to enable the display function of the terminal device 100.

[0203] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0204] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge terminal device 100, and can also be used for data transfer between terminal device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other terminal devices, such as AR devices.

[0205] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may also adopt different interface connection methods or a combination of multiple interface connection methods as described in the above embodiments.

[0206] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the terminal device 100. While charging the battery 142, the charging management module 140 can also supply power to the terminal device via the power management module 141.

[0207] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0208] The wireless communication function of the terminal device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0209] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0210] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the terminal device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0211] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0212] The wireless communication module 160 can provide solutions for wireless communication applications on the terminal device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0213] In some embodiments, antenna 1 of terminal device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling terminal device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0214] Terminal device 100 implements display functions through a GPU, display screen 194, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0215] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, terminal device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0216] Terminal device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0217] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0218] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the terminal device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0219] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when terminal device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0220] Video codecs are used to compress or decompress digital video. Terminal device 100 may support one or more video codecs. Thus, terminal device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0221] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in terminal devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0222] The external storage interface 120 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external storage card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external storage card.

[0223] Internal memory 121 can be used to store computer executable program code, which includes instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of terminal device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of terminal device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory located in the processor.

[0224] Terminal device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0225] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0226] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The terminal device 100 can listen to music or make hands-free calls through the speaker 170A.

[0227] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the terminal device 100 answers a phone call or voice message, the receiver 170B can be brought close to the listener's ear to hear the voice.

[0228] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Terminal device 100 may be equipped with at least one microphone 170C. In some embodiments, terminal device 100 may be equipped with two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, terminal device 100 may be equipped with three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0229] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0230] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Terminal device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, terminal device 100 detects the intensity of the touch operation based on pressure sensor 180A. Terminal device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example: when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0231] The gyroscope sensor 180B can be used to determine the motion attitude of the terminal device 100. In some embodiments, the gyroscope sensor 180B can determine the angular velocity of the terminal device 100 around three axes (i.e., the x, y, and z axes). The gyroscope sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the terminal device 100's shake, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the terminal device 100 through reverse movement, thus achieving image stabilization. The gyroscope sensor 180B can also be used in navigation and motion-sensing game scenarios.

[0232] The barometric pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal device 100 calculates altitude using the air pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0233] The magnetic sensor 180D includes a Hall sensor. The terminal device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover. In some embodiments, when the terminal device 100 is a flip phone, the terminal device 100 can detect the opening and closing of the flip cover using the magnetic sensor 180D. Then, based on the detected opening and closing state of the cover or the flip cover, features such as automatic flip unlocking can be set.

[0234] The 180E accelerometer can detect the magnitude of acceleration of the terminal device 100 in various directions (typically three axes). When the terminal device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the attitude of the terminal device, and can be applied to applications such as landscape / portrait switching and pedometers.

[0235] A distance sensor 180F is used to measure distance. The terminal device 100 can measure distance via infrared or laser. In some embodiments, during a shooting scene, the terminal device 100 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0236] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The terminal device 100 emits infrared light outward through the LED. The terminal device 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 100. When insufficient reflected light is detected, the terminal device 100 can determine that there is no object near the terminal device 100. The terminal device 100 may use the proximity sensor 180G to detect when a user holds the terminal device 100 close to their ear for a call, so as to automatically turn off the screen to save power. The proximity sensor 180G can also be used in holster mode and pocket mode for automatic unlocking and screen locking.

[0237] The ambient light sensor 180L is used to sense the ambient light intensity. The terminal device 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light intensity. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also work with the proximity sensor 180G to detect whether the terminal device 100 is in a pocket to prevent accidental touches.

[0238] The fingerprint sensor 180H is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0239] Temperature sensor 180J is used to detect temperature. In some embodiments, terminal device 100 uses the temperature detected by temperature sensor 180J to execute a temperature handling strategy. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, terminal device 100 reduces the performance of the processor located near temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is below another threshold, terminal device 100 heats battery 142 to prevent abnormal shutdown of terminal device 100 due to low temperature. In still other embodiments, when the temperature is below yet another threshold, terminal device 100 boosts the output voltage of battery 142 to prevent abnormal shutdown due to low temperature.

[0240] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of terminal device 100, in a different position than display screen 194.

[0241] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire vibration signals from the vibrating bone segments of the human vocal cords. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure signals. In some embodiments, the bone conduction sensor 180M can also be incorporated into headphones to form bone conduction headphones. The audio module 170 can parse the voice signals from the vibrating bone segments of the vocal cords acquired by the bone conduction sensor 180M to realize voice functionality. The application processor can parse heart rate information from the blood pressure signals acquired by the bone conduction sensor 180M to realize heart rate detection functionality.

[0242] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Terminal device 100 can receive button input and generate key signal inputs related to user settings and function control of terminal device 100.

[0243] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0244] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0245] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the terminal device 100. The terminal device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the terminal device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100.

[0246] The software system of terminal device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of terminal device 100.

[0247] Figure 2 This is a software structure block diagram of the terminal device 100 according to an embodiment of this application.

[0248] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0249] The application layer can include a series of application packages.

[0250] like Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0251] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0252] like Figure 2 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0253] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0254] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0255] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0256] The phone manager is used to provide communication functions for terminal device 100. For example, it manages call status (including connection, hang-up, etc.).

[0257] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0258] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of download completion or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating the device, and flashing indicator lights.

[0259] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0260] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0261] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0262] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0263] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0264] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0265] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0266] A 2D graphics engine is a graphics engine for 2D drawing.

[0267] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0268] The following example, using a scene of capturing a photograph, illustrates the workflow of the software and hardware of the terminal device 100.

[0269] When touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, etc.). The raw input event is stored in the kernel layer. The application framework layer retrieves the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking a touch click as an example, where the corresponding control is the camera application icon, the camera application calls the application framework layer's interface to launch the camera application, and then calls the kernel layer to launch the camera driver, capturing still images or videos through camera 193.

[0270] The content synchronization method provided in this application embodiment will be described in detail below with reference to the accompanying drawings and specific application scenarios.

[0271] Figure 3 This is a schematic diagram of a communication system provided in an embodiment of this application. For example... Figure 3 As shown, the content synchronization method provided in this application embodiment can be applied to a communication system 300, which may include a first terminal device 301 and at least one second terminal device 302. Figure 3(One is shown in the image). The first terminal device 301 can connect to the second terminal device 302 via a communication network. The communication network can be a local area network such as Bluetooth, WiFi, ZigBee, or NFC, or a wide area network such as 2G, 3G, 4G, 5G, the future evolved public land mobile network (PLMN), or the Internet.

[0272] In this context, the first terminal device 301 refers to the terminal device that shares content. For example, the first terminal device 301 can be a mobile phone, tablet computer, wearable device, in-vehicle device, AR / VR device, laptop computer, UMPC, netbook, PDA, desktop computer, or other terminal device. The second terminal device 302 refers to the terminal device that can receive content shared by the first terminal device 301. For example, the second terminal device 302 can be a mobile phone, tablet computer, wearable device, in-vehicle device, AR / VR device, laptop computer, UMPC, netbook, PDA, desktop computer, smart TV, smart speaker, or other terminal device.

[0273] After the first terminal device 301 establishes a connection with the second terminal device 302 through a communication network, the first terminal device 301 can share its content to the second terminal device 302 for playback. However, during content sharing, operations such as encoding, transmission, or decoding are required, and these operations can easily introduce latency, affecting the synchronization of content playback and impacting user experience.

[0274] In this embodiment, when the first terminal device 301 shares content to the second terminal device 302 for playback, the first terminal device 301 can first obtain the shared content and the corresponding second terminal device. Then, the first terminal device can determine the connection information between the first terminal device 301 and the second terminal device 302, and determine the synchronization delay for synchronous playback based on the shared content and the connection information between the first terminal device 301 and the second terminal device 302. This allows the first terminal device 301 to control content playback based on the synchronization delay, and / or the second terminal device 302 to control content playback synchronization, thereby improving user experience.

[0275] Synchronized playback can refer to the synchronization of video feeds played on two or more terminal devices, the synchronization of audio feeds played on two or more terminal devices, or the synchronization of video feeds and audio feeds when one or more terminal devices collaboratively play the same video. Therefore, shared content can be video and / or audio.

[0276] In one example, the shared content can be all or part of the content being played on the first terminal device 301. For example, when the first terminal device 301 is playing a video, the shared content can be the video footage and audio, or it can be either the video footage or the audio. In another example, the shared content can be content stored on the first terminal device 301. That is, the first terminal device 301 can directly share content stored on the first terminal device 301 but not played on the first terminal device 301 to one or more second terminal devices 302 for synchronous playback. For example, the first terminal device 301 can directly send audio stored on the first terminal device 301 to smart speaker A, smart speaker B, and smart speaker C for playback.

[0277] It is understandable that the second terminal device 302 corresponding to the shared content can include one or more. For example, when the first terminal device 301 projects both the video and audio from a video played on the first terminal device 301 onto a smart TV for playback, the shared content can include both video and audio. The second terminal device corresponding to the shared content can only include a smart TV; that is, the second terminal devices corresponding to both the video and audio in the video are smart TVs. For example, when the first terminal device 301 projects the video from a video played on the first terminal device 301 onto a smart TV for playback and projects the audio from the video onto a smart speaker for playback, the shared content can include both video and audio. The second terminal devices corresponding to the shared content can include both a smart TV and a smart speaker; that is, the second terminal device corresponding to the video video is a smart TV, and the second terminal device corresponding to the audio from the video is a smart speaker. For example, when the first terminal device 301 projects the audio stored on the first terminal device 301 onto smart speaker A and smart speaker B for playback, the shared content can include audio, and the second terminal devices corresponding to the shared content can include smart speaker A and smart speaker B.

[0278] In this embodiment, the connection information may include the network type and network topology of the network connecting the first terminal device 301 and the second terminal device 302. The network type may include Bluetooth, WiFi, ZigBee, etc. The network topology characterizes the connection method of the network connecting the first terminal device 301 and the second terminal device 302. For example, when the first terminal device 301 and the second terminal device 302 are directly connected via WiFi, the network topology between them can be: first terminal device 301 → second terminal device 302. For example, when the first terminal device 301 is connected to the second terminal device 302 through one or more routing nodes, such as through routing node A and routing node B, the network topology between them can be: first terminal device 301 → routing node A → routing node B → second terminal device 302.

[0279] It should be noted that the synchronization delay of synchronized playback can be the time that the shared content played by the second terminal device lags behind the content played by the first terminal device, or it can be the time that one shared content played by the second terminal device lags behind another shared content played by the second terminal device, or it can be the time that one shared content played by one second terminal device lags behind another shared content played by a different second terminal device.

[0280] For example, when a first terminal device (such as a mobile phone) projects the screen of a video being played on the mobile phone onto a second terminal device (such as a tablet computer) to achieve screen synchronization between the mobile phone and the tablet computer, the screen played on the tablet computer will lag behind the screen played on the mobile phone due to encoding, transmission, or decoding operations. In this case, the synchronization delay of the synchronized playback can be the time that the screen played on the tablet computer lags behind the screen played on the mobile phone.

[0281] For example, when a first terminal device (such as a mobile phone) sends audio stored on the phone to a second terminal device (such as smart speaker A, smart speaker B, and smart speaker C) for playback to achieve synchronized audio playback among smart speaker A, smart speaker B, and smart speaker C, there will be a time difference between the audio played by smart speaker A, smart speaker B, and smart speaker C. Assuming that the audio played by smart speaker C and smart speaker B are both delayed compared to the audio played by smart speaker A, the synchronization delay of synchronized playback can include a first synchronization delay and a second synchronization delay. The first synchronization delay can be the time that the audio played by smart speaker B is delayed compared to the audio played by smart speaker A, and the second synchronization delay can be the time that the audio played by smart speaker C is delayed compared to the audio played by smart speaker A.

[0282] For example, when a first terminal device (such as a mobile phone) projects both the video and audio from the mobile phone onto a second terminal device (such as a smart TV) for playback, there will be a time difference between the video and audio played on the smart TV. Suppose that the audio played on the smart TV lags behind the video played on the smart TV. In this case, the synchronization delay of the synchronous playback can be the time that the audio played on the smart TV lags behind the video played on the smart TV.

[0283] In this embodiment, after the first terminal device 301 obtains the synchronization delay of the synchronized playback, it can add a preset time delay to the content played by the first terminal device 301 or control the currently played content of the first terminal device 301 to repeat playback for a preset time, and / or add a preset time delay to the content played by the second terminal device 302 or control the currently played content of the second terminal device 302 to repeat playback for a preset time, so as to achieve content synchronization. The preset time can be determined based on the synchronization delay. For example, when there is only one synchronization delay, the synchronization delay can be directly determined as the preset time; when there are multiple synchronization delays, the largest synchronization delay can be used as the base delay, and then each preset time can be determined based on each synchronization delay and the base delay.

[0284] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating an application scenario for screen projection provided in an embodiment of this application. For example... Figure 4 As shown in (a), when a first terminal device (such as a mobile phone) projects the screen currently playing on the mobile phone to a second terminal device (such as a tablet computer) for synchronized playback, the projected screen needs to undergo encoding, transmission, and other operations before reaching the tablet computer. The tablet computer then decodes the screen before playback, resulting in a lag between the screen played on the tablet and the mobile phone. Therefore, to ensure synchronization between the screen played on the tablet and the mobile phone, the mobile phone can obtain the synchronization delay T1 based on the connection information between the mobile phone and the tablet computer and the shared content (i.e., the screen). Then, the mobile phone can either add a delay of T1 to the screen played on the mobile phone or repeat the currently playing screen for a time T1, ensuring that the screen played on the tablet and the mobile phone are synchronized. Figure 4 Synchronization is shown in (b) in the diagram. Here, repeating the playback for time T1 means adding T1 time to the original playback time of the scene, so that the playback time of the scene is (original playback time + T1).

[0285] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating an application scenario for simultaneously projecting video and audio from an embodiment of this application. Figure 5As shown in (a), when a first terminal device (such as a mobile phone) projects both the audio and video from a video being played on the mobile phone to a second terminal device (such as a smart TV) for playback, the audio in the video undergoes audio encoding and transmission to reach the smart TV, and is then decoded and played by the smart TV. Simultaneously, the video in the video undergoes video encoding and transmission to reach the smart TV, and is then decoded and played by the smart TV. At this time, asynchrony may occur between the video and audio played on the smart TV; for example, the video may lag behind the audio, or vice versa. Therefore, to ensure synchronization between the audio and video played on the smart TV, the mobile phone can obtain the synchronization delay T2 between the audio and video in the video played on the smart TV based on the connection information and shared content (i.e., audio and video) between the mobile phone and the smart TV. Then, the mobile phone can control the video playback or audio playback of the smart TV based on T2, so that the video and audio played on the smart TV are synchronized as follows: Figure 5 Synchronization is shown in (b) of the diagram. Wherein, Figure 5 The black boxes in the image can represent video frames, and the white boxes can represent the audio corresponding to the video frame, or vice versa.

[0286] Specifically, when T2 is the time lag between the audio played by the smart TV and the video played by the smart TV, the mobile phone can control the video playback of the smart TV according to T2, so that the audio played by the smart TV is synchronized with the video playback. For example, the mobile phone can send a control command to the smart TV, instructing the smart TV to add a delay of T2 to the video playback or to repeat the currently playing video playback for T2. The currently playing video can refer to the video that the smart TV is playing when the control command is received. For example, when sending video and audio from a video to the smart TV, the mobile phone can first send the audio from the video, and then send the video at a time interval T2.

[0287] Similarly, when the video playback on the smart TV lags behind the audio playback on the smart TV by a time interval T2, the mobile phone can control the audio playback on the smart TV based on T2. ​​For example, the mobile phone can send a control command to the smart TV, instructing it to add a delay of T2 to the audio playback or to repeat the currently playing audio for a time interval T2. For example, when sending video footage and audio to the smart TV, the mobile phone can first send the video footage, and then send the audio footage after a time interval T2.

[0288] Please seeFigure 6 , Figure 6 This is a schematic diagram illustrating an application scenario for audio delivery provided in an embodiment of this application. For example... Figure 6 As shown in (a), when a first terminal device (such as a mobile phone) transmits audio stored on the phone to a second terminal device (such as smart speaker A, smart speaker B, and smart speaker C) for synchronous playback, there will be a time difference between the audio played by smart speaker A, smart speaker B, and smart speaker C. In this case, the mobile phone can obtain the synchronization delay based on the connection information between the mobile phone and smart speaker A, the connection information between the mobile phone and smart speaker B, the connection information between the mobile phone and smart speaker C, and the shared content (i.e., audio). Assuming that the audio played by smart speaker B and smart speaker C lags behind the audio played by smart speaker A, the synchronization delay can include a first synchronization delay T3 and a second synchronization delay T4. T3 can be the time that the audio played by smart speaker B lags behind the audio played by smart speaker A, and T4 can be the time that the audio played by smart speaker C lags behind the audio played by smart speaker A. Then, the mobile phone can control the audio playback of smart speaker A and smart speaker B, or smart speaker A and smart speaker C, based on the synchronization delays T3 and T4, so that the audio played by smart speaker A, smart speaker B, and smart speaker C can achieve the same effect as... Figure 6 Synchronization is shown in (b) in the diagram.

[0289] Specifically, when T4 > T3, it indicates that the latency of smart speaker C > the latency of smart speaker B > the latency of smart speaker A. To ensure audio synchronization among smart speakers A, B, and C, a latency can be added to the audio playback of smart speakers A and B, using smart speaker C as a reference. Therefore, the phone can control the audio playback of smart speakers A and B based on T3 and T4. Specifically, it can add a first preset time delay to the audio played by smart speaker A, or control smart speaker A to repeat the currently played audio for the first preset time. Similarly, it can add a second preset time delay to the audio played by smart speaker B, or control smart speaker B to repeat the currently played audio for the second preset time. The first preset time can be T4, and the second preset time can be (T4 - T3). For example, when sending audio to smart speakers A, B, and C, the phone can first send audio to smart speaker C, then send audio to smart speaker B at intervals of (T4-T3), and finally send audio to smart speaker A at interval T4. Alternatively, when sending audio to smart speakers A, B, and C simultaneously, the phone can also send a first control command to smart speaker A and a second control command to smart speaker B. The first control command instructs smart speaker A to add a delay of T4 to the audio it is playing, or to repeat the currently playing audio for T4 seconds. The second control command instructs smart speaker B to add a delay of (T4-T3) to the audio it is playing, or to repeat the currently playing audio for (T4-T3) seconds. The currently playing audio can refer to the audio that smart speaker A / smart speaker B is playing at the time the control command is received.

[0290] Similarly, when T3 > T4, it indicates that the latency of smart speaker B > the latency of smart speaker C > the latency of smart speaker A. To ensure audio synchronization among smart speakers A, B, and C, a latency can be added to the audio playback of smart speakers A and C, using smart speaker B as a reference. Therefore, the phone can control the audio playback of smart speakers A and C based on T3 and T4. Specifically, it can add a first preset time delay to the audio played by smart speaker A, or control smart speaker A to repeat the currently played audio for the first preset time. Conversely, it can add a second preset time delay to the audio played by smart speaker C, or control smart speaker C to repeat the currently played audio for the second preset time, thus synchronizing the audio played by smart speakers A, B, and C. In this case, the first preset time can be T3, and the second preset time can be (T3 - T4).

[0291] It is understandable that when the synchronization type is audio synchronization between multiple second terminal devices, the first terminal device 301 can also obtain the relative position between the first terminal device 301 and each second terminal device, and obtain the audio gain according to the relative position. Thus, the gain of each second terminal device can be gradually adjusted according to the audio gain to ensure the volume balance of audio synchronization playback between multiple second terminal devices, improve the stereo playback effect of audio, and enhance the user experience.

[0292] In this embodiment, content synchronization can be an optional function in multi-screen interaction. For example, during multi-screen interaction, the first terminal device 301 can determine whether to use the content synchronization method provided in this embodiment to synchronize content playback based on the specific scenario.

[0293] Specifically, during multi-screen interaction, the first terminal device 301 can acquire the shared content and determine whether to use the content synchronization method provided in this application embodiment for synchronized playback based on the shared content. Specifically, when the shared content is dynamic, the first terminal device 301 can use the content synchronization method provided in this application embodiment for synchronized playback. When the shared content is static, the first terminal device 301 may not use the content synchronization method provided in this application embodiment for synchronized playback.

[0294] For example, the first terminal device 301 may also have a content synchronization function corresponding to the content synchronization method provided in this application embodiment pre-configured for user selection. In multi-screen interaction, if the user selects to activate the content synchronization function, the first terminal device 301 can synchronously play the content according to the content synchronization method provided in this application embodiment.

[0295] Please see Figure 7 and Figure 8 , Figure 7 and Figure 8 An example diagram showing the activation of the content synchronization function is provided. Figure 7 As shown in (a), when the first terminal device 301 is playing content, a screen mirroring button 701 can be displayed on the display interface of the first terminal device 301. When the user clicks the screen mirroring button 701, as shown in (a), Figure 7As shown in (b), a screen mirroring window 702 can pop up in the display interface. The screen mirroring window 702 can display a "Normal Screen Mirroring" button and a "Synchronous Screen Mirroring" button. The "Normal Screen Mirroring" button is used to mirror the content played on the first terminal device 301 to the second terminal device 302 for playback, but it does not need to ensure that the content played on the second terminal device 302 is synchronized with the content played on the first terminal device 301. The "Synchronous Screen Mirroring" button is used to mirror the content played on the first terminal device 301 to the second terminal device 302 for synchronized playback, that is, it needs to ensure that the content played on the second terminal device 302 is synchronized with the content played on the first terminal device 301.

[0296] When a user clicks or touches the "Synchronize Screen Casting" button, such as Figure 7 As shown in (c), a selection window 703 can pop up in the display interface of the first terminal device 301. The selection window 703 can display the shared content, the second terminal device that can receive the shared content, and "Confirm" and "Cancel" buttons. The user can select the shared content and the second terminal device to receive the shared content as needed. After the user selects the shared content and the second terminal device, for example, after the user selects to project the screen to terminal A, the user can click or touch the "Confirm" button. After the first terminal device 301 detects that the "Confirm" button has been clicked or touched, it can send the shared content (i.e., the screen) to the second terminal device (i.e., terminal A) selected by the user. At the same time, the first terminal device 301 can obtain the synchronization delay for synchronous playback based on the shared content and the connection information between the first terminal device 301 and the second terminal device, and can control the content playback of the first terminal device 301 and / or control the content playback of the second terminal device based on the synchronization delay to achieve content synchronization.

[0297] Alternatively, when a user projects shared content to their selected second terminal device 302 using standard screen mirroring, such as... Figure 8 As shown in (a), a content synchronization window 801 can pop up in the display interface of the first terminal device 301. This content synchronization window 801 can display "Synchronize content?", as well as a "Synchronize" button and a "Skip" button. When the user clicks or touches the "Synchronize" button, the first terminal device 301 can obtain the synchronization delay for synchronized playback based on the shared content and the connection information between the first terminal device 301 and the second terminal device 302. Therefore, it can control the content playback of the first terminal device 301 and / or control the content playback of the second terminal device 302 based on the synchronization delay to achieve content synchronization.

[0298] Alternatively, when a user projects shared content to their selected second terminal device 302 using standard screen mirroring, such as... Figure 8As shown in (b), the "Content Synchronization" button 802 can also be directly displayed on the display interface of the first terminal device 301. When the user clicks or touches the "Content Synchronization" button 802, the first terminal device 301 can obtain the synchronization delay for synchronized playback based on the shared content and the connection information between the first terminal device 301 and the second terminal device 302. Therefore, it can control the content playback of the first terminal device 301 and / or control the content playback of the second terminal device 302 based on the synchronization delay to achieve content synchronization.

[0299] It is understandable that for second terminal devices with display capabilities, users can also enable content synchronization on any of the second terminal devices 302. That is, after the first terminal device 301 projects content to the second terminal device 302 using normal screen mirroring, such as... Figure 8 As shown in (c), a content synchronization window can also pop up in the display interface of the second terminal device 302, or as shown in (c). Figure 8 As shown in (d), the display interface of the second terminal device 302 can display a "Content Synchronization" button to allow the user to enable the content synchronization function. When the user enables the content synchronization function on the second terminal device 302, the second terminal device 302 can send a synchronization command to the first terminal device 301. After receiving the synchronization command from the second terminal device 302, the first terminal device 301 can obtain the synchronization delay for synchronized playback based on the shared content and the connection information between the first terminal device 301 and the second terminal device 302. Therefore, it can control the content playback on the first terminal device 302 and / or control the content playback on the second terminal device 302 based on the synchronization delay to achieve content synchronization.

[0300] In this embodiment, when the first terminal device 301 obtains the synchronization delay for synchronized playback based on the shared content and the connection information between the first terminal device and the second terminal device, it can obtain the synchronization delay according to a preset correspondence. The preset correspondence can be stored in the first terminal device 301 or in a third-party storage device communicatively connected to the first terminal device 301. The preset correspondence refers to the correspondence between the synchronization delay and the shared content and the connection information between the first terminal device and the second terminal device.

[0301] It should be noted that the preset correspondence can be established based on the synchronization delay measured beforehand. Specifically, the synchronization delay can be measured using test data, and the preset correspondence can be established based on the measured synchronization delay. The following will provide a detailed explanation of the synchronization delay measurement in conjunction with application scenarios.

[0302] I. Measurement of synchronization delay during image synchronization

[0303] In video synchronization, the test data can be a specially designed first video, where each frame of the first video has a corresponding video identifier to identify the frame number, frame rate, and other video information. Alternatively, the video identifier can identify the frame number, frame rate, and other video information of the video frame, as well as the device identifier of the terminal device playing the video frame. The video identifier can be a QR code or other data encoding, and can be pre-set or generated in real time. For example, before projecting the first video to a second terminal device for playback, the first terminal device can obtain the device identifier of the second terminal device, generate a video identifier for each video frame based on the device identifier and the corresponding frame number, frame rate, and other video information, and add each video identifier to the corresponding video frame. The following example uses a QR code as the video identifier.

[0304] In one example, the measurement of synchronization delay can be performed by the first terminal device 301. See also... Figure 9 , Figure 9 This diagram illustrates an application scenario for synchronous latency measurement via a first terminal device 301. Specifically, after the first terminal device 301 establishes a network connection with the second terminal device 302 based on the first connection information, the user can initiate screen synchronization measurement on either the first terminal device 301 or the second terminal device 302. At this time, the first terminal device 301 can play a first video and project video frames from the first video onto the second terminal device 302 for display. While the second terminal device 302 displays the video frames projected from the first terminal device 301, the first terminal device 301 can activate its camera to capture the second video frame currently displayed on the second terminal device 302, obtaining an image containing the second video frame. Simultaneously, the first terminal device 301 can also acquire the first video frame currently displayed on the first terminal device 301 at the same time. For example, the first terminal device 301 can acquire the first video frame currently displayed on the first terminal device 301 at the same time in the background. The "same moment" refers to the moment when the second video frame displayed on the second terminal device 302 is captured by the camera, i.e., the moment when the second terminal device 302 plays the second video frame. Then, the first terminal device 301 can determine the synchronization delay between itself and the second terminal device 302 based on the captured image and the first video frame. The first connection information includes the network type and network topology.

[0305] For example, the first terminal device 301 can analyze the QR code in the captured image to obtain the second frame number corresponding to the second video frame and the frame rate corresponding to the first video. Here, the QR code in the captured image refers to the QR code in the second video frame. Simultaneously, the first terminal device 301 can directly obtain the first frame number corresponding to the first video frame in the background, or it can obtain the first frame number by analyzing the QR code in the first video frame. Then, the first terminal device 301 can determine the number of frames between the first video frame displayed by the first terminal device 301 and the second video frame displayed by the second terminal device 302 at the same time based on the first and second frame numbers. Furthermore, it can obtain the synchronization delay for image synchronization between the first terminal device 301 and the second terminal device 302 based on the difference in frame numbers and the frame rate corresponding to the first video. For example, the synchronization delay for image synchronization between the first terminal device 301 and the second terminal device 302 can be obtained by multiplying the frame number by the frame rate.

[0306] Alternatively, the first terminal device 301 can capture a second video frame played by the second terminal device 302 using a camera, obtaining a captured image containing the second video frame, and simultaneously acquiring the second playback time corresponding to the second video frame. For example, the second playback time and the subsequent first playback time can be the system time; that is, the first terminal device 301 can determine the system time when it captures the image as the second playback time corresponding to the second video frame. Then, the first terminal device 301 can analyze the QR code in the captured image to obtain the second frame number corresponding to the second video frame, and obtain the first playback time of the video frame with that second frame number in the background, i.e., obtain the system time when the first terminal device 301 plays the video frame with that second frame number. Finally, the first terminal device 301 can determine the synchronization delay for image synchronization between the first terminal device 301 and the second terminal device 302 based on the time difference between the first playback time and the second playback time.

[0307] In this system, the first terminal device 301 can record and save the system time corresponding to each video frame while playing the first video. Therefore, after determining the second frame number corresponding to the second video frame, the first terminal device 301 can directly obtain the system time when it plays the video frame with the second frame number, that is, obtain the first playback time of the video frame with the second frame number. Alternatively, after determining the second frame number corresponding to the second video frame, the first terminal device 301 can determine the first playback time of the video frame with the second frame number based on the system time when it starts playing the first video and the playback time of the video frame with the second frame number in the first video. For example, if the first terminal device 301 starts playing the first video at 8:30:00, and the playback time of the video frame with the second frame number in the first video is the 10th second, that is, if the video frame with the second frame number is the 10th second video frame in the first video, then the first playback time is 8:30:10.

[0308] For example, the first playback time and the second playback time can be the playback time of a video frame in the first video. Specifically, when the second terminal device 302 plays the first video, the display interface of the second terminal device 302 can display the playback time of the first video, that is, the playback time of the second video frame in the first video. Therefore, when a captured image containing the second video frame is obtained, the first terminal device 301 can obtain the second playback time corresponding to the second video frame based on the captured image. At the same time, the first terminal device 301 can also obtain the first frame number of the first video frame played by the first terminal device 301 at the same time, and can obtain the first playback time of the first video frame in the first video based on the first frame number, so as to determine the synchronization delay for the first terminal device 301 and the second terminal device 302 to synchronize the images based on the time difference between the first playback time and the second playback time.

[0309] Understandably, when the camera of the first terminal device 301 is activated to take a picture, the first terminal device 301 may display a prompt message on its display interface. This prompt message is used to instruct the user to point the camera towards the display interface of the second terminal device 302 to ensure that the camera can capture an image containing the second video frame. Alternatively, after activating the camera of the first terminal device 301, the first terminal device 301 can acquire a preview image through the camera and perform image analysis on the preview image to determine whether the preview image contains the display interface of the second terminal device 302. When the preview image does not contain the display interface of the second terminal device 302, the first terminal device 301 may display a prompt message on its display interface to instruct the user to point the camera towards the display interface of the second terminal device 302.

[0310] In another example, the synchronization delay measurement can be performed by a third terminal device, which is any terminal device other than the first terminal device 301 and the second terminal device 302. See also... Figure 10 , Figure 10 This diagram illustrates an application scenario for synchronization latency measurement via a third terminal device. Specifically, after the first terminal device 301 establishes a network connection with the second terminal device 302 based on the first connection information, the user can initiate screen synchronization measurement on either the first terminal device 301 or the second terminal device 302. At this time, a confirmation window for screen synchronization measurement can pop up on the third terminal device 303 connected to the first terminal device 301 or the second terminal device 302, requesting user confirmation. When the user confirms screen synchronization measurement via the third terminal device 303, the first terminal device 301 can play a first video and project video frames from the first video onto the second terminal device 302 for display. While the second terminal device 302 displays the video frames projected from the first terminal device 301, the third terminal device 303 can activate its camera to capture the first video frame currently displayed on the first terminal device 301 and the second video frame currently displayed on the second terminal device 302, thus obtaining a captured image. That is, the captured image can simultaneously contain the first video frame currently displayed by the first terminal device 301 and the second video frame currently displayed by the second terminal device 302. Then, the third terminal device 303 can perform image analysis on the captured image to obtain the synchronization delay between the first terminal device 301 and the second terminal device 302 for screen synchronization.

[0311] For example, when the QR code contains the device identifier of the terminal device, the third terminal device 303 can directly analyze each QR code in the captured image to obtain the first frame sequence number corresponding to the first video frame, the frame rate corresponding to the first video, and the device identifier of the first terminal device 301; as well as the second frame sequence number corresponding to the second video frame, the frame rate corresponding to the first video, and the device identifier of the second terminal device 302. Subsequently, the third terminal device 303 can determine the number of frames between the first video frame displayed by the first terminal device 301 and the second video frame displayed by the second terminal device 302 at the same time based on the first frame sequence number and the second frame sequence number, and can obtain the synchronization delay for the first terminal device 301 and the second terminal device 302 to synchronize their images based on the number of frames and the frame rate corresponding to the first video. For example, the synchronization delay for image synchronization can be obtained by multiplying the number of frames by the frame rate.

[0312] For example, when the QR code does not contain a device identifier, the third terminal device 303 can perform image recognition on the captured image to obtain a first image area corresponding to the first terminal device 301 and a second image area corresponding to the second terminal device 302. For instance, the display interface of the first terminal device 301 may include a first device number corresponding to the first terminal device 301 (i.e.,...). Figure 10 The terminal device shown is bbb), and the display interface of the second terminal device 302 can be set with the second device number corresponding to the second terminal device 302 (i.e., Figure 10 As shown in the diagram (e.g., terminal device aaa), the third terminal device 303 can identify the device number in the captured image to obtain the first image region corresponding to the first terminal device 301 and the second image region corresponding to the second terminal device 302. For example, when the shapes of the first terminal device 301 and the second terminal device 302 are different, the third terminal device 303 can distinguish between the first terminal device 301 and the second terminal device 302 by performing target recognition on the captured image, thereby obtaining the first image region corresponding to the first terminal device 301 and the second image region corresponding to the second terminal device 302. Then, the third terminal device 303 can analyze the QR code in the first image region to obtain the first frame number and the frame rate corresponding to the first video frame, and can analyze the QR code in the second image region to obtain the second frame number corresponding to the second video frame. Subsequently, the third terminal device 303 can determine the number of frames between the first video frame displayed by the first terminal device 301 and the second video frame displayed by the second terminal device 302 at the same time based on the first frame sequence number and the second frame sequence number. It can then obtain the synchronization delay between the first terminal device 301 and the second terminal device 302 for screen synchronization based on the number of frames differing and the frame rate corresponding to the first video. For example, the synchronization delay can be obtained by multiplying the frame number by the frame rate.

[0313] Alternatively, after obtaining the first video frame corresponding to the first terminal device 301 and the second video frame corresponding to the second terminal device 302, the third terminal device 303 can determine the first playback time of the first video frame and the second playback time of the second video frame based on the captured image. The first playback time is the playback time of the first video frame within the first video, and the second playback time is the playback time of the second video frame within the first video. Then, the third terminal device 303 can determine the synchronization delay between the first terminal device 301 and the second terminal device 302 for image synchronization based on the first and second playback times. For example, when the first video frame is the 10th second frame in the first video and the second video frame is the 10.5th second frame in the first video, the first playback time can be determined to be the 10th second, and the second playback time to be the 10.5th second. That is, the synchronization delay between the first terminal device 301 and the second terminal device 302 for image synchronization is that the second terminal device lags behind the first terminal device by 0.5 seconds.

[0314] Alternatively, the third terminal device 303 can simultaneously record the display interfaces of the first terminal device 301 and the second terminal device 302 using a camera to obtain a screen recording video. Then, the third terminal device 303 can perform image analysis on each frame of the screen recording video to obtain the playback time corresponding to each screen recording image, the first frame number corresponding to the first video frame displayed by the first terminal device 301, and the second frame number corresponding to the second video frame displayed by the second terminal device 302. The playback time corresponding to each video frame can be the system time from which the third terminal device 303 recorded that video frame. Subsequently, the third terminal device 303 can determine the first target screen recording image and the second target screen recording image from the screen recording video, and can determine the synchronization delay for screen synchronization between the first terminal device 301 and the second terminal device 302 based on the time difference between the first playback time corresponding to the first target screen recording image and the second playback time corresponding to the second target screen recording image. The first frame number in the first target screen recording image is the same as the second frame number in the second target screen recording image.

[0315] It should be noted that after the third terminal device 303 captures an image containing the first video frame currently displayed by the first terminal device 301 and the second video frame currently displayed by the second terminal device 302 through its camera, it can send the captured image to the first terminal device 301. The first terminal device 301 can then obtain the synchronization delay between itself and the second terminal device 302 based on the captured image. The specific details of the synchronization delay obtained by the first terminal device 301 from the captured image are the same as those obtained by the third terminal device 303 from the captured image, and will not be repeated here.

[0316] Understandably, after the third terminal device 303 activates its camera, to ensure that an image containing both the first and second video frames is captured, the third terminal device 303 may display a prompt message on its display interface. This prompt message instructs the user to simultaneously point the camera of the third terminal device 303 towards the display interfaces of both the first terminal device 301 and the second terminal device 302. Furthermore, when recording the screen using the camera of the third terminal device 303, the third terminal device 303 can perform real-time image analysis on the recorded image to determine whether the recorded image simultaneously contains the first video frame displayed by the first terminal device 301 and the second video frame displayed by the second terminal device 302. When the screen recording image does not contain the first video frame displayed by the first terminal device 301 and / or does not contain the second video frame displayed by the second terminal device 302, the third terminal device 303 can determine that the camera of the third terminal device 303 is not simultaneously facing the display interface of the first terminal device 301 and the display interface of the second terminal device 302. At this time, the third terminal device 303 can display a prompt message on the display interface of the third terminal device 303. The prompt message is used to prompt the user to simultaneously face the camera of the third terminal device 303 towards the display interfaces of the first terminal device 301 and the second terminal device 302.

[0317] It should be noted that when there are multiple second terminal devices 302, the first terminal device 301 can establish network connections with each of the second terminal devices. These network connections can be the same or different. When acquiring images, the first terminal device 301 can use its camera to capture images of the displays of the multiple second terminal devices 302, obtaining images that simultaneously include the displays of all the second terminal devices 302. Then, the first terminal device 301 can perform image analysis on the captured images and the first video frame displayed by the first terminal device 301 at the same time to determine the synchronization delay between each second terminal device 302 and the first terminal device 301. Alternatively, the third terminal device 303 can use its camera to capture or record images of the displays of the first terminal device 301 and the multiple second terminal devices 302, obtaining images or recorded videos that simultaneously include the displays of both the first terminal device 301 and the multiple second terminal devices 302. Then, the third terminal device 303 can perform image analysis on the captured images or screen recording videos to obtain the synchronization delay between each second terminal device 302 and the first terminal device 301 for screen synchronization.

[0318] It is understandable that when there are multiple second terminal devices 302, the first terminal device 301 can capture images of the display screens of each second terminal device 302 to obtain captured images containing the display screens of each second terminal device, thereby determining the synchronization delay between each second terminal device 302 and the first terminal device 301 for screen synchronization based on the captured images. Alternatively, the third terminal device can capture or record the display screens of the first terminal device 301 and each second terminal device 302 to obtain captured images or screen recording videos containing the display interface of the first terminal device 301 and the display interface of any one of the second terminal devices 302, thereby obtaining the synchronization delay between each second terminal device 302 and the first terminal device 301 for screen synchronization.

[0319] In this embodiment, after obtaining the synchronization delay for screen synchronization between the first terminal device 301 and the second terminal device 302, the first terminal device 301 (or the third terminal device) can establish a preset correspondence between the synchronization delay and the shared content (i.e., the screen) and the first connection information corresponding to the network connection between the first terminal device 301 and the second terminal device 302. Then, this preset correspondence can be associated and stored in the first terminal device 301, or associated and stored in a third-party storage device communicatively connected to the first terminal device 301. Subsequently, when the first terminal device 301 projects its displayed screen to the second terminal device 302 for synchronized display based on the same network connection, the first terminal device 301 can directly obtain the synchronization delay according to the preset correspondence, thereby controlling the screen playback of the first terminal device 301 and / or the screen playback of the second terminal device 302 according to the synchronization delay, achieving screen synchronization between the first terminal device 301 and the second terminal device 302.

[0320] II. Measurement of synchronization delay during audio synchronization

[0321] In audio synchronization, test data can consist of multiple specially designed first audio tracks, each with identical content and time-calibrated within the same time interval using a preset method. The duration of this time interval can be customized, for example, set to 3 milliseconds or 10 milliseconds. For instance, time calibration can be performed using different frequencies. For example, in the first first audio track, the frequency of this time interval could be set to 20 kHz; in the second first audio track, it could be set to 10 kHz; in the third first audio track, it could be set to 5 kHz, and so on.

[0322] Alternatively, time calibration can be performed using different frequency combinations, meaning multiple frequency combinations can be used within the same time interval. In other words, multiple sub-intervals can be selected from the same time interval, and different frequencies can be set for the sub-intervals of different first audio frequencies for time calibration. The frequencies of multiple sub-intervals within the same first audio frequency can be the same or different, while the frequencies of sub-intervals in different first audio frequencies must be distinct from each other; that is, the frequency of any sub-interval in any first audio frequency is different from the frequency of any sub-interval in any other first audio frequency, facilitating subsequent audio analysis. For example, in the first first audio frequency, the frequencies of sub-intervals A and B within the same time interval can be set to 30kHz, and the frequency of sub-interval C can be set to 25kHz; in the second first audio frequency, the frequencies of sub-intervals A and B can be set to 20kHz, and the frequency of sub-interval C can be set to 10kHz; in the third first audio frequency, the frequencies of sub-intervals A and B can be set to 5kHz, and the frequency of sub-interval C can be set to 2kHz, and so on.

[0323] It should be noted that the frequencies of other times in each of the first audio segments can be set to a uniform fixed frequency, such as 2kHz or 0. Here, "other times" refers to times other than the time interval in which a specific frequency is set. For example, in a scenario where time calibration is performed directly using different frequencies, "other times" refers to times other than that same time interval. Similarly, in a scenario where time calibration is performed using different frequency combinations, "other times" refers to times other than the selected sub-interval. The following example illustrates this using time calibration with different frequencies and a frequency of 2kHz for other times.

[0324] Therefore, when there is no time delay between the second terminal devices, each second terminal device plays the audio within the same time interval for the same duration. If the playback times of the audio within the same time interval are different for each second terminal device, it indicates that there is a time delay between the second terminal devices. Thus, the synchronization delay between the second terminal devices can be determined based on the playback time of the audio within the same time interval on each second terminal device.

[0325] Please see Figure 11 , Figure 11 This diagram illustrates an application scenario for measuring synchronization latency during audio playback via a first terminal device. Specifically, after the first terminal device 301 establishes a network connection with each of the second terminal devices 302, the user can initiate audio synchronization measurement on either the first terminal device 301 or the second terminal device 302. At this time, the first terminal device 301 can simultaneously play each first audio clip from its own audio clip to the corresponding second terminal device 302.Figure 11 (a) and Figure 11 As shown in (b), the first terminal device 301 can transmit audio A (i.e., audio with a frequency of 20kHz in the same time interval and a frequency of 2kHz at other times) to the first second terminal device 302 for playback, audio B (i.e., audio with a frequency of 10kHz in the same time interval and a frequency of 2kHz at other times) to the second second terminal device 302 for playback, audio C (i.e., audio with a frequency of 5kHz in the same time interval and a frequency of 2kHz at other times) to the third second terminal device 302 for playback, and so on. While each second terminal device 302 plays the first audio transmitted from the first terminal device 301, the first terminal device 301 can activate its microphone or other recording device to record the audio played by each second terminal device 302, obtaining recorded audio (i.e., second audio) containing the audio played by each second terminal device 302. Then, the first terminal device 301 can perform audio analysis on the recorded audio to obtain the synchronization delay of the audio synchronization among the multiple second terminal devices 301.

[0326] For example, since the frequencies of the second terminal devices are different within the same time interval, the first terminal device 301 can first separate the recorded audio according to the frequency to obtain the recorded audio corresponding to the same time interval in each second terminal device 302. For example, it can obtain... Figure 11 The recorded audio is shown in (c). Then, the first terminal device 301 can obtain the playback time of the recorded audio corresponding to each of the second terminal devices 302, and thus obtain the synchronization delay for audio synchronization of these multiple second terminal devices 302 based on the playback time corresponding to each of the second terminal devices 302. The playback time of the recorded audio corresponding to the second terminal device can refer to the time from when the second terminal device begins playing audio within the same time interval, as recorded by a microphone or other recording device.

[0327] Here, when determining the synchronization delay based on the playback time corresponding to each second terminal device 302, the latest playback time can be used as the basis for determining the synchronization delay. For example, when the playback time corresponding to the first second terminal device 302 is T1, the playback time corresponding to the second second terminal device 302 is T2, and the playback time corresponding to the third second terminal device 302 is T3, and T3 < T1 < T2, the synchronization delay for audio synchronization of these multiple second terminal devices 302 can be determined to include a first synchronization delay and a second synchronization delay. The first synchronization delay is the time by which the audio played by the first second terminal device 302 lags behind the audio played by the third second terminal device 302, and the second synchronization delay is the time by which the audio played by the second second terminal device 302 lags behind the audio played by the second third terminal device 302.

[0328] In this embodiment, after obtaining the synchronization delay of the audio synchronization of the multiple second terminal devices 302, the first terminal device 301 can establish a preset correspondence between the synchronization delay and the shared content (i.e., audio) and the connection information corresponding to the network connection between the first terminal device 301 and each of the second terminal devices 302. This preset correspondence can be associated and stored in the first terminal device 301, or associated and stored in a third-party storage device communicatively connected to the first terminal device 301. Subsequently, when the first terminal device 301 projects the audio from its own memory to the multiple second terminal devices 302 for synchronized playback based on the same network connection, the first terminal device 301 can directly obtain the synchronization delay based on the preset correspondence. Therefore, it can control the audio playback of each second terminal device 302 based on the synchronization delay, thereby achieving audio synchronization between the second terminal devices 302.

[0329] Optionally, when synchronously playing audio through multiple second terminal devices 302, to ensure the equalization of the audio volume played by each second terminal device 302, the first terminal device 302 can also acquire the playback volume of each second terminal device 302 and the relative position between each second terminal device 302 and the first terminal device 301. It can also determine the volume difference of the audio playback by each second terminal device 302 based on its playback volume, thereby determining the volume gain corresponding to each second terminal device 302 based on the volume difference. Then, the first terminal device 301 can establish a preset correspondence between volume gain and relative position, and can associate and save this preset correspondence within the first terminal device 301, or associate and save it to a third-party storage device communicatively connected to the first terminal device 301. Subsequently, when the first terminal device 301 projects the audio from the first terminal device 301 to the multiple second terminal devices 302 for playback based on the same network connection, the first terminal device 301 can obtain the relative position between each second terminal device 302 and the first terminal device 301, and obtain the volume gain according to the relative position and the preset correspondence. Thus, the playback volume of each second terminal device 302 can be adjusted according to the volume gain, so that the playback volume of each second terminal device 302 is balanced, thereby improving the audio playback effect.

[0330] Specifically, the first terminal device 301 can select one playback volume from all playback volumes as a reference volume and determine the volume difference between the other playback volumes and the reference volume. Then, the first terminal device 301 can determine the volume gain corresponding to each second terminal device 302 based on each volume difference. The reference volume can be the maximum volume among all playback volumes, or the minimum volume among all playback volumes, or a user-defined volume, etc.

[0331] It should be noted that when using three or more second terminal devices 302 to form a surround sound system, the placement and angle of each second terminal device 302 can be determined first based on the number of devices. Therefore, during surround sound audio synchronization measurement, the current positions of these second terminal devices 302 can be obtained using location-based service (LBS) and dual microphones. The user can then be prompted to adjust the positions of each second terminal device 302 based on its placement and angle. When prompting the user to adjust the positions of the second terminal devices 302, AR functionality can be incorporated to guide the user in placing the devices. After each second terminal device 302 is adjusted to its corresponding placement and angle, the synchronization delay and audio gain of the audio synchronization of these devices can be measured using multiple first audio measurements. Then, a first preset correspondence can be established between the synchronization delay and the shared content (i.e., audio) and the connection information corresponding to the network connection between each second terminal device 302 and the first terminal device 301. A second preset correspondence can also be established between the volume gain and the positional relationship between each second terminal device 302. The first and second preset correspondences can be stored in the first terminal device 301 or in a third-party storage device communicatively connected to the first terminal device 301. Subsequently, when the first terminal device 301 projects audio to multiple second terminal devices 302 based on the same network connection, the first terminal device 301 can obtain the synchronization delay based on the first preset correspondence and the volume gain based on the second preset correspondence, thereby controlling the audio playback of each second terminal device 302 according to the synchronization delay and audio gain.

[0332] III. Measurement of synchronization delay during video synchronization

[0333] In video synchronization, the test data can be a specially designed first video, where each frame of the first video is assigned a corresponding video identifier to identify the frame number, frame rate, and other video information. Alternatively, the video identifier can identify the frame number, frame rate, and other video information of the video frame, as well as the device identifier of the terminal device playing the video frame. The video identifier can be a QR code or other data encoding, and can be pre-set or generated in real time. For example, before projecting the first video to a second terminal device for playback, the first terminal device can obtain the device identifier of the second terminal device, generate video identifiers for each video frame based on the device identifier and the corresponding frame number, frame rate, and other video information, and add each video identifier to the corresponding video frame. Simultaneously, the audio in the first video is aligned with each video frame through a specific frequency design. For example, audio at frequency A can be associated with the first video frame, audio at frequency B with the second video frame, audio at frequency C with the third video frame, and so on. The following example uses a QR code as the video identifier for illustration.

[0334] In one example, when the video synchronization is synchronized between the screen displayed on the first terminal device 301 and the audio played on the second terminal device 302, after the first terminal device 301 establishes a network connection with the second terminal device 302 based on the first connection information, the user can initiate video synchronization measurement for audio casting on either the first terminal device 301 or the second terminal device 302. At this time, the first terminal device 301 can play the first video and can cast the audio from the first video to the second terminal device 302 for playback. When the second terminal device 302 plays the audio cast from the first terminal device 301, the first terminal device 301 can activate its microphone or other recording device to record the audio currently being played on the second terminal device 302, obtaining the recorded audio (i.e., the target audio). Then, the first terminal device 301 can perform audio analysis on the recorded audio, obtain the frequency and second playback time corresponding to the recorded audio, and determine the second frame number of the second video frame corresponding to the recorded audio based on the frequency of the recorded audio. Subsequently, the first terminal device 301 can obtain the first playback time of the video frame with the second frame number played by the first terminal device 301 in the background, and obtain the synchronization delay between the screen displayed by the first terminal device 301 and the audio played by the second terminal device 302 based on the time difference between the first playback time and the second playback time.

[0335] Alternatively, at the same time as obtaining the recorded audio, the first terminal device 301 can obtain the first frame number of the first video frame played by the first terminal device 301 in the background. Then, the first terminal device 301 can determine the synchronization delay between the screen displayed by the first terminal device 301 and the audio played by the second terminal device 302 based on the second frame number of the second video frame corresponding to the recorded audio, the first frame number, and the frame rate of the first video.

[0336] Alternatively, when the second terminal device 302 plays the audio projected from the first terminal device 301, the third terminal device 303 can activate its microphone or other recording device to record the audio currently being played by the second terminal device 302, thus obtaining recorded audio. Simultaneously, the third terminal device 303 can activate its camera to capture the image currently displayed on the first terminal device 301, thus obtaining a captured image. Then, the third terminal device 303 can perform audio analysis to obtain the frequency corresponding to the recorded audio, and determine the second video frame and its corresponding second frame number based on the frequency. Subsequently, the third terminal device 303 can perform image analysis to obtain the first frame number and frame rate of the first video frame currently displayed on the first terminal device 301, and determine the synchronization delay between the image displayed on the first terminal device 301 and the audio played by the second terminal device 302 based on the first frame number, the second frame number, and the frame rate.

[0337] In this embodiment, after obtaining the synchronization delay between the screen displayed by the first terminal device 301 and the audio played by the second terminal device 302, a preset correspondence can be established between the synchronization delay and the shared content (i.e., the audio in the video) and the first connection information corresponding to the network connection between the first terminal device 301 and the second terminal device 302. This preset correspondence can be stored in the first terminal device 301 or associated with a third-party storage device connected to the first terminal device 301. Subsequently, when the first terminal device 301 projects the audio from the video played by the first terminal device 301 to the second terminal device 302 for playback based on the same network connection, the first terminal device 301 can directly obtain the synchronization delay according to the preset correspondence and control the screen playback of the first terminal device 301 and / or control the audio playback of the second terminal device based on the synchronization delay, thereby achieving synchronization between the screen displayed by the first terminal device 301 and the audio played by the second terminal device 302.

[0338] In another example, when the video synchronization is synchronized between the audio played by the first terminal device 301 and the screen displayed by the second terminal device 302, after the first terminal device 301 establishes a network connection with the second terminal device 302 based on the first connection information, the user can initiate video synchronization measurement for screen projection on either the first terminal device 301 or the second terminal device 302. At this time, the first terminal device 301 can play a first video and project video frames from the first video to the second terminal device 302 for display. While the second terminal device 302 displays the video frames projected from the first terminal device 301, the first terminal device 301 can activate its camera to capture images of the currently displayed video frames on the second terminal device 302. Then, the first terminal device 301 can obtain the second playback time of the second video frame and perform image analysis on the captured image, that is, determine the second frame number corresponding to the second video frame based on the video identifier in the captured image, and determine the frequency of the second audio corresponding to the second video frame based on the second frame number. Subsequently, the first terminal device 301 can obtain the first playback time of the second audio based on the frequency of the second audio, and can obtain the synchronization delay between the audio played by the first terminal device 301 and the screen displayed by the second terminal device 302 based on the time difference between the first playback time and the second playback time. The first playback time and the second playback time can be the system time of the terminal device.

[0339] Alternatively, the first terminal device 301 can determine the second frame number and frame rate of the second video frame played by the second terminal device 302 based on the video identifier in the captured image, and obtain the frequency of the first audio played by the first terminal device 301 at the same time. Then, it can determine the first frame number of the first video frame corresponding to the first audio based on the frequency of the first audio, so as to determine the synchronization delay between the audio played by the first terminal device 301 and the screen displayed by the second terminal device 302 based on the first frame number, the second frame number and the frame rate.

[0340] Alternatively, when the second terminal device 302 displays the video frame projected from the first terminal device 301, the third terminal device 303 can activate its camera to capture the currently displayed image on the second terminal device 302. Simultaneously, the third terminal device 303 can activate its microphone or other recording device to record the audio currently played on the first terminal device 301. Then, the third terminal device 303 can perform image analysis on the captured image to obtain the second frame number corresponding to the second video frame currently displayed on the second terminal device 302 and the frame rate corresponding to the first video. Subsequently, the third terminal device 303 can perform audio analysis on the recorded audio to obtain the frequency corresponding to the recorded audio. Based on the frequency, it can determine the first frame number corresponding to the first video frame of the recorded audio. Therefore, based on the first frame number, the second frame number, and the frame rate corresponding to the first video, it can obtain the synchronization delay between the audio played by the first terminal device 301 and the image displayed on the second terminal device 302.

[0341] In this embodiment, after obtaining the synchronization delay between the audio played by the first terminal device 301 and the screen displayed by the second terminal device 302, a preset correspondence can be established between the synchronization delay and the shared content (i.e., the screen in the video) and the first connection information corresponding to the network connection between the first terminal device 301 and the second terminal device 302. This preset correspondence can be associated and stored in the first terminal device 301 or in a third-party storage device that is communicatively connected to the first terminal device 301. Subsequently, when the first terminal device 301 projects the screen of the video played by the first terminal device 301 onto the second terminal device 302 for playback based on the same network connection, the first terminal device 301 can directly obtain the synchronization delay according to the preset correspondence and control the audio playback of the first terminal device 301 and / or control the screen playback of the second terminal device 302 according to the synchronization delay, thereby achieving synchronization between the audio played by the first terminal device 301 and the screen displayed by the second terminal device 302.

[0342] In another example, when the video synchronization is synchronized between the audio played on the second terminal device 302 and the screen displayed on the second terminal device 302, after the first terminal device 301 establishes a network connection with the second terminal device 302 based on the first connection information, the user can initiate a video synchronization measurement that projects both audio and video onto the same terminal device, either on the first terminal device 301 or the second terminal device 302. At this time, the first terminal device 301 can play a first video and project both the video frames and audio from the first video onto the second terminal device 302 for playback. While the second terminal device 302 plays the video frames and audio projected from the first terminal device 301, the first terminal device 301 can activate its camera to capture images of the currently displayed video frames on the second terminal device 302. Simultaneously, the first terminal device 301 can activate its microphone or other recording device to record the audio currently played on the second terminal device 302, obtaining the recorded audio (i.e., the target audio). Then, the first terminal device 301 can perform image analysis on the captured image to obtain the second frame number corresponding to the second video frame in the captured image and the frame rate corresponding to the first video. Simultaneously, the first terminal device 301 can perform audio analysis on the recorded audio to obtain the frequency of the target audio currently being played by the second terminal device 302, and can determine the first frame number of the first video frame corresponding to the target audio based on the frequency. Therefore, based on the first frame number, the second frame number, and the frame rate corresponding to the first video, the synchronization delay between the audio and the video played by the second terminal device 302 can be determined.

[0343] Alternatively, when the second terminal device 302 plays the video frames and audio projected from the first terminal device 301, the first terminal device 301 can activate its camera to capture the currently displayed video frames on the second terminal device 302, obtaining a captured image. Simultaneously, the first terminal device 301 can activate its microphone or other recording device to record the audio currently playing on the second terminal device 302, obtaining recorded audio. Then, the first terminal device 301 can perform image analysis on the captured image to obtain the second frame number and first playback time of the corresponding second video frame, and determine the target audio corresponding to that second video frame based on the second frame number. Subsequently, the first terminal device 301 analyzes the recorded audio to determine the second playback time corresponding to the target audio, thereby determining the synchronization delay between the audio and video played by the second terminal device 302 based on the first and second playback times.

[0344] Alternatively, when the second terminal device 302 plays the video frames and audio projected from the first terminal device 301, the first terminal device 301 can activate its camera to capture the currently displayed video frames on the second terminal device 302, obtaining a captured image. Simultaneously, the first terminal device 301 can activate its microphone or other recording device to record the currently playing audio from the second terminal device 302, obtaining recorded audio. Then, the first terminal device 301 can perform image analysis on the captured image to obtain the second frame number of the corresponding second video frame and the frame rate of the first video, and obtain the first frame number of the first video frame played by the first terminal device 301 at the same time. Based on the first frame number, the second frame number, and the corresponding frame rate of the first video, a first time delay between the images played by the second terminal device 302 and those played by the first terminal device 301 can be determined. Simultaneously, the first terminal device 301 can perform audio analysis on the recorded audio to obtain the target audio played by the second terminal device 302 and the second playback time of the target audio, and obtain the first playback time of the target audio played by the first terminal device 301. Therefore, based on the first and second playback times, a second time delay can be determined between the audio played by the second terminal device 302 and the audio played by the first terminal device 301. Finally, the first terminal device 301 can determine the synchronization delay between the audio played by the second terminal device 302 and the video feed based on the first and second time delays.

[0345] In this embodiment, after obtaining the synchronization delay between the screen displayed and the audio played on the second terminal device 301, a preset correspondence can be established between the synchronization delay and the shared content (i.e., the audio and screen in the video) and the first connection information corresponding to the network connection between the first terminal device 301 and the second terminal device 302. This preset correspondence can be associated and stored on the first terminal device 301 or on a third-party storage device that is communicatively connected to the first terminal device 301. Subsequently, when the first terminal device 301 projects both the screen and audio played on the first terminal device 301 onto the second terminal device 302 for playback based on the same network connection, the first terminal device 301 can directly obtain the synchronization delay according to the preset correspondence and control the screen playback or audio playback of the second terminal device 302 according to the synchronization delay, thereby achieving synchronization between the screen and audio played on the second terminal device 302.

[0346] In another example, when video synchronization involves the audio played on one second terminal device 302 being synchronized with the video displayed on another second terminal device 302, after the first terminal device 301 establishes a network connection with each of the second terminal devices 302 based on the first connection information, the user can initiate video synchronization measurement by casting the audio and video to different terminal devices from the first terminal device 301 or any of the second terminal devices 302. This is illustrated by taking casting the audio from a video to a smart speaker and casting the video to a smart TV as an example. In this case, the first terminal device 301 can play a first video, and can cast the audio from the first video to a smart speaker for playback, and cast the video frames from the first video to a smart TV for playback. When the smart speaker plays audio projected from the first terminal device 301, and the smart TV plays video frames projected from the first terminal device 301, the first terminal device 301 can activate its microphone or other recording device to record the audio currently playing from the smart speaker. Simultaneously, the first terminal device 301 can activate its camera to capture images of the video frames currently displayed on the smart TV. Then, the first terminal device 301 can perform audio analysis on the recorded audio to obtain the frequency of the target audio currently playing from the smart speaker, and determine the first frame number of the first video frame corresponding to the target audio based on the frequency. At the same time, the first terminal device 301 can perform image analysis on the captured images to determine the second frame number of the second video frame currently displayed on the smart TV and the corresponding frame rate of the first video. Subsequently, the first terminal device 301 can determine the synchronization delay between the audio played by the smart speaker and the video played by the smart TV based on the first frame number, the second frame number, and the corresponding frame rate of the first video.

[0347] Alternatively, the first terminal device 301 can perform image analysis on the captured image to determine the second video frame currently displayed on the smart TV and the first playback time corresponding to the second video frame, and determine the target audio corresponding to the second video frame. Simultaneously, the first terminal device 301 can perform audio analysis on the recorded audio to determine the second playback time of the target audio played by the smart speaker. Therefore, the first terminal device 301 can determine the synchronization delay between the audio played by the smart speaker and the video played by the smart TV based on the first and second playback times.

[0348] In this embodiment, after obtaining the synchronization delay between the screen displayed on the smart TV and the audio played by the smart speaker, a preset correspondence can be established between the synchronization delay and the shared content (i.e., the audio and video in the video) and the first connection information corresponding to the network connection between the first terminal device 301 and each second terminal device (such as the smart speaker and the smart TV). This preset correspondence can be associated and stored in the first terminal device 301 or in a third-party storage device that is communicatively connected to the first terminal device 301. Subsequently, when the first terminal device 301 projects the audio from the video played on the first terminal device 301 to the smart speaker for playback based on the same network connection, and simultaneously projects the video image to the smart TV for display, the first terminal device 301 can directly obtain the synchronization delay according to the preset correspondence, and can control the audio playback of the smart speaker or control the video playback of the smart TV according to the synchronization delay, thereby achieving synchronization between the audio played by the smart speaker and the video displayed on the smart TV.

[0349] Based on the above description, the synchronization delay measurement method and content synchronization method provided in the embodiments of this application will be briefly described below.

[0350] Please see Figure 12 , Figure 12 A schematic flowchart of a synchronization delay measurement method according to an embodiment of this application is shown. This method is applied to a first terminal device to measure the synchronization delay during screen synchronization. Figure 12 As shown, the method may include:

[0351] S1201. The first terminal device establishes a network connection between the first terminal device and the second terminal device based on the first connection information.

[0352] S1202, The first terminal device plays the first video.

[0353] S1203, The first terminal device projects the video frames in the first video to the second terminal device.

[0354] S1204, The second terminal device plays the received video frames.

[0355] S1205. The first terminal device acquires the captured image and determines the synchronization delay between the first terminal device and the second terminal device for screen synchronization based on the captured image. The captured image includes the second video frame currently being played by the second terminal device, and the second video frame is a video frame in the first video.

[0356] During video synchronization, the first terminal device can play a first video and project video frames from the first video onto a second terminal device for playback. While the second terminal device is playing the received video frames, the first terminal device can take a picture of the second terminal device to obtain an image. The first terminal device can then determine the synchronization delay of the video synchronization based on the captured image, thereby improving the accuracy of the synchronization delay measurement.

[0357] Please see Figure 13 , Figure 13 A schematic flowchart of a synchronization delay measurement method according to another embodiment of this application is shown. This method is applied to a first terminal device to measure synchronization delay in video synchronization. Figure 13 As shown, the method may include:

[0358] S1301. The first terminal device establishes a network connection between the first terminal device and the second terminal device based on the first connection information.

[0359] S1302, The first terminal device plays the first video.

[0360] S1303, the first terminal device projects the audio and / or video frames from the first video to the second terminal device.

[0361] S1304. The second terminal device plays the received audio and / or video frames.

[0362] S1305, the first terminal device acquires the target audio and / or captured image, and determines the synchronization delay in video synchronization based on the target audio and / or captured image.

[0363] In video synchronization, the first terminal device can play a first video and project video frames and / or audio from the first video to the second terminal device. While the second terminal device plays the received video frames and / or audio, the first terminal device can take a picture of the second terminal device's display screen to obtain a captured image, and / or capture audio from the second terminal device to obtain the target audio. Then, the first terminal device can determine the synchronization delay in video synchronization based on the captured image and / or the target audio.

[0364] Please see Figure 14 , Figure 14 A schematic flowchart of a synchronization delay measurement method according to another embodiment of this application is shown. This method is applied to a first terminal device to measure synchronization delay in audio synchronization. Figure 14 As shown, the method may include:

[0365] S1401. The first terminal device establishes a network connection between itself and each of the second terminal devices based on the first connection information.

[0366] S1402. The first terminal device acquires each first audio, all of which are the same, and each first audio is time-calibrated in the same time interval by a preset method.

[0367] S1403, the first terminal device respectively transmits the corresponding first audio to each of the second terminal devices.

[0368] S1404. Each second terminal device plays the received first audio.

[0369] S1405. The first terminal device acquires the second audio, determines the playback time of the same time interval in each second terminal device based on the second audio, and determines the synchronization delay in audio synchronization based on each playback time. The second audio includes the audio played by each second terminal device.

[0370] In audio synchronization, a first terminal device can copy the same first audio file multiple times and time each file within the same time interval using a preset method. Then, the first terminal device can send the corresponding first audio file to each second terminal device. While each second terminal device plays the received first audio file, the first terminal device can record the audio played by each second terminal device to obtain a second audio file. Based on the second audio file, the first terminal device can determine the playback time of the same time interval on each second terminal device, thereby determining the synchronization delay of the audio synchronization based on the playback time.

[0371] Please see Figure 15 , Figure 15 A schematic flowchart illustrating a content synchronization method provided in an embodiment of this application is shown. Figure 15 As shown, the method may include:

[0372] S1501, The first terminal device establishes a network connection between the first terminal device and the second terminal device.

[0373] S1502, the first terminal device determines the connection information between the first terminal device and the second terminal device, obtains the shared content, and determines the synchronization delay for synchronous playback based on the shared content and the connection information.

[0374] S1503, The first terminal device sends shared content to the second terminal device.

[0375] S1504. The first terminal device controls the playback of content on the first terminal device according to the synchronization delay, and / or:

[0376] S1505, The first terminal device controls the playback of content on the second terminal device according to the synchronization delay.

[0377] In this embodiment, when the first terminal device shares content to the second terminal device for playback, the first terminal device can obtain the shared content and the corresponding second terminal device. Then, the first terminal device can determine the connection information between itself and the second terminal device, and determine the synchronization delay for synchronous playback based on the shared content and the connection information. This allows it to control the content playback on the first terminal device and / or the content playback on the second terminal device based on the synchronization delay, ensuring synchronized content playback and improving user experience.

[0378] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0379] This application also provides a terminal device, which includes at least one memory, at least one processor, and a computer program stored in the at least one memory and executable on the at least one processor. When the processor executes the computer program, it causes the terminal device to implement the steps in any of the above-described method embodiments. For example, the structure of the terminal device can be as follows: Figure 1 As shown.

[0380] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a computer, causes the computer to perform the steps in any of the above method embodiments.

[0381] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps in any of the above method embodiments.

[0382] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include at least: any entity or device capable of carrying computer program code to a device / terminal equipment, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable storage media cannot be electrical carrier signals or telecommunication signals.

[0383] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0384] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0385] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0386] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0387] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A synchronization delay measurement method, applied to a first terminal device, for measuring the synchronization delay during screen synchronization, characterized in that, The method includes: The first terminal device establishes a network connection between the first terminal device and the second terminal device based on the first connection information; The first terminal device plays the first video and projects video frames from the first video onto the second terminal device. The first terminal device acquires a captured image, the captured image including a second video frame currently being played by the second terminal device, the second video frame being a video frame in the first video; The first terminal device determines the synchronization delay between the first terminal device and the second terminal device for image synchronization based on the captured image; A preset correspondence is established between the first connection information, the screen, and the synchronization delay. The preset correspondence is used to obtain the synchronization delay when the first terminal device projects the screen displayed by the first terminal device to the second terminal device for synchronous display based on the network connection corresponding to the first connection information, and to realize screen synchronization between the first terminal device and the second terminal device according to the synchronization delay.

2. The method according to claim 1, characterized in that, The first terminal device acquires the captured images including: The first terminal device acquires the image captured by the second terminal device through its camera.

3. The method according to claim 2, characterized in that, The first terminal device determines the synchronization delay between the first terminal device and the second terminal device for image synchronization based on the captured image, including: The first terminal device determines the second playback time of the second video frame based on the captured image; The first terminal device obtains the first playback time of the first video frame played by the first terminal device at the same time. The first playback time and the second playback time are the playback times of the video frame in the first video. The same time is the moment when the second terminal device plays the second video frame. The first terminal device determines the synchronization delay for synchronizing the images between the first terminal device and the second terminal device based on the first playback time and the second playback time.

4. The method according to claim 2, characterized in that, Each video frame of the first video is provided with a corresponding video identifier, and the video identifier includes a frame number; The first terminal device determines the synchronization delay between the first terminal device and the second terminal device for image synchronization based on the captured image, including: The first terminal device determines the second playback time of the second video frame based on the captured image, and determines the second frame number of the second video frame based on the video identifier in the captured image; The first terminal device obtains the first playback time of the video frame with the second frame sequence number played by the first terminal device; The first terminal device determines the synchronization delay between the first terminal device and the second terminal device for screen synchronization based on the first playback time and the second playback time; Wherein, the first playback time and the second playback time are the system time of the first terminal device or the second terminal device.

5. The method according to claim 2, characterized in that, Each video frame of the first video is provided with a corresponding video identifier, which includes a frame number and a frame rate; The first terminal device determines the synchronization delay between the first terminal device and the second terminal device for image synchronization based on the captured image, including: The first terminal device obtains the second frame number and frame rate of the second video frame based on the video identifier in the captured image, and obtains the first frame number of the first video frame played by the first terminal device at the same time, wherein the same time is the moment when the second terminal device plays the second video frame; The first terminal device determines the synchronization delay for screen synchronization between the first terminal device and the second terminal device based on the first frame sequence number, the second frame sequence number, and the frame rate.

6. The method according to claim 5, characterized in that, The first terminal device obtains the first frame sequence number of the first video frame played by the first terminal device at the same time, including: The first terminal device obtains the first frame sequence number of the first video frame played by the first terminal device at the same time based on the video identifier in the first video frame.

7. The method according to claim 1, characterized in that, The first terminal device acquires the captured images including: The first terminal device acquires the captured image through the camera of the third terminal device, wherein the third terminal device is a terminal device other than the first terminal device and the second terminal device, and the captured image also includes the first video frame currently being played by the first terminal device.

8. The method according to claim 7, characterized in that, The first terminal device determines the synchronization delay between the first terminal device and the second terminal device for image synchronization based on the captured image, including: The first terminal device determines the first playback time of the first video frame and the second playback time of the second video frame based on the captured image. The first playback time and the second playback time are the playback times of the video frames in the first video. The first terminal device determines the synchronization delay for synchronizing the images between the first terminal device and the second terminal device based on the first playback time and the second playback time.

9. The method according to claim 7, characterized in that, Each video frame of the first video is provided with a corresponding video identifier, which includes a frame number and a frame rate; The first terminal device determines the synchronization delay between the first terminal device and the second terminal device for image synchronization based on the captured image, including: The first terminal device determines the first frame number and frame rate of the first video frame based on the first video identifier in the captured image, and determines the second frame number of the second video frame based on the second video identifier in the captured image; The first terminal device determines the synchronization delay for screen synchronization between the first terminal device and the second terminal device based on the first frame sequence number, the second frame sequence number, and the frame rate.

10. The method according to claim 7, characterized in that, The captured images include multiple images. The first terminal device determines the synchronization delay between the first terminal device and the second terminal device for image synchronization based on the captured images, including: The first terminal device determines the second playback time of the second video frame based on the first captured image, and determines the second frame number of the second video frame based on the video identifier in the first captured image, wherein the first captured image is any one of the captured images; The first terminal device determines the second captured image based on the second frame number, and the second captured image includes a video frame with the second frame number played by the first terminal device; The first terminal device determines the first playback time for playing the video frame with the second frame number based on the second captured image; The first terminal device determines the synchronization delay between the first terminal device and the second terminal device for screen synchronization based on the first playback time and the second playback time; Wherein, the first playback time and the second playback time are the system time of the first terminal device or the second terminal device.

11. The method according to any one of claims 1 to 10, characterized in that, The first connection information includes network type and network topology.

12. The method according to any one of claims 4 to 6 or 9 to 10, characterized in that, The video is identified by a QR code.

13. A method for measuring synchronization delay, characterized in that, Applied to a first terminal device, for measuring synchronization delay in video synchronization, the method includes: The first terminal device establishes a network connection between the first terminal device and the second terminal device based on the first connection information; The first terminal device plays the first video and projects the audio and / or video frames from the first video onto the second terminal device. The first terminal device acquires target audio and / or captured images, and determines the synchronization delay in video synchronization based on the target audio and / or captured images; A preset correspondence is established between the first connection information, the video, and the synchronization delay. The preset correspondence is used to obtain the synchronization delay when the first terminal device projects the audio and / or video frames in the video played on the first terminal device to the second terminal device for synchronous playback based on the network connection corresponding to the first connection information, and to achieve video synchronization based on the synchronization delay.

14. The method according to claim 13, characterized in that, Each video frame of the first video is provided with a corresponding audio, and the frequencies of the audio corresponding to each video frame are different. When video synchronization is achieved by synchronizing the video frames played by the first terminal device with the audio played by the second terminal device, the target audio is the audio currently played by the second terminal device. The first terminal device determines the synchronization delay in video synchronization based on the target audio and / or the captured image, including: The first terminal device acquires the frequency of the target audio and determines the second frame number of the second video frame corresponding to the target audio based on the frequency of the target audio. The first terminal device obtains the first frame sequence number of the first video frame played by the first terminal device at the same time, wherein the same time is the moment when the second terminal device plays the target audio. The first terminal device determines the synchronization delay for video synchronization between the first terminal device and the second terminal device based on the first frame sequence number, the second frame sequence number, and the frame rate of the first video.

15. The method according to claim 13, characterized in that, Each video frame of the first video is provided with a corresponding audio, and the frequencies of the audio corresponding to each video frame are different. When video synchronization is achieved by synchronizing the video frames played by the first terminal device with the audio played by the second terminal device, the target audio is the audio currently played by the second terminal device. The first terminal device determines the synchronization delay in video synchronization based on the target audio and / or the captured image, including: The first terminal device acquires the frequency of the target audio and the second terminal device acquires the second playback time of the target audio; The first terminal device determines the second frame number of the second video frame corresponding to the target audio based on the frequency of the target audio, and obtains the first playback time of the video frame with the second frame number played by the first terminal device; The first terminal device determines the synchronization delay between the first terminal device and the second terminal device for video synchronization based on the first playback time and the second playback time; Wherein, the first playback time and the second playback time are the system time of the first terminal device.

16. The method according to claim 13, characterized in that, Each video frame of the first video is provided with a corresponding video identifier and audio. The video identifier includes a frame number and a frame rate. The frequency of the audio corresponding to each video frame is different. When the video synchronization is synchronized between the audio played by the first terminal device and the video frame played by the second terminal device, the captured image is the image corresponding to the second terminal device, and the captured image includes the second video frame currently being played by the second terminal device. The first terminal device determines the synchronization delay in video synchronization based on the target audio and / or the captured image, including: The first terminal device determines the second frame number and frame rate of the second video frame based on the video identifier in the captured image; The first terminal device obtains the frequency of the first audio played by the first terminal device at the same time, and determines the first frame number of the first video frame corresponding to the frequency of the first audio. The first terminal device determines the synchronization delay for video synchronization between the first terminal device and the second terminal device based on the first frame sequence number, the second frame sequence number, and the frame rate.

17. The method according to claim 13, characterized in that, Each video frame of the first video is provided with a corresponding video identifier and audio. The video identifier includes a frame number and a frame rate. The frequency of the audio corresponding to each video frame is different. When the video synchronization is synchronized between the audio played by the first terminal device and the video frame played by the second terminal device, the captured image is the image corresponding to the second terminal device, and the captured image includes the second video frame currently being played by the second terminal device. The first terminal device determines the synchronization delay in video synchronization based on the target audio and / or the captured image, including: The first terminal device acquires the second playback time of the second video frame and determines the second frame number of the second video frame based on the video identifier in the captured image; The first terminal device determines the second audio corresponding to the second frame sequence number and obtains the first playback time of the second audio played by the first terminal device; The first terminal device determines the synchronization delay for video synchronization between the first terminal device and the second terminal device based on the first playback time and the second playback time.

18. The method according to claim 13, characterized in that, Each video frame of the first video is provided with a corresponding video identifier and audio. The video identifier includes a frame number and a frame rate. The frequency of the audio corresponding to each video frame is different. When the video synchronization is synchronized with the audio and video frames played by the second terminal device, the target audio is the audio currently played by the second terminal device, the captured image is the image corresponding to the second terminal device, and the captured image includes the second video frame currently played by the second terminal device; The first terminal device determines the synchronization delay in video synchronization based on the target audio and / or the captured image, including: The first terminal device determines the second frame number and frame rate of the second video frame played by the second terminal device based on the video identifier in the captured image; The first terminal device determines the first frame number of the first video frame corresponding to the target audio based on the frequency of the target audio. The first terminal device determines the synchronization delay for the second terminal device to perform video synchronization based on the first frame sequence number, the second frame sequence number, and the frame rate.

19. A content synchronization method, characterized in that, Applied to a first terminal device, the method includes: The first terminal device determines the connection information between the first terminal device and the second terminal device; The first terminal device acquires shared content and determines the synchronization delay for synchronous playback based on the shared content and the connection information, wherein the synchronization delay is obtained by the method of any one of claims 1 to 18; The first terminal device sends the shared content to the second terminal device, and controls the playback of the content on the first terminal device according to the synchronization delay, and / or controls the playback of the shared content on the second terminal device.

20. The method according to claim 19, characterized in that, The first terminal device sends the shared content to the second terminal device, and controls the playback of the content on the first terminal device according to the synchronization delay, and / or controls the playback of the shared content on the second terminal device, including: The first terminal device sends the shared content, the synchronization delay, and a control instruction to the second terminal device. The control instruction is used to instruct the second terminal device to control the playback of the shared content according to the synchronization delay.

21. The method according to claim 19, characterized in that, The first terminal device sends the shared content to the second terminal device, and controls the playback of the content on the first terminal device according to the synchronization delay, and / or controls the playback of the shared content on the second terminal device, including: The first terminal device determines the playback time of the second terminal device based on the synchronization delay, and sends the shared content to the second terminal device during the playback time.

22. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it causes the terminal device to implement the method as described in any one of claims 1 to 21.

23. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a computer, it causes the computer to implement the method as described in any one of claims 1 to 21.

Images