A display screen synchronization method, system and electronic device

Abstract

The application provides a display picture synchronization method, which can comprise the following steps: a source terminal device sends a first picture frame and a first specified display time corresponding to the first picture frame to each of receiving terminal devices; if each of the receiving terminal devices successfully receives the first picture frame and determines that the first picture frame can be displayed according to the first specified display time, each of the devices for synchronized display of picture frames displays the first picture frame according to the first specified display time; if there is a first receiving terminal device that does not successfully receive the first picture frame or successfully receives the first picture frame but determines that the first picture frame cannot be displayed according to the first specified display time, each of the devices for synchronized display of picture frames does not display the first picture frame. Thus, as long as there is a first receiving terminal device that cannot display the first picture frame according to the first specified display time, the devices for synchronized display of picture frames do not display the first picture frame, thereby ensuring synchronized display of the display picture.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of display, and particularly relates to a display picture synchronization method and system and an electronic device. BACKGROUND

[0002] With the development of live broadcast, education, conference and other application scenarios, the demand for multi-screen display is growing, and the split screen and spliced screen technologies are also constantly improving. If multiple display devices are required to display the same picture frame at the same time, display picture synchronization needs to be performed. Current display picture synchronization is mainly realized through hardware, for example, picture frames are transmitted through wired connection, and the receiving display device directly displays the received picture frames. However, this way has high implementation cost and complex wiring. At present, there is no software solution that can well realize the synchronized display of picture frames. SUMMARY

[0003] Therefore, the embodiments of the present application provide a display picture synchronization method, system and electronic device, which can effectively avoid the occurrence of picture frame asynchronization during multi-screen display.

[0004] A first aspect of the embodiments of the present application provides a display picture synchronization method applied to a receiving end device. The receiving end device includes frame loss information, and the frame loss information is used to record the identification of picture frames that are not displayed. The method can include the following steps: the receiving end device establishes a communication connection with a source end device; the receiving end device receives a first picture frame sent by the source end device and a first specified display time corresponding to the first picture frame; if the receiving end device determines that the first picture frame can be displayed according to the first specified display time, and the frame number of the first picture frame is not in the frame loss information, the receiving end device displays the first picture frame according to the first specified display time; if the receiving end device determines that the first picture frame cannot be displayed according to the first specified display time, the identification of the first picture frame is added to the frame loss information, and the identification of the first picture frame is notified to other devices; wherein the other devices are used for synchronized display of picture frames with the receiving end device.

[0005] In the display picture synchronization method provided by the embodiments of the present application, the receiving end device can determine under what circumstances the first picture frame can be displayed and at which time the first picture frame should be displayed according to the first specified display time and the frame loss information; the receiving end device can notify the identification of the first picture frame to other devices in the case that it has received the first picture frame but cannot display the first picture frame. Thus, the synchronized display of the display picture is realized through the software mode, without relying on the wired connection on the hardware, which is low in cost and easy to implement. In some scenarios, the above receiving end device includes the frame loss information, which can be understood as that the frame loss information is maintained in the receiving end device, for example, the storage space of the frame loss information is allocated, the frame loss information is dynamically updated, and the like.

[0006] In a possible implementation, the method further includes the following steps: if the identifier of the first picture frame is in the lost frame information, the receiving end device does not display the first picture frame.

[0007] In some scenarios, the identifier of the first picture frame being in the lost frame information indicates that the first picture frame cannot be displayed by other devices, and thus the receiving end device also does not display the first picture frame. In this way, the situation that the first picture frame is not displayed by other devices but is displayed by the receiving end device can be avoided, and the synchronous display of the picture frames is facilitated.

[0008] In a possible implementation, the method further includes the following steps: the receiving end device determines the lost frame identifiers according to the identifiers of the picture frames that have been successfully received; the lost frame identifiers include the identifiers of the picture frames that have not been successfully received by the receiving end device; the receiving end device adds the lost frame identifiers to the lost frame information and notifies other devices of the lost frame identifiers.

[0009] In some scenarios, for example, network fluctuation, network instability, bandwidth occupied by other devices, and the like, the receiving end device may fail to successfully receive some picture frames (i.e., lost frames). In this case, the receiving end device adds the lost frame identifiers to the lost frame information and notifies other devices. In this way, the situation that the lost frames are not displayed by the receiving end device but are displayed by other devices can be avoided, and the synchronous display of the picture frames is facilitated.

[0010] In a possible implementation, the step in which the receiving end device determines that the first picture frame can be displayed at the first specified display time includes the following steps: the receiving end device obtains an estimated decoding time delay required for decoding the first picture frame, determines an estimated display time of the first picture frame according to the estimated decoding time delay, and determines that the first picture frame can be displayed at the first specified display time if the estimated display time is earlier than or equal to the first specified display time.

[0011] In a possible implementation, the step in which the receiving end device determines that the first picture frame cannot be displayed at the first specified display time includes the following steps: the receiving end device obtains an estimated decoding time delay required for decoding the first picture frame, determines an estimated display time of the first picture frame according to the estimated decoding time delay, and determines that the first picture frame cannot be displayed at the first specified display time if the estimated display time is later than the first specified display time.

[0012] Since it also takes a certain time for the receiving end device to decode the first picture frame, after receiving the first picture frame, the receiving end device can first determine whether it is in time to display the first picture frame. In the case of determining that it is in time to display the first picture frame, the receiving end device determines that the first picture frame can be displayed at the first specified time, and optionally, the receiving end device decodes the first picture frame. In the case of determining that it is not in time to display the first picture frame, the receiving end device notifies other devices of the identifier of the first picture frame. In this way, the occurrence of the situation that the receiving end device does not display the first picture frame but other devices still display the first picture frame can be avoided, which helps to realize synchronous display of the displayed pictures.

[0013] Optionally, in the case of determining that it is not in time to display the first picture frame, the receiving end device can also not decode the first picture frame, so as to avoid wasting operation resources on decoding the first picture frame which will not be displayed, and help to save operation resources of the receiving end device.

[0014] Optionally, in the case of determining that it is not in time to display the first picture frame, and the first picture frame is not a key frame (i.e., a frame on which other picture frames depend, which can be generally referred to as an “I frame”), the receiving end device decodes the first picture frame. That is, although it is determined that it is not in time to display the first picture frame, if the first picture frame is a frame on which other picture frames depend, the first picture frame still needs to be decoded to ensure that the other picture frames can be normally displayed.

[0015] In a possible implementation, the method further includes the following steps: the receiving end device performs clock synchronization with the source end device, and determines a clock deviation between the system time of the receiving end device and the system time of the source end device; and the step of determining that the first picture frame can be displayed at the first specified display time if the estimated display time is earlier than or equal to the first specified display time specifically includes: correcting the estimated display time using the clock deviation to obtain a corrected estimated display time, and determining that the first picture frame can be displayed at the first specified display time if the corrected estimated display time is earlier than or equal to the first specified display time.

[0016] Due to the inherent physical properties of hardware devices, there is usually an inevitable clock deviation between the system time of the receiving end device and the system time of the source end device. The receiving end device displays the first picture frame at the first specified display time corrected using the clock deviation, which helps to avoid picture frame asynchronization caused by the clock deviation.

[0017] In a possible implementation, the step of obtaining, by the receiving end device, the estimated decoding time delay required for decoding the first picture frame, specifically comprises: obtaining, by the receiving end device, the estimated decoding time delay required for decoding the first picture frame according to actual decoding time delays consumed by decoding historical picture frames; wherein the historical picture frames include at least one picture frame whose identification is located before the identification of the first picture frame.

[0018] Generally, the decoding time delay required for the receiving end device to decode a picture frame is related to the processor performance of the receiving end device, the current processor occupancy, the adopted coding algorithm and other factors, and thus is often continuous. Therefore, the receiving end device can obtain (for example, estimate, predict, etc.) the estimated decoding time delay to be consumed by decoding the first picture frame according to the actual decoding time delay (that is, the actual decoding time delay) consumed by the receiving end device to decode historical picture frames. In this way, it is helpful for the receiving end device to more accurately determine whether the receiving end device can display the first picture frame in time.

[0019] In a possible implementation, the method further comprises the following steps: determining, by the receiving end device, the current decoding time delay of the receiving end device according to the actual decoding time delay of the at least one successfully received picture frame; and sending, by the receiving end device, the current decoding time delay to the source end device. Correspondingly, the source end device receives the current decoding time delay.

[0020] In a possible implementation, the method further comprises the following steps: determining, by the receiving end device, the current transmission time delay of the receiving end device according to the actual transmission time delay of the at least one successfully received picture frame; and sending, by the receiving end device, the current transmission time delay to the source end device. Correspondingly, the source end device receives the current transmission time delay.

[0021] In a scenario, the number of receiving end devices is at least one, and the number of current decoding delays and / or current transmission delays received by the source end device is also at least one. The source end device can determine the current delay according to the at least one current decoding delay and / or current transmission delay, for example, according to the maximum value in the at least one current decoding delay and / or current transmission delay. In turn, the source end device specifies the specified display time of the subsequent picture frame based on the current delay. Thus, the source end device determines the current delay based on the current decoding delay and / or current transmission delay of the receiving end device, which can make the current delay dynamically change and be consistent with the actual network transmission situation in the actual application scenario and / or the actual decoding delay of the receiving end device. In this way, it helps to make the specified display time specified by the source end device for the picture frame a suitable value, so that the receiving end device can display as much as possible, and helps to avoid the situation that the receiving end device cannot receive and display some picture frames. Moreover, with this dynamic current delay changing mechanism, in some embodiments, the delay of the first frame can not need to be set to a relatively large value (for example, 200 ms, 600 ms), but can be set to a relatively small value, which helps to reduce the start-up delay and improve user experience.

[0022] In a possible implementation, the method further includes the following steps: the receiving end device receives the missing frame identifier and / or the actively discarded frame identifier sent by the other device, and adds the missing frame identifier and / or the actively discarded frame identifier sent by the other device to the missing frame information; in some embodiments, the missing frame identifier sent by the other device can include the identifier of the picture frame that the other device has not successfully received, which is determined by the other device according to the identifier of the picture frame that the other device has successfully received; the actively discarded frame identifier sent by the other device includes the frame number of the picture frame that the other device determines cannot be displayed according to the second specified display time; in some embodiments, the second specified display time can be the specified display time corresponding to the picture frame corresponding to the actively discarded frame identifier sent by the other device.

[0023] Thus, the picture frame (i.e., the missing frame and / or the actively discarded frame) that the other device cannot display will not be displayed by the receiving end device. In this way, it can avoid the situation that the other device does not display the missing frame and / or the actively discarded frame, but the receiving end device still displays the missing frame and / or the actively discarded frame, and helps to realize the synchronous display of the displayed picture.

[0024] In a possible implementation, the receiving end device receives the first picture frame sent by the source end device, and the step of receiving the first specified display time corresponding to the first picture frame specifically includes: the receiving end device receives a first message, wherein the first message carries the first picture frame and the first specified display time, and optionally, the first message further carries a sending time of the first message and an identifier of the first picture frame.

[0025] The source end device packs the related information (for example, the first picture frame, the first specified display time, and the identifier of the first picture frame) of the first picture frame into the first message for sending, which helps the receiving end device to accurately determine the information corresponding to the first picture frame.

[0026] In a possible implementation, the receiving end device and the source end device are in the same local area network. For example, they are in the same WiFi local area network, access the same router, or are in the same Mesh network, so that the receiving end device and the source end device can conveniently interact with each other.

[0027] In a possible implementation, the identifier of the picture frame includes a frame number of the picture frame. It should be understood that the identifier of the picture frame (for example, the identifier of the first picture frame) can also include other content besides the frame number, which is not limited in the embodiments of the application.

[0028] The second aspect of the embodiments of the application provides a display picture synchronization method applied to a source end device. The source end device is configured to acquire picture frames and send the acquired picture frames to a receiving end device. The method can include the following steps: the source end device establishes a communication connection with the receiving end device; and the source end device sends a first picture frame and a first specified display time corresponding to the first picture frame to the receiving end device. The first specified display time is determined by the source end device according to a current transmission delay and / or a current decoding delay. The current transmission delay is related to an actual transmission delay of a historical picture frame, and the current decoding delay is related to an actual decoding delay of the receiving end device in decoding the historical picture frame. The historical picture frame includes at least one picture frame whose identifier is before the identifier of the first picture frame.

[0029] In some embodiments of the method for synchronizing display of a picture frame, the source device can determine a current time delay according to a current transmission time delay and / or a current decoding time delay, and then determine the first specified display time of the first picture frame according to the determined current time delay. The current time delay is not a fixed value set in advance, but is dynamically changed and is associated with the current network transmission state and / or the decoding capability of the current receiving device. Thus, the first specified display time of the first picture frame specified by the source device is a suitable value, so that the receiving device can display the picture frame as much as possible, and the situation that the receiving device cannot receive or display some picture frames can be avoided. Moreover, in some embodiments, the time delay of the first picture frame can be set to a relatively small value, instead of a relatively large value (e.g., 200 ms or 600 ms), which helps to reduce the start-up time delay and improve user experience.

[0030] In a possible implementation, the source device is further configured to synchronize display of the picture frames with the receiving devices, and in this case, the source device includes frame loss information, and the frame loss information is used to record the identification of the picture frames that are not displayed. The method further includes the following steps: if the frame number of the first picture frame is not in the frame loss information, the source device displays the first picture frame according to the first specified display time; and if the frame number of the first picture frame is in the frame loss information, the source device does not display the first picture frame.

[0031] In some scenarios, if the frame number of the first picture frame is not in the frame loss information, it indicates that each of the receiving devices successfully receives the first picture frame and determines that the first picture frame can be displayed according to the first specified display time. In this case, if the source device also synchronizes display of the picture frames, the first picture frame can be displayed.

[0032] In some scenarios, if the frame number of the first picture frame is in the frame loss information, it indicates that at least one of the receiving devices fails to successfully receive the first picture frame, or even successfully receives the first picture frame but determines that the first picture frame cannot be displayed according to the first specified display time. In this case, if the source device also synchronizes display of the picture frames, the first picture frame cannot be displayed.

[0033] Thus, the synchronization of the picture frames displayed by the source device and the receiving devices is ensured.

[0034] In some scenarios, the source device includes the frame loss information, which means that the frame loss information is maintained in the source device, for example, a storage space is allocated for the frame loss list, and the frame loss information is dynamically updated.

[0035] In a possible implementation, when the source device is used to synchronize the display of the picture frames with the sink device, the method further includes the following steps: the source device receives the missing frame identifier and / or the actively discarded frame identifier sent by the sink device, and adds the missing frame identifier and / or the actively discarded frame identifier sent by the sink device to the frame loss information; wherein the missing frame identifier sent by the sink device includes the identifier of the picture frame that is not successfully received by the sink device, which is determined by the sink device according to the identifier of the picture frame successfully received by the sink device; the actively discarded frame identifier sent by the sink device includes the identifier of the picture frame that cannot be displayed at the second specified display time point, which is determined by the sink device; and the second specified display time point is the specified display time point corresponding to the picture frame corresponding to the actively discarded frame identifier sent by the sink device.

[0036] Therefore, the picture frame that cannot be displayed by the sink device (i.e., the missing frame and / or the actively discarded frame) will not be displayed by the source device. In this way, the occurrence of the situation that the sink device does not display the missing frame and / or the actively discarded frame, but the source device still displays the missing frame and / or the actively discarded frame, can be avoided, which helps to realize the synchronized display of the picture frames.

[0037] In a possible implementation, the method further includes the following steps: the source device receives the current transmission delay of the at least one sink device and / or the current decoding delay of the at least one sink device sent by the at least one sink device; and the source device determines the maximum value in the current transmission delay of the at least one sink device as the current transmission delay, and / or determines the maximum value in the current decoding delay of the at least one sink device as the current decoding delay.

[0038] As described above, the first specified display time point is determined by the source device according to the current transmission delay and / or the current decoding delay, and the current decoding delay and / or the current transmission delay is sent by the sink device to the source device. Determining the maximum value as the current decoding delay and / or the current transmission delay helps to ensure that the first specified display time point specified by the source device is as appropriate as possible, and to ensure that all sink devices can receive and display the first picture frame as soon as possible.

[0039] In a possible implementation, the current transmission delay of the sink device is determined by the sink device according to the actual transmission delay of the historical picture frames received by the sink device, and the current decoding delay of the sink device is determined by the sink device according to the actual decoding delay of the historical picture frames received by the sink device.

[0040] In this way, the current transmission delay and the current decoding experiment can be consistent with the actual network transmission in the actual application scenario and / or the actual decoding delay of the receiving end device, and the first specified display time specified by the source end device can be a suitable time.

[0041] In a possible implementation, the method further includes the following step: the source end device performs clock synchronization with the receiving end device. Due to the inherent physical properties of hardware devices, there is usually an inevitable clock deviation between the system time of the source end device and the system time of the receiving end device. The clock synchronization can enable the receiving end device to obtain and save the clock deviation. In turn, the receiving end device can display the first picture frame according to the first specified display time corrected by using the clock deviation, and the picture frame asynchronization caused by the clock deviation can be avoided.

[0042] In a possible implementation, the source end device and the receiving end device are in the same local area network. For example, they are in the same WiFi local area network, access the same router, or are in the same Mesh network. Thus, the receiving end device and the source end device can conveniently perform information interaction.

[0043] In a possible implementation, the identifier of the picture frame includes a frame number of the picture frame. It should be understood that the identifier of the picture frame (such as the identifier of the first picture frame) can also include other contents besides the frame number, which is not limited in the embodiments of the application.

[0044] A third aspect of the embodiments of the application provides a picture frame display synchronization method, applied to a system including a source end device and at least one receiving end device. The receiving end device is a device for performing picture frame synchronous display, or the receiving end device and the source end device are devices for performing picture frame synchronous display. The method can include the following steps: the source end device sends a first picture frame and a first specified display time corresponding to the first picture frame to each of the receiving end devices; if each of the receiving end devices successfully receives the first picture frame and determines to display the first picture frame according to the first specified display time, each of the devices for performing picture frame synchronous display displays the first picture frame according to the first specified display time; if there is a first receiving end device that does not successfully receive the first picture frame or successfully receives the first picture frame but determines to be unable to display the first picture frame according to the first specified display time among the receiving end devices, each of the devices for performing picture frame synchronous display does not display the first picture frame.

[0045] Thus, the method for synchronizing display of a display picture provided by the embodiments of the present application ensures that the device for synchronously displaying picture frames displays the first picture frame only when each receiving end device can display the first picture frame at the first specified display time; and the device for synchronously displaying picture frames does not display the first picture frame as long as there is a first receiving end device that cannot display the first picture frame at the first specified display time. The synchronization display of the display picture is ensured.

[0046] In a possible implementation, each of the devices for synchronously displaying picture frames includes loss frame information, which is used to record the identification of the picture frame that is not displayed; and the step of each of the devices for synchronously displaying picture frames not displaying the first picture frame when there is a first receiving end device that does not successfully receive the first picture frame or successfully receives the first picture frame but determines that the first picture frame cannot be displayed at the first specified display time, specifically includes: when there is a first receiving end device that does not successfully receive the first picture frame or successfully receives the first picture frame but determines that the first picture frame cannot be displayed at the first specified display time, the first receiving end device adds the identification of the first picture frame to the loss frame information of the first receiving end device, and notifies the other devices of the devices for synchronously displaying picture frames of the identification of the first picture frame; and each of the devices for synchronously displaying picture frames does not display the first picture frame according to the identification of the first picture frame in the loss frame information.

[0047] The identification of the first picture frame in the loss frame information indicates that there is a first receiving end device that cannot display the first picture frame, and thus the device for synchronously displaying picture frames does not display the first picture frame in this case, which is helpful to realize the synchronization display of the display picture. In some scenarios, each of the devices for synchronously displaying picture frames includes the loss frame information, which can be understood as that the loss frame information is maintained in each device, for example, the loss frame information is allocated with storage space, the loss frame information is dynamically updated, and the like.

[0048] In a possible implementation, the method further includes the following step: the source end device determines the first specified display time according to a current transmission delay and / or a current decoding delay; wherein the current transmission delay is related to the actual transmission delay of the historical picture frames, and the current decoding delay is related to the actual decoding delay of the receiving end device decoding the historical picture frames, and the historical picture frames include at least one picture frame whose identification is before the identification of the first picture frame.

[0049] In some embodiments, the source device can determine a current time delay according to the current transmission time delay and / or the current decoding time delay, and then determine the first specified display time of the first picture frame according to the determined current time delay. The current time delay is not a pre-set fixed value, but a dynamically changing value associated with the current network transmission state and / or the decoding capability of the current sink device. Thus, the first specified display time of the first picture frame specified by the source device is a suitable value, which allows the sink device to display the first picture frame as much as possible, and helps to avoid the situation that the sink device cannot receive and / or display some picture frames. Moreover, with the mechanism of dynamically changing the current time delay, in some embodiments, the time delay of the first frame can not be set to a relatively large value (e.g., 200 ms, 600 ms), but can be set to a relatively small value, which helps to reduce the start-up time delay and improve user experience.

[0050] In a possible implementation, before the step of determining the first specified display time according to the current transmission time delay and / or the current decoding time delay by the source device, the method further includes the following steps: determining, by the sink device, the current transmission time delay of the sink device according to the actual transmission time delays of the historical picture frames received by the sink device, and / or determining the current decoding time delay of the sink device according to the actual decoding time delays of the historical picture frames received by the sink device; sending, by the sink device, the determined current transmission time delay of the sink device and / or the determined current decoding time delay of the sink device to the source device; and the step of determining the first specified display time according to the current transmission time delay and / or the current decoding time delay by the source device specifically includes: determining, by the source device, the maximum value of the received at least one current transmission time delay of the sink device as the current transmission time delay, and / or determining the maximum value of the received at least one current decoding time delay of the sink device as the current decoding time delay.

[0051] As described above, the first specified display time is determined by the source device according to the current transmission time delay and / or the current decoding time delay, and the current decoding time delay and / or the current transmission time delay is sent by the sink device to the source device. Determining the maximum value of the current decoding time delay and / or the current transmission time delay helps the first specified display time specified by the source device to be a suitable time as much as possible, and helps to ensure that all sink devices can receive and display the first picture frame as much as possible.

[0052] In a possible implementation, the method further includes the following step: the source device and the sink device perform clock synchronization. Due to inherent physical properties of hardware devices, there is usually an inevitable clock offset between the system time of the source device and the system time of the sink device. The clock synchronization can enable the sink device to obtain and save the clock offset. In turn, the sink device can display the first picture frame at the first specified display time after correction using the clock offset, which helps to avoid picture frame asynchronization caused by the clock offset.

[0053] In a possible implementation, the source device and the sink device are in the same local area network. For example, they are in the same WiFi local area network, access the same router, or are in the same Mesh network. Thus, the sink device and the source device can conveniently exchange information.

[0054] The fourth aspect of the embodiments of the present application provides a sink device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor is configured to enable the sink device to implement the method in the first aspect or any possible implementation manner of the first aspect when executing the computer program.

[0055] The fifth aspect of the embodiments of the present application provides a source device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor is configured to enable the source device to implement the method in the second aspect or any possible implementation manner of the second aspect when executing the computer program.

[0056] The sixth aspect of the embodiments of the present application provides a display picture synchronization system, which includes a source device and at least one sink device. The source device and the sink device are respectively configured to perform the steps in the third aspect or any possible implementation manner of the third aspect.

[0057] The seventh aspect of the embodiments of the present application provides a computer readable storage medium configured to store a computer program. When the computer program is executed by a processor, the method in the first aspect or any possible implementation manner of the first aspect is implemented, or the method in the second aspect or any possible implementation manner of the second aspect is implemented.

[0058] An eighth aspect of an embodiment of the present application provides a computer program product configured to, when running on a receiving end device, cause the receiving end device to perform the method according to the first aspect or any possible implementation manner of the first aspect, or, the computer program product is configured to, when running on a source end device, cause the source end device to perform the method according to the second aspect or any possible implementation manner of the second aspect.

[0059] A ninth aspect of an embodiment of the present application provides a chip system, comprising a memory and a processor, the processor being configured to execute a computer program stored in the memory to implement the method according to the first aspect or any possible implementation manner of the first aspect, or, implement the method according to the second aspect or any possible implementation manner of the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 is a schematic diagram of an application scenario of display screen synchronization provided by an embodiment of the present application;

[0061] Figure 2 is another schematic diagram of an application scenario of display screen synchronization provided by an embodiment of the present application;

[0062] Figure 3 is a schematic diagram of a multi-screen system provided by an embodiment of the present application;

[0063] Figure 4 is a schematic diagram of a specified display time provided by an embodiment of the present application;

[0064] Figure 5 is a schematic diagram of interaction between a display screen synchronization method and a device provided by an embodiment of the present application;

[0065] Figure 6 is a schematic diagram of a message M1 provided by an embodiment of the present application;

[0066] Figure 7 is another schematic diagram of a specified display time provided by an embodiment of the present application;

[0067] Figure 8 is a flowchart of a method for a receiving end device to acquire and process a lost frame number provided by an embodiment of the present application;

[0068] Question 9 is a flowchart of a method for a receiving end device to acquire and process an actively discarded frame number provided by an embodiment of the present application;

[0069] Figure 10 is a flowchart of a method for a receiving end device to synchronously display a display screen provided by an embodiment of the present application. DETAILED DESCRIPTION

[0070] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, circuits, and

[0071] It will be understood that the terms "comprises" and / or "comprising," when used in this specification, include the presence of one or more features, integers, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0072] It will be understood that the term "and / or," when used in the specification and in the following claims, refers to one and / or a combination of the associated listed items.

[0073] As used in this specification and claims, the terms "if" and "when" can be interpreted to mean "upon determination" or "in response to a determination" or "upon detection" or "in response to a detection," depending on the context. Similarly, the phrase "if determined" or "if detected [a described condition or event]" can be interpreted to mean "upon determination" or "in response to a determination" or "upon detection" or "in response to a detection," depending on the context.

[0074] In addition, the terms "first," "second," "third," etc. are used herein for purposes of description and are in no way intended to indicate or imply relative importance or degree of

[0075] The description in the specification of "one embodiment," "some embodiments," and the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. Thus, the appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in additional embodiments," and so on, in various places in the specification are not necessarily all referring to the same embodiment, unless otherwise specified. The terms "including," "containing," "having," and variations thereof are meant to encompass the terms "including," but not limited to," "comprising," and "consisting of," unless otherwise specified.

[0076] In the present application, the terms "picture", "picture frame", "display picture", "display content", "display picture frame", "frame" and the like can be understood as having the same meaning, referring to the same thing, in most cases, unless otherwise specifically emphasized.

[0077] In the present application, the "frame number" is used as a specific example of "frame identifier", but this does not constitute a limitation on the solutions provided by the embodiments of the present application. Those skilled in the art can replace the frame number in the example with any other type of frame identifier without going beyond the scope of the solutions provided by the embodiments of the present application.

[0078] First, the application scenario of the display picture synchronization method, system and electronic device provided by the embodiments of the present application is introduced in conjunction with the drawings.

[0079] Figure 1 An application scenario diagram of the display picture synchronization provided by the embodiments of the present application is exemplarily shown. Figure 1 The scenario shown is the display picture synchronization of a spliced screen, which can also be called a split screen, a combined screen, etc., and can be usually used in scenarios requiring large-area display, such as outdoor or shopping mall billboards, airport flight information display, etc. The spliced screen can include multiple display devices, such as 4, 6, 9, etc., each of which displays a part of the picture, and the display pictures of the multiple display devices are spliced to form a complete display picture. For example, Figure 1 As shown, the spliced screen includes display device A, display device B, display device C and display device D, which respectively display Figure 1 the upper left part, the upper right part, the lower left part and the lower right part of the display picture shown in FIG. 1B. In order to ensure that the pictures displayed by the multiple display devices in the spliced screen at a certain moment can form a complete display picture, the multiple display devices in the spliced screen need to be synchronized in display picture.

[0080] Figure 2 Another application scenario diagram of the display picture synchronization provided by the embodiments of the present application is exemplarily shown. Figure 2 The scenario shown is the display picture synchronization of multiple display devices displaying the same picture, which can be usually applied to scenarios such as teaching, conference, etc. For example, Figure 2 As shown, display device E and display device F need to display the same picture at the same moment, so display device E and display device F need to be synchronized in display picture.

[0081] Based on Figure 1 and Figure 2 , Figure 3 A multi-screen system diagram provided by the embodiments of the present application is exemplarily shown. The multi-screen system 300 can include a spliced screen, and / or one or more display devices.

[0082] In other words, the multi-screen system 300 described in this application embodiment may only include a splicing screen, wherein the splicing screen may include at least two display devices, each of the at least two display devices displaying a portion of the display screen, and the display screens of each display device are spliced ​​together to form a complete display screen; or, it may only include one or more display devices, which display the same display screen; or, it may include both a splicing screen and one or more display devices displaying the same display screen. The multi-screen system 300 described in this application embodiment can be any system that includes at least two display devices and requires display screen synchronization.

[0083] To achieve synchronized display, as described in the background section, a hardware solution is typically used, transmitting the display image via a wired connection. For example, in a multi-screen system, any one display device can be selected as the source device, and the others as sink devices. The source device acquires the display image, for example, from a network, local storage, or other devices, and sends the acquired image to the sink device; the sink device then displays the image upon receiving it.

[0084] by Figure 1 Taking the illustrated scenario as an example, display device A can be used as the source device, and display devices B, C, and D as receiving devices. Display device A can then be wired to display devices B, C, and D respectively, transmitting the display image to each device. Since the transmission latency of wired connections is typically very small and negligible, time synchronization is unnecessary; display devices B, C, and D can directly decode and display the received content. However, achieving display image synchronization via wired connections requires additional hardware costs and involves complex wiring.

[0085] Of course, in some implementations, all display devices in a multi-screen system can function as receiving devices. These devices connect to a source device outside the multi-screen system and receive the displayed image from it. In this case, the source device doesn't necessarily need to synchronize its display with the display devices in the multi-screen system; instead, it functions as a display provider. However, if this implementation uses a wired hardware transmission scheme, it will also suffer from the problems of increased hardware costs and complex wiring, as mentioned above.

[0086] Therefore, the embodiment of the present application provides a display picture synchronization method realized by software: a source device specifies a display time of each frame of display picture, and a receiving device displays each frame of display picture according to the display time of each frame of display picture specified by the source device. In order to facilitate description, in the embodiment of the present application,

[0087] The display time of the i th frame of display picture specified by the source device is referred to as "specified display time" and is denoted as playTime i , where i is used to represent a frame number, i = 1, 2, …, M. For example, the specified display time of the 1 st frame of display picture is denoted as playTime 1, the specified display time of the 2 nd frame of display picture is denoted as playTime 2, and so on.

[0088] It should be understood that, in order to facilitate implementation, the source device can specify the display time of each frame according to a system time of the source device itself. In some embodiments, the system time of the receiving device can have a clock deviation from the system time of the source device, and then the receiving device can correct the specified display time provided by the source device according to the clock deviation, so that the receiving device displays each frame of display picture according to the corrected specified display time to realize display picture synchronization. The display of the receiving device according to the specified display time described in the embodiment of the present application can refer to the display of the receiving device according to the corrected specified display time.

[0089] The time when the source device acquires the 1 st frame of display picture is denoted as startTime. For example, it can be the system time of the source device when the source device acquires the 1 st frame of display picture from the network, acquires the 1 st frame of display picture locally, or acquires the 1 st frame of display picture from another device.

[0090] The delay time of the i th frame of display picture relative to the 1 st frame of display picture is denoted as pts i , where "pts" can represent a presentation time stamp.

[0091] Suppose that the display frame rate is 60 frames per second (fps), then the time interval between adjacent two frames of display picture is 1 / 60 ≈ 16.67 milliseconds (ms). The pts 1 corresponding to the 1 st frame of display picture is 0, so there is pts 2 corresponding to the 2 nd frame of display picture = 16.67 × 1 = 16.67, pts 3 corresponding to the 3 rd frame of display picture = 16.67 × 2 = 33.34 ms, …, and pts i = 16.67 × (i-1) ms, and so on.

[0092] Therefore, the i th frame of display picture should be theoretically displayed at startTime+pts iThe time is displayed. For ease of description, startTime + pts i This is often referred to as the "theoretical display time." In some embodiments, the source device can use the theoretical display time as the specified display time, i.e., playTime. i =startTime+pts i .

[0093] However, in practical applications, from the time the source display device acquires the i-th frame of the display image until the receiving display device can display it, the transmission and decoding of the i-th frame still require time. If the source and receiving display devices transmit the display content wirelessly, the transmission time will also be affected by the wireless network transmission status, resulting in some fluctuations.

[0094] Therefore, in the display synchronization method provided in this application embodiment, the source display device can add a delay (denoted as delayTime) to the theoretical display time to represent the delayed display time of the display screen. This serves as a reserve margin to ensure that the receiving display device can acquire and decode the display content before the specified display time. This avoids situations where the specified display time arrives but the receiving display device has not yet received the display screen or has not completed decoding the display screen. That is, playTime. i =startTime+pts i +delayTime.

[0095] like Figure 4 As shown, the time when the source device acquires the first frame of the display is startTime. Then, the specified display time for the first frame of the display is playTime1 = startTime + pts1 + delayTime. The specified display time for the second frame of the display is playTime2 = startTime + pts2 + delayTime. The specified display time for the third frame of the display is playTime3 = startTime + pts3 + delayTime, and so on.

[0096] In some embodiments, to address the issue of unstable network transmission, the delayTime can be set to a relatively large value, such as 200ms or 600ms. This ensures that the display synchronization is not affected as much as possible when network transmission fluctuates.

[0097] However, this implementation that uses the same delayTime for each frame still has a long start-up delay, which affects the user experience. That is, for example, there is a large, user-perceptible delay between the time the user presses the start button and the time the display device in the multi-screen system begins to display the first frame, which affects the user experience.

[0098] In addition, if the network transmission state between different receiving end devices and the source end device changes inconsistently, the display screens may still not be synchronized. For example, in the scenario shown in FIG. 1, assume that display device A is the source end device, and display devices B, C, and D are receiving end display devices. Assume that the network transmission state between display device B and display device A and between display device C and display device A is normal, and the network transmission state between display device D and display device A is poor. In this case, display devices B and C may be able to display the ith frame at the specified display time, but display device D may not be able to display the ith frame at the specified display time. In this case, display device D can only remain at the (i-1)th frame, which causes the display screens to be unsynchronized. Figure 1

[0099] Therefore, embodiments of the present application provide another method for synchronizing display screens by using software:

[0100] Each receiving end device maintains drop frame information, which records the frame numbers of frames that are not displayed. For example, the drop frame information can be in the form of a “drop frame list” (denoted as dropList j ). The drop frame information can also be in other forms, such as a drop frame pool, a drop frame queue, a drop frame linked list, and the like, which are not limited in the present application. For ease of description, embodiments of the present application mainly use the drop frame list as an example for description.

[0101] When a receiving end device determines that a frame cannot be displayed, and / or determines that a frame cannot be displayed at the specified display time (denoted as playTime i ), the receiving end device adds the frame number to its own drop frame list and notifies other receiving end devices to also add the frame number to their drop frame lists, so that the drop frame lists maintained by each receiving end device are the same. Each receiving end device does not display the frames recorded in the drop frame list and only displays the frames not recorded in the drop frame list. As long as one receiving end device cannot display a frame, and / or cannot display a frame at the specified display time, other receiving end devices also do not display the frame. This avoids the situation in which different receiving end devices display different frames at the same time, and achieves synchronization of the display screens.

[0102] ​In some embodiments, if a certain receiving end device fails to successfully receive a frame due to a failure in the transmission process, the receiving end device determines that the frame cannot be displayed.

[0103] In some embodiments, a certain receiving end device can estimate (or predict) the estimated decoding time delay (denoted as decodeTime i_Sj_pred ) of a frame according to the decoding time delay of historical frames, and further determine the estimated display time of the frame. If the estimated display time exceeds the specified display time of the frame, the receiving end device determines that the receiving end device cannot display the frame at the specified display time.

[0104] In some embodiments, the specified display time can be determined by the source end device for each frame display picture according to the current time delay (denoted as delayTime i ). The current time delay is dynamically changed, rather than a pre-set fixed value.

[0105] In some embodiments, the current time delay can be determined by the source end device according to the current transmission time delay (denoted as transTime Sj ) and the current decoding time delay (denoted as decodeTime Sj ) sent by the receiving end device to the source end device.

[0106] In some embodiments, the current transmission time delay and the current decoding time delay sent by the receiving end device to the source end device can be estimated by the receiving end device according to the transmission time delay of historical frames and the decoding time delay of historical frames received by the receiving end device.

[0107] Thus, the display picture synchronization method provided by the embodiments of the present application realizes the synchronization of display pictures by maintaining the same frame loss list by the receiving end device and determining the specified display time of each frame by the source end device according to the dynamically changed current time delay. The display picture synchronization method provided by the embodiments of the present application does not need to increase additional hardware cost, can realize the synchronization of display pictures by using a pure software method, is easy to implement and has good user experience.

[0108] Next, the display picture synchronization method provided by the embodiments of the present application will be described in detail in combination with the accompanying drawings.

[0109] Figure 5 An exemplary display picture synchronization method device interaction diagram provided by the embodiments of the present application is shown.

[0110] The method involves a source end device S0 and at least one receiving end device S [1-N] . For ease of description, any one of the receiving end devices S [1-N] in the embodiments of the present application is denoted as S j, where j represents the receiver device number, j = 1, 2, ..., N.

[0111] For example, in Figure 1 In the scenario shown, display device A can be designated as source device S0, and display devices B, C, and D can be designated as receiving devices S1, S2, and S3, respectively; source device S0 and receiving device S1... [1 -3] Synchronization of the displayed screen is required. Of course, in some other implementations, in Figure 1 In the scenario shown, display devices A, B, C, and D can also be designated as receiving devices S1, S2, S3, and S4, respectively, while another device outside the multi-screen system can be designated as the source device S0; receiving device S... [1-4] Synchronous display of the displayed screen is required. It should be understood that this application embodiment does not limit whether the source device S0 must perform synchronous display of the displayed screen.

[0112] like Figure 5 As shown, the method may include steps 501 to 509, specifically:

[0113] Step 501, Source device S0 and receiver device S [1-N] Establish a connection.

[0114] In some embodiments, source device S0 and receiver device S [1-N] They are located within the same local area network (LAN), which can be wired or wireless. For example, source device S0 and receiver device S... [1-N] Both are connected to the same router. Source device S0 and receiver device S... [1-N] The established connection could be, for example, a WiFi connection.

[0115] In some embodiments, the source device S0 can be connected to each receiver device S0 respectively. j Establish a dedicated transmission channel, and communicate with the receiving device S through this dedicated transmission channel. j Data interaction is performed; in some embodiments, the source device S0 and the receiver device S0 interact. [1-N] They form a mesh network and exchange data by broadcasting within the mesh network.

[0116] It should be understood that the embodiments of this application do not limit the source device S0 and the receiver device S0. [1-N] The methods and types of connections established.

[0117] Step 502, Source device S0 and receiver device S [1-N] Perform clock synchronization.

[0118] Electronic devices typically have their own system time set, and they execute designated functions based on this system time. It is usually unavoidable that there will be some discrepancy between the system times of different electronic devices.

[0119] Therefore, in some embodiments, the source device S0 and the receiver device S [1-N] Clock synchronization can be performed to determine the S of each receiving device. j The clock deviation between the system time of the source device S0 and the system time of the source device S0 is denoted as offset in this embodiment. Sj .

[0120] For example, clock synchronization can be achieved using the Precision Time Protocol (PTP). It should be understood that the embodiments of this application do not limit the source device S0 and the receiving device S... [1-N] What specific method is used for clock synchronization?

[0121] In some embodiments, each receiving device S j The receiving device S can be synchronized via clock. j Clock offset between the system time of the source device S0 and the system time of the source device S0 Sj Recorded on the receiving device S j Locally. Therefore, whenever the receiving device S... j Upon receiving a display frame sent by the source device S0, the clock offset can be adjusted accordingly. Sj This is used to correct the specified display time of the frame so that the display screen can be displayed synchronously according to the corrected specified display time.

[0122] Step 503: Source device S0 sends a message to receiver device S0. [1-N] Send the display screen, and correspondingly, the receiving device S [1-N] Receive the displayed screen.

[0123] In some embodiments, such as Figure 6 As shown, the source device S0 can display the frame number of the i-th frame (denoted as frameID). i The system time of the source device S0 when it sends the frame to display the image (referred to as "send time"). i The source device S0 specifies the display time (playTime) for this frame. i The frame displays the image (referred to as the payload). i The message is packaged into message M1 and sent to each receiving device S. jWhere i represents the frame number, i = 1, 2, ..., M. j represents the receiver device number, j = 1, 2, ..., N.

[0124] It should be understood that Figure 6 This application provides only one example of an implementation method. It does not limit the specific implementation of step 503, nor does it limit the specific order of frame number, transmission time, specified display time, or displayed images in message M1. Message M1 may include... Figure 6 Show more or fewer fields.

[0125] In some embodiments, the specified display time of the i-th frame is playTime. i It can be determined by the source device S0 based on the current delayTime. i , is specifically determined for the display of the i-th frame.

[0126] For example, playTime i The calculation method can be playTime i =startTime+pts i +delayTime i Among them, "startTime" and "pts" i "delayTime" i The specific meaning of "" is the same as described in the previous section, and will not be repeated here.

[0127] like Figure 7 As shown, the time when the source device S0 acquires the first frame of the display is startTime. Then, the specified display time playTime1 for the first frame of the display is playTime1 = startTime + pts1 + delayTime1. The specified display time playTime2 for the second frame of the display is playTime2 = startTime + pts2 + delayTime2. The specified display time playTime3 for the third frame of the display is playTime3 = startTime + pts3 + delayTime3, and so on.

[0128] Figure 7 The implementation shown is the same as... Figure 4 The difference between the implementations shown is that: Figure 4 In the implementation shown, for each frame of the display, the source device S0 uses the same delayTime to determine the specified display time; Figure 7 In the implementation shown, for each frame of the displayed image, the source device S0 determines the time delay based on the dynamically changing current delayTime. i To determine the specified display time.

[0129] In some embodiments, the current delay time is delayTime. i It can be the source device S0 based on the receiver device S [1-N] The current transmission delay (denoted as transTime) S[1-N] ), receiving device S [1-N] The current decoding latency (denoted as decodeTime) S[1-N] It is certain.

[0130] Due to network transmission status, receiving device S [1-N] The processing power fluctuates; therefore, the latency generated during the transmission of each display frame and the S of the receiving device also affect the latency. [1-N] The latency incurred during the decoding and display of this frame changes in real time. For example, if the network transmission status deteriorates, the current transmission latency will increase; if the receiving device S... [1-N] If the processor load increases, the current decoding latency will increase; conversely, the same applies.

[0131] In view of this, in the display screen synchronization method provided in the embodiments of this application, the source device S0 can adjust the display screen synchronization method according to the real-time changing current transmission delay transTime. S[1-N] Current decoding latency decodeTime S[1-N] To determine the current delayTime for the display of the i-th frame. i This enables the source device S0 to specify the display time (playTime) for the i-th frame of the display screen. i It can dynamically adapt to the current latency, which helps to avoid the occurrence of display screen desynchronization.

[0132] In some embodiments, the source device S0 can receive data from N receiving devices S0. [1-N] Current transmission delay transTime S[1-N] Determine the current transmission delay transTime for the i-th frame of the display. i .

[0133] In one possible implementation, specifically, the current transmission delay of the i-th frame can be calculated as: transTime i =max(transTime) S1 ,…,transTime SN ), where transTime Sj (j = 1, ..., N) is used to represent the receiving device S. j The receiving device S sent to the source device S0 jThe current transmission delay. In some embodiments, transTime Sj It can be the receiving device S j It is obtained by fitting and estimating the transmission delay of received historical frames. It is used to calculate transTime. Sj Some specific implementation methods will be described in detail in step 505 later, and will not be repeated here. That is to say, in this implementation method, the source device S0 can receive data from N receiving devices S... [1-N] It sends to N receiving devices S [1-N] Current transmission delay transTime S[1-N] Then, the maximum value is taken as the current transmission delay transTime for the i-th frame of the display. i .

[0134] In some embodiments, the source device S0 can receive data from N receiving devices S0. [1-N] Current decoding latency decodeTime S[1-N] Determine the current decoding delay decodeTime for the i-th frame of the display. i .

[0135] In one possible implementation, specifically, the current decoding latency of the i-th frame can be calculated as: decodeTime i =max(decodeTime) S1 ,…,decodeTime SN ). Among them, decodeTime Sj (j = 1, ..., N) is used to represent the receiving device S. j The receiving device S sent to the source device S0 j The current decoding latency. In some embodiments, decodeTime Sj It can be the receiving device S j It is obtained by fitting and estimating the decoding latency of received historical frames. It is used to calculate decodeTime. Sj Some specific implementation methods will be described in detail in step 508 later, and will not be repeated here. That is to say, in this implementation method, the source device S0 can receive data from N receiving devices S... [1-N] It sends to N receiving devices S [1-N] Current decoding latency (decoeTime) S[1-N] Then, the maximum value is taken as the current decoding delay decodeTime for the i-th frame of the display. i .

[0136] It should be understood that the embodiments of the present application do not limit the specific implementation manner of the source device S0 determining the current transmission delay transTime i , current decoding delay decodeTime i for the i-th frame of display picture.

[0137] In some embodiments, for the 1st frame of display picture, the source device S0 can set the delayTime1 to a small value according to experience or tentatively, for example, 30 ms, 60 ms. Subsequently, the delayTime [1-N] for the subsequent frames of display picture can be dynamically adjusted according to whether the N sink devices S [1-N] can successfully display the picture frames, and / or according to the current transmission delay delayTime S[1-N] , current decoding delay decodeTime S[1-N] fed back by the N sink devices S i . For example, if the sink device S j determines that it cannot display the 1st frame at the specified display time playTime1, then none of the N sink devices S [1-N] will display the 1st frame, and the source device S0 can set the delayTime2 for the 2nd frame to be slightly larger than the delayTime1 for the 1st frame, so as to make the N sink devices S [1-N] as far as possible to display the 2nd frame.

[0138] The display picture synchronization method provided by the embodiments of the present application can set the delayTime1 for the 1st frame of display picture to a small value, and the delayTime i for the subsequent frames can be dynamically adjusted according to actual conditions; and it is not necessary to use a large fixed value (for example, 200 ms, 600 ms) as the delayTime for all the frames of display picture (including the 1st frame) as in the foregoing embodiment scheme; thus, the display picture synchronization method provided by the embodiments of the present application can guarantee display picture synchronization, and also solves the problem of large start-up delay in the foregoing embodiment scheme.

[0139] In implementation, the value of the current delay delayTime i usually needs to be greater than handleTime+transTime i +decodeTime iHere, handleTime represents the processing time for one display frame. Since it has relatively little fluctuation, it can be set to a fixed value, such as 10ms. The processing time can refer to the time consumed by the source device S0 in processing the display frame, such as the time consumed in processes like encrypting the display frame, encapsulating the message containing the display frame according to the adopted transmission protocol, calling encryption software, and calling message encapsulation software.

[0140] In some embodiments, exemplarily, the source device S0 has a current delay time delayTime i The specific calculation method can be: delayTime i =handleTime + transTime i +decodeTime i +2×framePlayTime.

[0141] `framePlayTime` represents the duration of a single frame, which is the reciprocal of the frame rate. For example, if the frame rate is 60 frames per second, then `framePlayTime` = 1 / 60 ≈ 16.67 ms. Doubling `framePlayTime` as a buffer improves the fault tolerance for display synchronization. Of course, this is just an example and not a limitation. In implementation, any value (e.g., 20ms, 40ms) can be set as a buffer; it doesn't have to be an integer multiple of `framePlayTime`.

[0142] Step 504, Receiver device S [1-N] Each receiving device S in j Each device sends its lost frame number to other receiving devices. Optionally, the lost frame number can also be sent to the source device S0.

[0143] In some embodiments, such as Figure 5 As shown, each receiving device S j The device can package its lost frame number into message M2 and send it to other receiving devices. Optionally, message M2 can also be sent to the source device S0.

[0144] In this embodiment of the application, the lost frame number is used to indicate the sequence number of the display frame that the receiving device failed to receive due to a fault during transmission.

[0145] For example, receiver device S jIt can determine whether it has failed to receive the display of a certain frame number, identify its lost frame number, add its lost frame number to its own lost frame list, and notify other devices that are synchronizing their display to also add the lost frame number to their lost frame lists.

[0146] If any one of the devices performing synchronized display fails to receive a frame, then none of the devices performing synchronized display will display that frame, regardless of whether they successfully received the frame.

[0147] Therefore, the method provided in this application embodiment can, through step 504, avoid the situation where the display screen is out of sync due to a failure in the transmission process causing one or more receiving devices to fail to receive one or more frames.

[0148] In some embodiments, such as Figure 8 As shown, one specific implementation of step 504 can be:

[0149] Step 5041, at the receiving device S j After receiving message M1, the receiving device S j The frame number (framID) of the displayed frame can be obtained from message M1. i .

[0150] Step 5042, Receiver device S j Determine if the frame numbers are consecutive. If the frame numbers are consecutive, proceed to step 5046; if the frame numbers are not consecutive, proceed to step 5043.

[0151] For example, suppose the receiving device S j If the last received frame number was 4 and the current received frame number is 5, then the frame numbers are consecutive; assuming the receiving device S... j If the last received frame number was 4 and the current received frame number is 6, then the frame numbers are not consecutive.

[0152] It should be understood that the above description is only an example for determining whether frame numbers are consecutive, and not a limitation. If the display frame uses a preset frame sorting or frame transmission algorithm during transmission, the receiving device Sj can combine the preset frame sorting or frame transmission algorithm to determine whether the frame numbers are consecutive.

[0153] Step 5043, Receiver device S j Calculate the lost frame number.

[0154] In some embodiments, the receiving device S j The lost frame number is calculated based on the historically received frame numbers. For example, assume the receiving device S... jThe historical received frame numbers are 1, 2, 3, and 5 respectively, and the receiving end device Sj calculates that the display picture of frame number 4 is lost.

[0155] It should be understood that the above description is only an example of calculating the lost frame number, and is not limiting. If a preset frame ordering or frame transmission algorithm is used in the transmission of the display picture, and / or a preset lost frame retransmission algorithm is used, the receiving end device Sj can combine the preset frame ordering or frame transmission algorithm, and the preset lost frame retransmission algorithm to calculate the lost frame number.

[0156] Step 5044, the receiving end device Sj j adds the lost frame number to its own lost frame list dropList j .

[0157] In some embodiments, each receiving end device Sj j maintains its own lost frame list dropList j to record the frame numbers of the display pictures that are not displayed. In some embodiments, each receiving end device Sj j may create the lost frame list dropList j before step 503. The dropList j may be implemented in the form of an array, a vector, a list, a linked list, or other computer data structures, and the embodiments of the present application do not limit the specific implementation. Thus, in step 5044, the receiving end device Sj j may add the calculated lost frame number to its own lost frame list dropList j . For example, the lost frame number can be added to the dropList j by appending, inserting, updating, or other methods.

[0158] Step 5045, the receiving end device Sj j sends the lost frame number to other receiving end devices; correspondingly, after receiving the lost frame number of the receiving end device Sj j sent by the receiving end device Sj j , the other receiving end devices also add the lost frame number to their own lost frame list dropList j’ .

[0159] Optionally, in step 5045, the receiving end device Sj j may also send the lost frame number to the source end device S0. For example, in the case where the source end device S0 also performs synchronized display of the display picture, the source end device S0 also maintains its own lost frame list dropList0, and the receiving end device Sj jAlternatively, the lost frame number can be sent to the source device S0; accordingly, after receiving the lost frame number, the source device S0 adds these lost frame numbers to its lost frame list dropList0.

[0160] Step 5046, Receiver device S j frame ID i The corresponding display payload i The data is sent to the decoder. The decoder can be an instrument S installed at the receiving end. j The display decoding software can be preset with the required decoding algorithms for decoding the display screen.

[0161] Step 505, Receiver device S [1-N] Each receiving device S in j Each determines its current transmission delay, transTime. Sj It then sends its determined current transmission delay to the source device S0.

[0162] In some embodiments, such as Figure 5 As shown, each receiving device S j The current transmission delay can be packaged into a message M3 and sent to the source device S0.

[0163] In some embodiments, the receiving device S j After receiving message M1, the receiving device S j The sendTime of the i-th frame can be obtained from message M1. i .

[0164] As mentioned earlier, the sendTime event time is used to send the message. i This is the system time of source device S0 when it packages and sends message M1. Also, because the receiving device S... j There may be a clock offset between the system time of the source device S0 and the system time of the source device S0. Sj .

[0165] Therefore, in one implementation, the receiving device S j The display can be based on the clock offset for the i-th frame. Sj and receiving device S j When the i-th frame of the display is received, the receiving device S j System time (denoted as rcvTime) i_Sj The calculation shows that the i-th frame of the display image is sent from the source device S0 to the receiving device S0. j The actual transmission delay (denoted as transTime)i_Sj ):

[0166] That is, transTime i_Sj =rcvTime i_Sj -offset Sj -sendTime i .

[0167] The actual transmission delay transTime i_Sj This can be understood as the i-th frame of the display being sent from the source device S0 to the receiving device S0. j The actual time consumed on the transmission path.

[0168] In some embodiments, the receiving device S j The receiver device S can record the actual transmission delay of the displayed images it has received historically. Therefore, the receiver device S... j Based on the actual transmission delays of these historically received display images, the current transmission delay (transTime) can be estimated (or predicted) to determine its own. Sj .

[0169] Specifically, in one possible implementation:

[0170] (1) When the receiving device S j When the historical cumulative number of received frames is less than a preset number (e.g., 30), the receiving device Sj directly uses the actual transmission delay transTime of the i-th frame to display the image. i_Sj_real As the receiving device S j Its own current transmission delay transTime Sj ;

[0171] (2) When the receiving device S j When the historical cumulative number of received frames is greater than or equal to a preset number (e.g., 30), the receiving device S... j The actual transmission delay of these historical frames is fitted, for example, by using the least squares method for linear fitting, to obtain the receiver device S. j Fitted transmission delay transTime Sj_fitting and frame number framID i Linear relationship between them: transTime Sj_fitting = a×frameID i +b. Here, a and b are hyperparameters, which can be calculated through linear fitting. a represents the slope of the fitted linear function in the coordinate system, and b represents the intercept. Thus, the frame number (frameID) of the i-th frame is determined. i Substitute transTime Sj_fitting= a×frameID i +b allows us to calculate the fitted transmission delay transTime. Sj_fitting (frameID i As the receiving device S j Its own current transmission delay transTime Sj .

[0172] In addition to the receiver device S mentioned above j Determine your current transmission delay (transTime) Sj Alternatively, it can be determined by calculating the average or mode of the actual transmission delay of historical frames. It should be understood that the embodiments of this application do not limit the receiving display device S. j Determine your current transmission delay (transTime) Sj The specific implementation method.

[0173] By executing step 505, the receiving device S [1-N] Each receiving device S in j Each will determine its own current transmission delay transTime Sj The data is sent to the source device S0, thereby allowing the source device S0 to obtain information from N receiving devices S. [1-N] The N current transmission delays transTime sent to it S[1-N] .

[0174] In the next loop (i.e., during the synchronous display of the next frame), when step 503 is executed next time, as described above, the source device S0 can, based on the current N receiving devices S... [1-N] It sends to N receiving devices S [1-N] Their respective current transmission delay transTime S[1-N] Determine the current transmission delay transTime for the displayed image in the (i+1)th frame. i+1 For example, get transTime S[1-N] The maximum value in.

[0175] Furthermore, the source device S0 can determine the current transmission delay transTime for the (i+1)th frame of the display screen. i+1 Determine the specified display time (playTime) for the (i+1)th frame. i+1 .

[0176] Therefore, in the display synchronization method provided in this application embodiment, the source device S0 can synchronize the display of the i-th frame according to the dynamically changing current transmission delay transTime. iDetermine the specified display time (palyTime) i .

[0177] It should be understood that the execution order of step 505 is not limited in the embodiments of this application. In some embodiments, step 505 only needs to be completed before the next execution of step 503.

[0178] Step 506, Receiver device S [1-N] Each receiving device S in j Each device sends its own actively discarded frame number (if it exists) to other receiving devices. Optionally, the actively discarded frame number can also be sent to the source device S0.

[0179] In some embodiments, such as Figure 5 As shown, each receiving device S j The user can package their own actively discarded frame number into message M4 and send it to other receiving devices. Optionally, message M4 can also be sent to the source device S0, for example, if the source device S0 is also displaying the screen synchronously.

[0180] In this embodiment of the application, the actively discarded frame number is used to indicate the sequence number of the display frame that the receiving device cannot display at the specified display time because the estimated display time exceeds the specified display time.

[0181] For example, receiver device S j It is possible to estimate (or predict) the decoding delay required for the currently received i-th frame of the display image (this is called "estimated decoding delay", denoted as decodeTime). i_Sj_pred Then, based on the estimated decoding delay, the estimated display time of the current i-th frame is obtained. If the estimated display time exceeds the specified display time designated by the source device S0 for the i-th frame, the receiving device S... j It can be reasonably assumed that even if the i-th frame of the display is decoded this time, the decoding completion time may have already exceeded the designated display time, making it too late to display. Therefore, the receiving device S j The frame needs to be actively discarded, and its frame number should also be sent to other receiving devices.

[0182] If any one of the devices performing synchronized display fails to display the frame at the specified display time, then none of the devices performing synchronized display will display the frame.

[0183] Therefore, the method provided in this application embodiment can avoid the situation where the display screen is out of sync due to one or more receiving devices failing to successfully display one or more frames at a specified display time through step 506.

[0184] In some embodiments, considering that the current i-th frame display has been sent to the decoder after step 504 is completed, the decoder can be the execution subject of step 506.

[0185] In some embodiments, such as Figure 9 As shown, one specific implementation of step 506 can be:

[0186] Step 5061, Receiver device S j Estimate the payload of the current i-th frame. i The required decoding latency (i.e., the "estimated decoding latency") is denoted as decodeTime. i_Sj_pred ).

[0187] In some embodiments, the receiving device S j The receiver device S can record the actual decoding latency of the displayed images it has received historically. Therefore, the receiver device S... j The decoding latency required for the current i-th frame of display can be estimated based on the actual decoding latency of these historically received display frames.

[0188] Specifically, in one possible implementation:

[0189] (1) When the receiving device S j When the cumulative number of historically received frames is less than a preset number (e.g., 30), the receiving device Sj directly uses the actual decoding delay of the (i-1)th frame as the estimated decoding delay (decodeTime) of the current i-th frame. i_Sj_pred ;

[0190] (2) When the receiving device S j When the historical cumulative number of received frames is greater than or equal to a preset number (e.g., 30), the receiving device S... j Based on the actual decoding latency of these historical display images, a fitting method is used, for example, by performing linear fitting using the least squares method, to obtain the receiver device S. j The fitting decoding delay decodeTime Sj_fitting and frame number framID i Linear relationship between them: decodeTime Sj_fitting = c×frameID i +d. Here, c and d are hyperparameters, which can be calculated through linear fitting. c represents the slope of the fitted linear function in the coordinate system, and d represents the intercept. Therefore, the frame number (frameID) of the currently displayed i-th frame is... i Input decodeTime Sj_fitting= c×frameID i +d will give the calculated fitted decoding delay decodeTime Sj_fitting (frameID i As the receiving device S j The estimated decoding delay decodeTime for the current i-th frame of the display. i_Sj_pred .

[0191] In addition to the receiver device S mentioned above j The estimated transmission delay for the i-th frame of display can also be determined by calculating the average or mode of the actual decoding delays of historical frames. It should be understood that the embodiments of this application do not limit the receiving display device S. j Determine the specific implementation method for estimating the transmission delay of the i-th frame of the image.

[0192] Step 5062, Receiver device S j Based on the estimated decoding latency decodeTime i_Sj_pred Determine whether the display time can be specified by `palyTime`. i If decoding is successful, proceed to step 507 to perform decoding; otherwise, proceed to step 5063.

[0193] In some embodiments, if a preset condition is met, the receiving device S j Determine if it can be displayed at the specified time (playTime) i Decoding has been completed previously. For example, the above preset conditions could be:

[0194] systemTime Sj +decodeTime i_Sj_pred -offset Sj <playTime i +framePlayTime.

[0195] Among them, "systemTime" Sj "Used to represent the receiving device S" j The system time; the meanings of other symbols are as described above and will not be repeated here.

[0196] Step 5063, Receiver device S j It is determined that the current frame needs to be actively discarded.

[0197] Since step 5062 determines "no", the receiving device S j It can be reasonably assumed that if the current i-th frame is decoded, the time at which decoding is completed may have already exceeded the specified display time specified by the source device S0. Therefore, the receiving device S...j There is no need to decode the i-th frame of display picture any more, and the current frame needs to be actively discarded.

[0198] Step 5064, receiving end device Sj j The frame number of the current frame is added to the dropList of the receiving end device Sj. j That is, the frame number of the current frame is identified by the receiving end device Sj as an actively discarded frame number.

[0199] This step can be analogized to the description of step 5044 in the foregoing part, and will not be described here.

[0200] Step 5065, receiving end device Sj j The frame number of the current frame (actively discarded frame number) is sent to other receiving end devices; correspondingly, after receiving the actively discarded frame number of the receiving end device Sj sent by the receiving end device Sj, the other receiving end devices also add the actively discarded frame number to the dropList of the other receiving end devices. j The actively discarded frame number of the receiving end device Sj sent by the receiving end device Sj is added to the dropList of the other receiving end devices. j The actively discarded frame number of the receiving end device Sj sent by the receiving end device Sj is added to the dropList of the other receiving end devices. j’ The actively discarded frame number of the receiving end device Sj sent by the receiving end device Sj is added to the dropList of the other receiving end devices.

[0201] Optionally, in step 5065, the receiving end device Sj j may also send the actively discarded frame number to the source end device S0. For example, in the case that the source end device S0 also performs synchronous display of display pictures, the source end device S0 also maintains its own dropList0, and the receiving end device Sj j may also send the actively discarded frame number to the source end device S0; correspondingly, after receiving the actively discarded frame number, the source end device S0 adds the actively discarded frame number to the dropList0 thereof.

[0202] Step 507, receiving end device Sj [1-N] decodes the i-th frame of display picture payload i . Optionally, the source end device S0 may also decode the i-th frame of display picture payload i .

[0203] In some embodiments, a decoder can be installed in each receiving end device Sj j for decoding display pictures according to a preset decoding algorithm.

[0204] In some embodiments, if the source end device S0 also performs synchronous display of display pictures, a decoder can also be installed in the source end device S0, which can also decode the i-th frame of display picture payload iIf the source device S0 only acts as a provider of the display screen and does not perform synchronous display of the display screen, then the source device S0 may not decode the i-th frame display screen payload. i .

[0205] In some embodiments, if step 506 is Figure 9 If implemented in the manner shown, then since step 5062 determines "yes", the receiving device S... j It can be reasonably assumed that if the current i-th frame is decoded, the time at which the decoding is completed is likely before the specified display time specified by the source device S0. Therefore, the i-th frame can be decoded. For example, the i-th frame's payload can be decoded using a preset decoding algorithm in the decoder. i Decode it.

[0206] Step 508, Receiver device S [1-N] Each receiving device S in j Each determines its current decoding latency (decodeTime). Sj It then sends its determined current decoding latency to the source device S0.

[0207] In some embodiments, such as Figure 5 As shown, each receiving device S j The current decoding latency can be packaged into a message M5 and sent to the source device S0.

[0208] In some embodiments, after decoding is completed in step 507, the receiving device S j We can determine the actual decoding time consumed in decoding the i-th frame of the display, which is called the "actual decoding delay" and is denoted as decodeTime. i_Sj_real .

[0209] At this time, the receiving device S j According to the receiving device S j The actual decoding latency of the displayed images it has received throughout its history is recorded in the data (wherein, the actual decoding latency of the i-th frame of the displayed image is decodeTime). i_Sj_real ), estimate the receiving device S j Your current decoding latency (decodeTime) Sj .

[0210] Specifically, in one possible implementation:

[0211] (1) When the receiving device S jWhen the cumulative number of historically received frames is less than a preset number (e.g., 30), the receiving device Sj directly uses the actual decoding delay (decodeTime) of the i-th frame to display the image. i_Sj_real As the receiving device S j Your current decoding latency (decodeTime) Sj ;

[0212] (2) When the receiving device S j When the cumulative number of historically received frames is greater than or equal to a preset number (e.g., 30), the receiving device Sj determines the actual decoding latency of these historical frames (including the actual decoding latency decodeTime of the i-th frame displayed). i_Sj_real To obtain the receiver device S, a fitting method can be used, for example, by performing a linear fitting using the least squares method. j The fitting decoding delay decodeTime Sj_fitting’ and frame number framID i Linear relationship between them: decodeTime Sj_fitting’ =c'×frameID i +d'. Here, c' and d' are hyperparameters, which can be calculated through linear fitting. c' represents the slope of the fitted linear function in the coordinate system, and d' represents the intercept. Therefore, the frame number (frameID) of the next frame, i.e., the (i+i)th frame, is... i+1 Substituting the above linear relationship formula, we can obtain the calculated fitting decoding delay decodeTime. Sj_fitting’ (frameID i+1 As the receiving device S j Your current decoding latency (decodeTime) Sj .

[0213] In addition to the receiver device S mentioned above j Determine your current decoding latency (decodeTime) Sj Alternatively, it can be determined by calculating the average or mode of the actual decoding delay of historical frames. It should be understood that the embodiments of this application do not limit the receiving device S. j Determine your current decoding latency (decodeTime) Sj The specific implementation method.

[0214] Of course, in some other possible implementations, the receiving device S j Alternatively, the estimated decoding delay determined in step 5061 can be used as the decodeTime. i_Sj_pred As the receiving device S j The current decoding latency.

[0215] By executing step 508, the receiving device S [1-N] Each receiving device S in j Each of them will determine its current decoding latency (decodeTime). Sj The data is sent to the source device S0, thereby allowing the source device S0 to obtain information from N receiving devices S. [1-N] The N current decoding delays (decodeTime) sent to it S[1-N] .

[0216] In the next loop (i.e., during the synchronous display of the next frame), when step 508 is executed next time, as described above, the source device S0 can, based on the current N receiving devices S... [1-N] It sends to N receiving devices S [1-N] Their respective current decoding latency (decodeTime) S[1-N] Determine the current decoding delay decodeTime for the displayed image in frame (i+1). i+1 For example, get decodeTime S[1-N] The maximum value in.

[0217] Furthermore, the source device S0 can determine the current decoding latency (decodeTime) for the (i+1)th frame of the displayed image based on the current decoding latency (decodeTime). i+1 Determine the specified display time (playTime) for the (i+1)th frame. i+1 .

[0218] Therefore, the display synchronization method provided in this application embodiment enables the source device S0 to synchronize the display of the i-th frame according to the dynamically changing current decoding delay decodeTime. i Determine the specified display time (palyTime) i .

[0219] It should be understood that the execution order of step 508 is not limited in the embodiments of this application. Step 508 only needs to be completed before the next execution of step 503.

[0220] Step 509, Receiving device S [1-N] Each receiving device S in j The display screen is displayed synchronously. Optionally, the source device S0 can also display the display screen synchronously.

[0221] In some embodiments, after the preceding steps 504 and 506, each receiving device S j dropList of dropped frames jIn this way, the frame numbers of the recorded display pictures which are not displayed are all the same. Therefore, in step 509, each receiving end device Sj can determine whether to display a frame according to its own frame loss list dropList j to realize the synchronous display of the display pictures.

[0222] For example, if the frame number of the frame display picture is in the frame loss list, the frame display picture is not displayed; if the frame number of the frame display picture is not in the frame loss list, the frame display picture is displayed.

[0223] Therefore, the method provided by the embodiment of the present application realizes the synchronous display of the display pictures by maintaining a dynamically updated frame loss list which records the same frame numbers of the display pictures which are not displayed in each device which displays the display pictures synchronously. The synchronous display of the display pictures is realized by a pure software manner, which is easy to implement and has good user experience.

[0224] In some embodiments, as shown in FIG. 5, a specific implementation manner of step 509 can be as follows: Figure 10

[0225] Step 5091, the receiving end device Sj determines whether the frame number frameID j of the current i-th frame display picture payload j has been decoded.

[0226] In some embodiments, as described above, in step 507, the receiving end device Sj decodes the current i-th frame display picture payload i . Therefore, in step 5091, it is determined that the current i-th frame display picture payload i has been decoded, and the decoded i-th frame display picture is obtained.

[0227] The decoded i-th frame display picture can refer to the i-th frame display picture data which can be sent to a display for display. The display can display the i-th frame display picture according to the decoded i-th frame display picture.

[0228] Step 5092, the receiving end device Sj determines whether the frame number frameID j is not in its own frame loss list dropList i . j If not, step 5093 is performed; if yes, step 5094 is performed.

[0229] In implementation, the frame loss list dropList j of each receiving end device Sj can be dynamically updated all the time. Therefore, at any moment before the sending and displaying, the frame loss list dropList j of each receiving end device Sj is dynamically updated. j j ​​A new frame number of a display frame that is not displayed can be added at any time. Therefore, after decoding is completed, step 5092 is also needed to determine whether the frame number of the display frame that is completed this time is not in the lost frame list.

[0230] Step 5093, the i-th display frame that is completed is not sent for display.

[0231] If the frame number is in the lost frame list, it indicates that at least one of all the devices that perform synchronized display of display frames has not successfully received or cannot display the frame display frame at the specified display time. In order to ensure the synchronized display of display frames, at this time, all the devices that perform synchronized display of display frames should not display the frame display frame. Therefore, step 5093 is executed in the case where step 5092 is determined to be "No".

[0232] Step 5094, the receiving end device S j Determine whether the sending time has arrived, if yes, execute step 5096; if no, execute step 5095.

[0233] In some embodiments, if a preset condition is met, the receiving end device S j Determine that the sending time has arrived. Exemplarily, the above-mentioned preset condition can be: systemTime Sj -offset Sj >=playTime i .

[0234] Wherein, the meaning of each symbol is as described above, which will not be repeated here.

[0235] In the preset condition, the left side is the system time of the receiving end device S j minus the clock offset between the system time of the receiving end device S j and the system time of the source end device S0, and the right side is the specified display time specified by the source end device S0 for the i-th display content.

[0236] Of course, the above-mentioned preset condition can also be written as: systemTime Sj >=playTime i +offset Sj .

[0237] The above determination method can be understood as: determining whether the sending time has arrived by determining whether the system time of the receiving end device S j at this time has reached the corrected specified display time.

[0238] Step 5095, the receiving end device S j waits for sending, and determines in real time whether the frame number is in the lost frame list.

[0239] In case of "No" in step 5049, the receiving end device S j has not reached the corrected designated display time, and thus the receiving end device S j needs to wait.

[0240] In some embodiments, it is possible that during the waiting time, the receiving end device S j receives a message from other receiving end devices S j that the frame number i needs to be added to the lost frame list, and thus in step 5095, the receiving end device S j still needs to determine in real time whether the frame number is in the lost frame list. Only when the frame number reaches the display time and still does not appear in the lost frame list, step 5096 is executed.

[0241] In step 5096, the receiving end device S j displays the display screen corresponding to the frame number.

[0242] In case of "Yes" in step 5049, or in case that step 5095 is executed and until the display time is reached and the frame number still does not appear in the lost frame list, step 5096 is executed. Thus, the method provided by the present application can ensure that all the devices that display the display screen display the display screen of the same frame number at the same time, and do not display the display screen of different frame numbers at the same time, so that the synchronization of the display screen can be ensured.

[0243] In some embodiments, the receiving end device S j initially determines that the i-th frame is lost, and thus informs other receiving end devices to add the frame number i to their lost frame lists; but later, the receiving end device S j receives the i-th frame again due to lost frame retransmission, and thus the display screen synchronization method provided by the embodiments of the present application can further include the following step: the receiving end device S j informs other receiving end devices to delete the frame number i from their lost frame lists. That is, the display screen synchronization method provided by the embodiments of the present application can add frame numbers to the lost frame list and delete frame numbers from the lost frame list, and the lost frame list is dynamically updated and maintains the frame numbers of the display screen frames that do not perform display. Thus, more frames can be displayed as much as possible while ensuring the synchronization of the display screen.

[0244] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0245] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / equipment and method can be implemented in other ways. For example, the apparatus / equipment embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. There can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication connection between the units can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0246] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place, or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0247] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0248] The integrated module / unit, if implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable storage medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content included in the computer readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable storage medium does not include electrical carrier signals and telecommunication signals.

[0249] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, any change or replacement within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A display screen synchronization method, applied to a receiving end device, characterized in that, The receiving device includes frame loss information, which is used to record the identifiers of frame images that are not displayed. The method includes: Establish a communication connection with the source device; The system receives a first frame sent by the source device and a first designated display time corresponding to the first frame. The first designated display time is determined by the source device based on the current latency. The current latency is determined based on the current transmission latency of the receiving device, the current decoding latency of the receiving device, the processing time of one frame of display, and the continuous display time of one frame of display. The current transmission latency of the receiving device is obtained by the receiving device through linear fitting based on the actual transmission latency of historical frame. The current decoding latency of the receiving device is obtained by the receiving device through linear fitting based on the actual decoding latency of historical frame. The historical frame includes at least a preset number of frame identifiers located before the identifier of the first frame. Obtain the estimated decoding delay required to decode the first frame, and determine the estimated display time corresponding to the first frame based on the estimated decoding delay; if the estimated display time is earlier than or equal to the first specified display time, it is determined that the first frame can be displayed according to the first specified display time. The estimated decoding delay is obtained by linear fitting based on the actual decoding delay of historical frames. If it is determined that the first frame can be displayed at the first specified display time, and the frame number of the first frame is not in the lost frame information, then the first frame is displayed at the first specified display time. If it is determined that the first frame cannot be displayed at the first specified display time, then the identifier of the first frame is added to the frame loss information, and the identifier of the first frame is broadcast to all other receiving devices that are synchronizing the display of frames with the receiving device.

2. The method according to claim 1, characterized in that, The method further includes: If the identifier of the first frame is in the frame loss information, then the first frame will not be displayed.

3. The method according to claim 1, characterized in that, The method further includes: The lost frame identifier is determined based on the identifier corresponding to the successfully received frame; wherein, the lost frame identifier includes the identifier corresponding to the frame that was not successfully received. The lost frame identifier is added to the lost frame information, and the lost frame identifier is notified to other devices.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: The current decoding delay of the receiving device is determined based on the actual decoding delay of at least one successfully received frame. The current decoding latency is sent to the source device.

5. The method according to any one of claims 1-3, characterized in that, The method further includes: The current transmission delay of the receiving device is determined based on the actual transmission delay of at least one successfully received frame. The current transmission delay is sent to the source device.

6. The method according to any one of claims 1-3, characterized in that, The method further includes: Receive lost frame identifiers and / or actively discarded frame identifiers sent by other devices, and add the lost frame identifiers and / or actively discarded frame identifiers sent by other devices to the lost frame information.

7. The method according to any one of claims 1-3, characterized in that, The receiving device and the source device are located on the same local area network.

8. The method according to any one of claims 1-3, characterized in that, The identifier of the frame includes the frame number of the frame.

9. A display screen synchronization method, applied to a source-end device, characterized in that, The source device is used to acquire a frame and send the frame to the receiving device, the method including: Establish a communication connection with the receiving device; Send a first screen frame and a first specified display time corresponding to the first screen frame to the receiving device; Wherein, the first specified display time is determined by the source device based on the current latency, the current latency is determined based on the current transmission latency of the receiving device, the current decoding latency of the receiving device, the processing time of one frame of display, and the continuous display time of one frame of display. The current transmission latency of the receiving device is obtained by the receiving device through linear fitting based on the actual transmission latency of historical frame, and the current decoding latency of the receiving device is obtained by the receiving device through linear fitting based on the actual decoding latency of historical frame. The historical frame includes at least a preset number of frame frames whose frame identifier is located before the identifier of the first frame. The source device is further configured to synchronize the display of the screen frames with the receiving device. The source device includes frame loss information, which is used to record the identifiers of screen frames that are not displayed. The method further includes: If the frame number of the first frame is not in the lost frame information, then the first frame is displayed according to the first specified display time; If the frame number of the first frame is in the lost frame information, then the first frame will not be displayed.

10. The method according to claim 9, characterized in that, The method further includes: Receive the lost frame identifier and / or actively discarded frame identifier sent by the receiving device, and add the lost frame identifier and / or actively discarded frame identifier sent by the receiving device to the lost frame information.

11. The method according to any one of claims 9-10, characterized in that, The method further includes: Receive at least one of the receiving devices' current transmission delay and / or at least one of the receiving devices' current decoding delay; The maximum value among the current transmission delays of at least one of the receiving devices is determined as the current transmission delay, and / or the maximum value among the current decoding delays of at least one of the receiving devices is determined as the current decoding delay.

12. The method according to any one of claims 9-10, characterized in that, The source device and the receiver device are located on the same local area network.

13. A display screen synchronization method, applied in a system including a source device and at least one receiving device, characterized in that, The receiving device is a device for synchronized display of screen frames, or the receiving device and the source device are both devices for synchronized display of screen frames. The method includes: The source device sends a first frame and a first designated display time corresponding to the first frame to each of the receiving devices. The first designated display time is determined by the source device based on the current latency. The current latency is determined based on the current transmission latency of the receiving device, the current decoding latency of the receiving device, the processing time of one frame of display, and the duration of continuous display of one frame of display. The current transmission latency of the receiving device is obtained by the receiving device through linear fitting based on the actual transmission latency of historical frame of display. The current decoding latency of the receiving device is obtained by the receiving device through linear fitting based on the actual decoding latency of historical frame of display. The historical frame of display includes at least a preset number of frame of display whose identifier is located before the identifier of the first frame of display. Each device in the device that performs synchronized display of screen frames includes frame loss information, which is used to record the identifiers of screen frames that are not displayed. If any of the receiving devices fails to receive the first frame, or successfully receives the first frame but determines that it cannot be displayed at the specified display time, then each device in the frame synchronization display devices will not display the first frame. Specifically, this includes: If the first receiving device fails to receive the first screen frame, or successfully receives the first screen frame but determines that it cannot be displayed at the first specified display time, then the first receiving device adds the identifier of the first screen frame to the frame loss information of the first receiving device and notifies the other devices in the device that performs screen frame synchronization display of the identifier of the first screen frame. Each of the devices performing synchronized display of the first frame does not display the first frame in the lost frame information according to the identifier of the first frame.

14. The method according to claim 13, characterized in that, The method further includes: The source device determines the first specified display time based on the current transmission delay and / or the current decoding delay; The current transmission delay is related to the actual transmission delay of the historical frame, and the current decoding delay is related to the actual decoding delay of the receiving device in decoding the historical frame. The historical frame includes at least one frame whose frame identifier is located before the identifier of the first frame.

15. The method according to claim 14, characterized in that, Before the source device determines the first specified display time based on the current transmission delay and / or the current decoding delay, the method further includes: The receiving device determines its current transmission delay based on the actual transmission delay of the historical frame received by the receiving device, and / or determines its current decoding delay based on the actual decoding delay of the historical frame received by the receiving device. The receiving device sends the determined current transmission delay and / or current decoding delay of the receiving device to the source device; The source device determines the first specified display time based on the current transmission delay and / or the current decoding delay, specifically including: The source device determines the maximum value among the current transmission delays received from at least one of the receiving devices as the current transmission delay, and / or determines the maximum value among the current decoding delays received from at least one of the receiving devices as the current decoding delay.

16. The method according to any one of claims 13-15, characterized in that, The source device and the receiver device are located on the same local area network.

17. A receiving device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor is configured to execute the computer program, it causes the receiving device to implement the method as described in any one of claims 1-8.

18. A source device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor is configured to execute the computer program, it causes the source device to implement the method as described in any one of claims 9-12.

19. A display screen synchronization system, the display screen synchronization system comprising a source device and at least one receiving device, wherein, The source device and the receiving device are respectively configured to perform the steps performed by the source device and the receiving device in the method as described in any one of claims 13-16.

20. A computer-readable storage medium configured to store a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8, or the method as described in any one of claims 9-12.

21. A computer program product, characterized in that, The computer program product is configured to run on a receiving device, causing the receiving device to perform the method as described in any one of claims 1-8, or the computer program product is configured to run on a source device, causing the source device to perform the method as described in any one of claims 9-12.

22. A chip system, characterized in that, The chip system includes a memory and a processor, the processor being configured to execute a computer program stored in the memory to implement the method as described in any one of claims 1-8, or the method as described in any one of claims 9-12.

Images