Screen projection method and display device

Abstract

The present disclosure relates to a screen projection method and a display device, comprising: in response to a received starting instruction, triggering a display to display a follow-up training interface; obtaining a training video stream corresponding to the follow-up training interface from a server, and receiving a rendered follow-up training video stream sent by a terminal device; when a first training image corresponding to a first playing time in the training video stream does not match a first follow-up training image action corresponding to the first playing time in the rendered follow-up training video stream, searching for a second training image matching the first follow-up training image action in the training video stream; when the second training image exists in the training video stream, sending a first instruction signal to the terminal device to instruct the terminal device to send an unrendered follow-up training video stream, so as to ensure that the training video stream played by the display device is consistent with the follow-up training video stream, and improve the user experience.

Description

Technical Field

[0001] This disclosure relates to the field of display technology, and more particularly to a projection method and display device. Background Technology

[0002] With the rise of national fitness activities, more and more users can choose different ways to participate in fitness activities. For example, users can participate in fitness check-ins by following training video streams played on a TV. Specifically, the user's mobile phone captures the training video stream and sends the data to the TV. The TV then displays both the training video stream and the video stream captured from the mobile phone.

[0003] In existing technologies, after a mobile device captures a user's practice video stream, it typically performs a beautification process before sending it to the TV. Because the mobile device needs to beautify the practice video stream, it inevitably leads to a lack of synchronization between the training video stream played on the TV and the practice video stream. This results in a significant delay between the practice video stream cast from the mobile device to the TV and the actual training video stream, thus reducing the user experience. Summary of the Invention

[0004] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a screen projection method and a display device, which improves the user experience.

[0005] In a first aspect, embodiments of this disclosure provide a screen mirroring method, including:

[0006] In response to the received start command, the display is triggered to show the training interface, wherein the training interface is provided with a display window and a local window, and the display window is used to display the training video stream;

[0007] The training video stream corresponding to the training interface is obtained from the server, and the rendered training video stream sent by the terminal device is received. The rendered training video stream is used to be displayed directly in the local window without being rendered by the display device.

[0008] When the first training image corresponding to the first playback time in the training video stream does not match the action of the first follow-up image corresponding to the first playback time in the rendered follow-up video stream, a second training image that matches the action of the first follow-up image is searched in the training video stream, wherein the first playback time is greater than the second playback time corresponding to the second training image.

[0009] When the second training image is present in the training video stream, a first indication signal is sent to the terminal device to instruct the terminal device to send an unrendered follow-up training video stream.

[0010] As one possible implementation, when the first training image in the training video stream corresponding to the first playback time does not match the action of the first follow-up image in the rendered follow-up video stream corresponding to the first playback time, searching for a second training image in the training video stream that matches the action of the first follow-up image includes:

[0011] Obtain the first training image of the training video stream at the first playback moment, and the first follow-up image of the rendered follow-up video stream corresponding to the first playback moment;

[0012] When the degree of motion matching between the first follow-up image and the first training image is lower than a preset threshold, a second training image that matches the degree of motion matching between the first follow-up image and the first training image in the training video stream is found to meet the preset threshold.

[0013] When the second training image exists in the training video stream, sending a first indication signal to the terminal device includes:

[0014] When the difference between the first playback time and the second playback time is greater than or equal to a preset difference, a first indication signal is sent to the terminal device to instruct the terminal device to send an unrendered follow-up video stream.

[0015] As one possible implementation, after sending the first indication signal to the terminal device, the method further includes:

[0016] Receive the unrendered follow-up video stream sent by the terminal device;

[0017] In the unrendered follow-up video stream, a second follow-up image is determined that matches the action of the first training image in the training video stream;

[0018] After rendering the unrendered follow-up video stream, the rendered follow-up video stream is displayed in a local window on the monitor. The follow-up video stream displayed in the local window is played from the playback time corresponding to the second follow-up image.

[0019] As one possible implementation, the method may further include:

[0020] If the difference between the first playback time and the second playback time is less than a preset difference, the first indication signal is not sent to the terminal device.

[0021] As one possible implementation, the first indication signal may also be used to instruct the terminal device to pause the rendering operation.

[0022] As one possible implementation, the step of searching for a second training image in the training video stream that matches the action of the first follow-up image includes:

[0023] In the training video stream, starting from the first playback moment, candidate training images of the preset moments before the first playback moment of the training video stream are obtained, and the candidate training images and the first follow-up image are sent to the server.

[0024] Obtain the comparison results between the candidate training image and the first follow-up image obtained by the server;

[0025] The candidate training image whose matching degree with the first training image meets a preset threshold is determined as the second training image.

[0026] As one possible implementation, the step of obtaining the comparison result between the candidate training image and the first follow-up image by the server includes:

[0027] Obtain the feature point comparison results of the first image feature points of the candidate training image and the second image feature points of the first follow-up training image from the server.

[0028] In a second aspect, embodiments of this disclosure provide a display device, including:

[0029] monitor;

[0030] The controller is configured as follows:

[0031] In response to the received start command, the display is triggered to show the training interface, wherein the training interface is provided with a display window and a local window, and the display window is used to display the training video stream;

[0032] The training video stream corresponding to the training interface is obtained from the server, and the rendered training video stream sent by the terminal device is received. The rendered training video stream is used to be displayed directly in the local window without being rendered by the display device.

[0033] When the first training image corresponding to the first playback time in the training video stream does not match the action of the first follow-up image corresponding to the first playback time in the rendered follow-up video stream, a second training image that matches the action of the first follow-up image is searched in the training video stream, wherein the first playback time is greater than the second playback time corresponding to the second training image.

[0034] When the second training image is present in the training video stream, a first indication signal is sent to the terminal device to instruct the terminal device to send an unrendered follow-up training video stream.

[0035] As one possible implementation method, optional,

[0036] The controller is also configured to:

[0037] Receive the unrendered follow-up video stream sent by the terminal device;

[0038] In the unrendered follow-up video stream, a second follow-up image is determined that matches the action of the first training image in the training video stream;

[0039] After rendering the unrendered follow-up video stream, the rendered follow-up video stream is displayed in a local window on the monitor. The follow-up video stream displayed in the local window is played from the playback time corresponding to the second follow-up image.

[0040] As one possible implementation method, optional,

[0041] The controller is also configured to:

[0042] If the difference between the first playback time and the second playback time is less than a preset difference, the first indication signal is not sent to the terminal device.

[0043] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0044] The screen projection method and display device provided in this disclosure first respond to a received start command, triggering the display to show a follow-up interface. Then, it obtains the training video stream corresponding to the follow-up interface from the server and receives the rendered follow-up video stream sent by the terminal device. If the action of the first training image corresponding to the first playback time in the training video stream does not match the action of the first follow-up image corresponding to the first playback time in the rendered follow-up video stream, it searches for a second training image in the training video stream that matches the action of the first follow-up image. Finally, when the second training image exists in the training video stream, it sends a first indication signal to the terminal device to instruct the terminal device to send the unrendered follow-up video stream. That is, when the display device receives the terminal device's signal, it sends the first indication signal to the terminal device. After the terminal device sends the rendered follow-up video stream, if it is determined by comparison that the first training image at the first playback moment in the training video stream played by the display device does not match the action of the first follow-up image at the first playback moment in the rendered follow-up video stream, a first indication signal is sent to the terminal device to instruct the terminal device to send an unrendered follow-up video stream. That is, the display device directly receives the unrendered follow-up video stream from the terminal device. Therefore, the display delay between the training video stream and the follow-up video stream displayed by the display device can be reduced due to the time spent by the terminal device in rendering the follow-up video stream, ensuring that the follow-up video stream and the training video stream displayed by the display device are synchronized and improving the user experience. Attached Figure Description

[0045] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0046] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1A This is a network architecture diagram of a screen mirroring system according to an embodiment of the screen mirroring method provided in this disclosure;

[0048] Figure 1B This is a schematic diagram of the structure of a terminal device provided in an embodiment of this disclosure;

[0049] Figure 2A This is a flowchart illustrating a screen mirroring method provided in an embodiment of this disclosure;

[0050] Figure 2B This is a schematic diagram of the signal flow of a screen projection method provided in an embodiment of the present disclosure;

[0051] Figure 2C This disclosure provides an interactive schematic diagram.

[0052] Figure 3A This is a flowchart illustrating another screen mirroring method provided in this embodiment of the disclosure;

[0053] Figure 3B This is a schematic diagram of the signal flow of another screen projection method provided in this embodiment of the disclosure;

[0054] Figure 3C This is a flowchart illustrating yet another screen projection method provided in this embodiment of the disclosure;

[0055] Figure 3D This is a schematic diagram of the signal flow of another screen projection method provided in this embodiment;

[0056] Figure 4A This is a flowchart illustrating yet another screen projection method provided in this embodiment of the disclosure;

[0057] Figure 4B This is a schematic diagram of the signal flow of another screen projection method provided in this embodiment;

[0058] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0059] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0060] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0061] Figure 1A This diagram schematically illustrates a screen projection system network architecture diagram according to an embodiment of this application. The screen projection system network architecture diagram includes a terminal device 10, a display device 20, a router 30, and a server 40, etc. Specifically: The terminal device can act as the sending end in the screen projection system network architecture, used to send multimedia content and various instructions to the display device. In exemplary embodiments, the terminal device includes, but is not limited to, smartphones, tablets, game consoles, laptops, etc. The display device can act as the display end in the screen projection system network architecture, used to receive multimedia content and various instructions from the terminal device, and synchronously play the received multimedia content on its own display panel or the display panel of an associated device. The display device can be a relatively large-screen device such as a virtual reality device, set-top box, smart TV, vehicle terminal, or large outdoor display panel; of course, it can also be a relatively small-screen device such as a smartphone, tablet, or laptop.

[0062] A router can act as a gateway in the network architecture of a screen mirroring system, used to build a local area network (LAN) and provide internet services. A router provides local connectivity between terminal devices and display devices.

[0063] A server can serve as a cloud device in the network architecture of a screen mirroring system, providing cloud services such as cloud account management. Servers can be rack servers, blade servers, tower servers, or cabinet servers (including standalone servers or server clusters composed of multiple servers).

[0064] The terminal device can discover the display device through a first network or a second network, and establish a network connection with the display device based on the first network or the second network. Through this established network connection, the terminal device can project multimedia content to be displayed on the display device, which will then play it out. The first network can be a local area network (LAN), which can be a wired LAN or a wireless LAN. Figure 1AAs shown, a wireless local area network (WLAN) can be a Wi-Fi network built by router 8. The second network can be a wide area network (WAN), such as the Internet. A WAN can include physical links, such as coaxial cable links, twisted-pair cable links, fiber optic links, and combinations thereof. A WAN can also include wireless links, such as cellular links and satellite links.

[0065] In an exemplary embodiment, such as Figure 1B As shown, the terminal device and display device include at least one of the following: tuner 210, communicator 220, detector 230, external device interface 240, controller 250, display 260, audio output interface 270, memory, power supply, and user interface.

[0066] In some embodiments, the controller includes a processor, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first interface to an nth interface for input / output.

[0067] The display 260 includes a display screen assembly for presenting images, a driving assembly for driving image display, a component for receiving image signals from the controller output, and a user control UI interface for displaying video content, image content, menu control interface, and user control UI interface.

[0068] The display 260 can be an LCD display, an OLED display, or a projection display, and can also be a projection device and a projection screen.

[0069] The communicator 220 is a component used to communicate with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of the following: a Wi-Fi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver. The display device 200 can establish the transmission and reception of control signals and data signals with the external control device 100 or the server 400 through the communicator 220.

[0070] The user interface can be used to receive control signals from the control device 100 (such as an infrared remote control).

[0071] Detector 230 is used to collect signals from the external environment or to interact with the external environment. For example, detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to collect external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0072] The external device interface 240 may include, but is not limited to, one or more of the following: High Definition Multimedia Interface (HDMI), analog or high-definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), RGB port, etc. It may also be a composite input / output interface formed by multiple interfaces mentioned above.

[0073] The tuner / demodulator 210 receives broadcast television signals via wired or wireless means, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals.

[0074] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0075] The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command to select a UI object to display on the monitor 260, the controller 250 can execute operations related to the object selected by the user command.

[0076] In some embodiments, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0077] Users can input commands through a graphical user interface (GUI) displayed on the monitor 260, and the user input interface receives the user input commands through the GUI. Alternatively, users can input commands by entering specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.

[0078] A "user interface" is the medium through which an application or operating system interacts and exchanges information with the user. It converts information from its internal form to a form that the user can accept. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0079] In addition, the terminal device and display device also include a rendering module, which is used to provide audio or video effects rendering services. Audio effects may include audio reverb, speed change, pitch change, limiting, echo, compression, gain control, equalization control, etc., while video effects may include beautification, makeup, face recognition, 2D stickers, 3D stickers, etc.

[0080] The above provides a brief introduction to the various functions of terminal devices and display devices.

[0081] Terminal devices can perform screen mirroring operations to display devices.

[0082] However, in the existing technology, when the user is following the training video stream played on the display device, the terminal device collects the user's training video stream and projects it onto the display device. The terminal device then renders the collected user's training video stream with special effects and sends it to the display device. However, the process of the terminal device rendering the user's training video stream with special effects will inevitably cause the training video stream and the training video stream played on the display device to be unable to play synchronously, which reduces the user's experience.

[0083] This disclosure provides a screen mirroring method. When a display device receives a rendered follow-up video stream sent by a terminal device, and it is determined by comparison that the first training image at the first playback time in the training video stream played by the display device does not match the action of the first follow-up image at the first playback time in the rendered follow-up video stream, a first indication signal is sent to the terminal device to instruct the terminal device to send an unrendered follow-up video stream. That is, the display device directly receives the unrendered follow-up video stream sent by the terminal device. Therefore, the display delay between the training video stream and the follow-up video stream displayed by the display device can be reduced due to the time spent by the terminal device in rendering the follow-up video stream, ensuring that the follow-up video stream and the training video stream displayed by the display device are synchronized, and improving the user experience.

[0084] Figure 2A This is a flowchart illustrating a screen mirroring method provided in an embodiment of this disclosure. Figure 2BThis is a schematic diagram of the signal flow of a screen projection method provided in an embodiment of the present disclosure. Figure 2C This disclosure provides an interactive schematic diagram, combined with... Figure 2A , Figure 2B and Figure 2C Screen mirroring methods include:

[0085] S100: In response to the received start command, trigger the display to show the follow-up interface.

[0086] The training interface includes a display window and a local window. The display window is used to show the training video stream.

[0087] The startup command can be a voice-based startup command, a startup command sent by a control device, or a startup command sent by a smart device.

[0088] In some embodiments, the control device may be a remote control. Communication between the remote control and the display device includes infrared protocol communication, Bluetooth protocol communication, and other short-range communication methods, controlling the display device wirelessly or via wired means. Users can control the display device by inputting user commands through buttons on the remote control, voice input, control panel input, etc.

[0089] In some embodiments, smart devices (such as mobile terminals, tablets, computers, laptops, etc.) can also be used to control the display device. For example, an application running on a smart device can be used to control the display device.

[0090] In some embodiments, the display device may receive instructions not through the aforementioned smart devices or control devices, but through touch or gestures.

[0091] In some embodiments, the display device can also be controlled in ways other than control devices and smart devices. For example, it can be controlled by receiving user voice commands directly through a module configured inside the display device that acquires voice commands, or it can be controlled by receiving user voice commands through a voice control device set outside the display device.

[0092] S200: Obtain the training video stream corresponding to the follow-up interface from the server, and receive the rendered follow-up video stream sent by the terminal device.

[0093] Among them, the rendered follow-up video stream is used to be displayed directly in the local window without being rendered by the display device.

[0094] The training video stream displayed on the monitor's interface is retrieved from the server on the display device.

[0095] In a specific implementation, the terminal device includes an image acquisition module. For example, the image acquisition module can be a camera, and the terminal device's image acquisition module acquires user video data.

[0096] In one specific implementation, the display device plays a training video stream, such as a fitness training video stream or a dance training video stream. The user follows along with the training video stream played by the display device, and the image acquisition module of the terminal device captures the user's follow-along video stream.

[0097] Specifically, the distance between the terminal device and the display device is set to be less than a preset distance threshold, and the angle between the orientation of the image acquisition module of the terminal device and the orientation of the display screen of the display device is less than a preset angle threshold.

[0098] In this embodiment of the disclosure, when a user performs continuous limb movements, such as dance or gymnastic movements, by following a training video stream played on a display device, the image acquisition module of the terminal device captures the user's training video stream. Understandably, to keep the image acquisition module fixed and prevent shaking, the terminal device can be mounted on a fixed bracket, meaning the distance between it and the display device is less than a preset distance threshold, and the angle between the orientation of the terminal device's image acquisition module and the orientation of the display screen is less than a preset angle threshold. This ensures that while the user follows the training video stream played on the display device, the terminal device's image acquisition module can also capture the user's training video stream.

[0099] To ensure a good user experience, in existing technologies, the image acquisition module of the terminal device sends the acquired video stream to the rendering module after acquiring it. The rendering module then renders the video stream before sending it to the display device, ensuring that the video stream played on the display device is the rendered video stream.

[0100] S300: When the action of the first training image corresponding to the first playback moment in the training video stream does not match the action of the first follow-up image corresponding to the first playback moment in the rendered follow-up video stream, search for a second training image in the training video stream that matches the action of the first follow-up image.

[0101] The first playback time is greater than the second playback time corresponding to the second training image.

[0102] Because the terminal device needs to render the acquired training video stream, the rendering operation takes a certain amount of time. Therefore, there may be a problem that the rendered training video stream received by the display device is not synchronized with the training video stream. For example, at a certain playback moment, the training video stream plays up to the tenth frame, while the rendered training video stream only plays up to the eighth frame. This causes the rendered training video stream and the training video stream to be viewed by the user on the display device as different frames. The user cannot confirm whether the actions are consistent with the training video stream, which reduces the user experience.

[0103] Based on the problems existing in the prior art, in this embodiment of the disclosure, after receiving the rendered follow-up video stream sent by the terminal device, the first training image corresponding to the first playback time in the training video stream played by the display device and the first follow-up image corresponding to the first playback time in the rendered follow-up video stream are first obtained. Then, the actions of the first training image and the first follow-up image are compared to see if they match. When the actions of the first training image and the first follow-up image do not match, it indicates that the training video stream played by the display device and the follow-up video stream are out of sync. The asynchrony between the training video stream and the follow-up video stream may be caused by the terminal device needing to perform rendering operations on the training video stream. At this time, a second training image matching the first follow-up image is searched in the training video stream, that is, a second training image matching the first follow-up image before the first playback time and the second playback time corresponding to the second training image are searched in the training video stream.

[0104] In a specific implementation, finding a second training image in the training video stream that matches the action of the first follow-up image includes: in the training video stream, starting from the first playback time, obtaining candidate training images at preset times before the first playback time of the training video stream, and sending the candidate training images and the first follow-up image to the server; obtaining the server's comparison result of the candidate training images and the first follow-up image; and determining the candidate training image whose matching degree with the first follow-up image meets a preset threshold as the second training image.

[0105] Specifically, starting from the first playback moment of the first training image in the training video stream played on the display device, candidate training images at preset moments preceding the first playback moment are obtained. The matching degree between the candidate training images at the previous playback moment and the first follow-up image is compared. When the matching degree between the candidate training images at the previous playback moment and the first follow-up image meets a preset threshold, the second training image is determined as the candidate training image corresponding to the previous playback moment. When the matching degree between the candidate training images at the previous playback moment and the first follow-up image does not meet the preset threshold, the first training image of the training video stream is obtained. The candidate training images corresponding to the two moments before the first playback moment are compared with the first follow-up image. When the matching degree between the candidate training images corresponding to the two moments before the first playback moment and the first follow-up image meets the preset threshold, the second training image is determined as the candidate training image corresponding to the two moments before the first playback moment. This process is repeated to compare each candidate training image until the second training image is determined. If no candidate image matching degree with the first follow-up image meets the preset threshold is obtained after the comparison of each candidate training image, it can be determined that there is no second training image in the training video stream.

[0106] In a specific implementation, the process of obtaining the comparison results between the candidate training image and the first follow-up training image by the server includes:

[0107] After receiving the candidate training image and the first follow-up image sent by the controller of the display device, the server first extracts the first image feature points of the candidate training image and the second image feature points of the first follow-up image. Then, based on the image processing algorithm, it compares the extracted first image feature points of the candidate training image and the second image feature points of the first follow-up image to determine the feature point comparison result.

[0108] S400: When a second training image exists in the training video stream, a first indication signal is sent to the terminal device to instruct the terminal device to send the unrendered follow-up training video stream.

[0109] When the first training image in the training video stream corresponding to the first playback moment does not match the action of the first follow-up image in the rendered follow-up video stream corresponding to the first playback moment, and a second training image exists in the training video stream, this indicates that the training video stream played by the display device and the follow-up video stream are out of sync. By sending a first indication signal to the terminal device, the terminal device is instructed to send the unrendered follow-up video stream. Since the follow-up video stream received by the display device does not need to go through the rendering operation of the terminal device, that is, the display device directly receives the unrendered follow-up video stream of the terminal device, thus reducing the display delay between the training video stream and the follow-up video stream displayed by the display device caused by the time spent by the terminal device to perform the rendering operation of the follow-up video stream.

[0110] The screen projection method provided in this embodiment first responds to a received start command by triggering the display to show a follow-up interface. Then, it obtains the training video stream corresponding to the follow-up interface from the server and receives the rendered follow-up video stream sent by the terminal device. If the action of the first training image corresponding to the first playback time in the training video stream does not match the action of the first follow-up image corresponding to the first playback time in the rendered follow-up video stream, it searches for a second training image in the training video stream that matches the action of the first follow-up image. Finally, if the second training image exists in the training video stream, it sends a first indication signal to the terminal device to instruct the terminal device to send the unrendered follow-up video stream. That is, when the display device receives the signal from the terminal device... After sending the rendered follow-up video stream, if it is determined by comparison that the first training image at the first playback moment in the training video stream played by the display device does not match the action of the first follow-up image at the first playback moment in the rendered follow-up video stream, a first indication signal is sent to the terminal device to instruct the terminal device to send an unrendered follow-up video stream. That is, the display device directly receives the unrendered follow-up video stream from the terminal device. Therefore, the display delay between the training video stream and the follow-up video stream displayed by the display device can be reduced due to the time spent by the terminal device in rendering the follow-up video stream, ensuring that the follow-up video stream and the training video stream displayed by the display device are synchronized and improving the user experience.

[0111] Figure 3A This is a flowchart illustrating another screen mirroring method provided in this embodiment of the disclosure. Figure 3B This is a schematic diagram of the signal flow of another screen projection method provided in this disclosure embodiment. This disclosure embodiment is based on the above embodiment and combines... Figure 3A and Figure 3B One specific implementation of step S300 includes:

[0112] S301. Obtain the first training image of the training video stream at the first playback moment, and the first follow-up image of the rendered follow-up video stream corresponding to the first playback moment.

[0113] Since the training video stream received by the display device is a rendered training video stream, the first training image and the first training image played by the terminal device may be different images. In this case, the first training image of the training video stream at the first playback time and the first training image of the rendered training video stream at the corresponding first playback time are first obtained.

[0114] S302. When the degree of motion matching between the first follow-up image and the first training image is lower than a preset threshold, find a second training image from the training video stream whose degree of motion matching with the first follow-up image meets the preset threshold.

[0115] After obtaining the first training image from the training video stream at the first playback moment and the first follow-up image from the rendered follow-up video stream corresponding to the first playback moment, the first step is to determine the degree of motion matching between the first follow-up image and the first training image. When the degree of motion matching between the first training image and the first follow-up image is lower than a preset threshold, it indicates that the training video stream played by the display device and the follow-up video stream are out of sync. The reason for the asynchrony between the training video stream and the follow-up video stream is that the terminal device performs rendering operations on the follow-up video stream before sending it to the display device for display. Therefore, the playback progress of the follow-up video stream is slower than that of the training video stream. The follow-up image played by the follow-up video stream has already been played in the training video stream. Therefore, a second training image that matches the motion of the first follow-up image can be found in the training video stream.

[0116] When the screen projection method includes steps S301 and S302, one specific implementation of step S400 includes:

[0117] S401. When the difference between the first playback time and the second playback time is greater than or equal to a preset difference, a first indication signal is sent to the terminal device to instruct the terminal device to send the unrendered follow-up video stream.

[0118] In a specific implementation, after finding a second training image from the training video stream whose action matching degree with the first follow-up image meets a preset threshold, the second playback time corresponding to the second training image can be determined. Based on the first playback time of the first training image and the second playback time of the second training image, the playback time of the training video stream played by the display device and the playback time of the rendered follow-up video stream played by the display device can be determined. Based on the relationship between the difference between the first playback time and the second playback time and a preset difference, the delay time between the rendered follow-up video stream played by the display device and the training video stream can be determined. When the difference between the first playback time and the second playback time is greater than or equal to the preset difference, it indicates that the rendered follow-up video stream played by the display device is... The delay between the training video stream and the practice video stream, which causes a delay between the rendered training video stream played on the display device, may be due to the terminal device needing to render the training video stream acquired by the image acquisition module. This causes a delay in sending the training video stream data from the terminal device to the display device. To ensure synchronization between the training video stream and the practice video stream played on the display device, the display device's controller sends a first indication signal to the terminal device, instructing the terminal device to send the unrendered training video stream. The display device's controller directly receives the training video stream acquired by the terminal device's image acquisition module without performing any rendering operations, thus avoiding the problem of desynchronization between the training video stream and the practice video stream played on the display device due to rendering operations.

[0119] When the screen projection method includes steps S301 and S302, another specific implementation of step S400 includes:

[0120] S402. When the difference between the first playback time and the second playback time is less than a preset difference, the first indication signal is not sent to the terminal device.

[0121] Specifically, as shown in the figure Figure 3C and Figure 3D As shown, when the difference between the first playback time and the second playback time is less than a preset difference, it indicates that the delay between the rendered follow-up video stream played by the display device and the training video stream is within the user's acceptable range. The controller of the display device does not need to send the first instruction signal to the terminal device. At this time, the display device normally receives the rendered follow-up video stream sent by the terminal device, ensuring that the follow-up video streams displayed by both the terminal device and the display device are rendered follow-up video streams, thus improving the user experience.

[0122] Figure 4A This is a flowchart illustrating another screen mirroring method provided in this embodiment of the disclosure. Figure 4B This is a schematic diagram of the signal flow of another screen projection method provided in this disclosure embodiment. This disclosure embodiment is based on the above embodiments and combines... Figure 4A and Figure 4B Screen mirroring methods also include:

[0123] S500 receives unrendered follow-up video streams sent by terminal devices.

[0124] When the first training image in the training video stream corresponding to the first playback moment does not match the action of the first follow-up image in the rendered follow-up video stream corresponding to the first playback moment, this indicates a delay between the rendered follow-up video stream played by the display device and the training video stream. The reason for this delay may be that the terminal device needs to render the follow-up video stream acquired by the image acquisition module, causing a delay in the terminal device sending the follow-up video stream data to the display device. To ensure synchronization between the follow-up video stream played by the display device and the training video stream, the controller of the display device sends a first indication signal to the terminal device, instructing the terminal device to send an unrendered follow-up video stream. The controller of the display device receives the unrendered follow-up video stream sent by the terminal device.

[0125] S600, Identify and train a second follow-up image that matches the action of the first training image in the unrendered follow-up video stream.

[0126] In the initial state, the controller receives a rendered follow-up video stream from the terminal device. When the actions of the first training image in the training video stream and the first follow-up image in the follow-up video stream displayed on the monitor do not match, to ensure that the training video stream and the follow-up video stream displayed on the monitor play synchronously, the controller first sends a first indication signal to the terminal device, instructing the terminal device to send an unrendered follow-up video stream. However, since the playback progress of the rendered follow-up video stream played on the monitor is slower than that of the training video stream, if the controller directly performs a rendering operation on the unrendered follow-up video stream from the second playback time and plays the rendered follow-up video stream from the second playback time, the follow-up video stream played on the monitor and the training video stream will still be out of sync. In this embodiment of the present disclosure, after receiving the follow-up video stream sent by the terminal device, the controller first determines a second follow-up image in the unrendered follow-up video stream that matches the actions of the first training image in the training video stream.

[0127] S700: After rendering the unrendered follow-up video stream, the rendered follow-up video stream is displayed in a local window on the monitor. The follow-up video stream displayed in the local window starts playing from the playback time corresponding to the second follow-up image.

[0128] After determining the second follow-up image that matches the action of the first training image in the training video stream from the unrendered follow-up video stream, the unrendered follow-up video stream is rendered and then displayed in the local window of the monitor. The follow-up video stream displayed in the local window is played from the playback time corresponding to the second follow-up image.

[0129] However, in order to ensure the user experience, the controller of the display device renders the unrendered follow-up video stream before playing it on the display device, to avoid directly playing the unrendered follow-up video stream on the display device, which would reduce the user experience. That is, the follow-up video stream played on the display device is the rendered follow-up video stream, while the unrendered follow-up video stream is played on the terminal device.

[0130] As one possible implementation, when a second training image exists in the training video stream, a first indication signal is sent to the terminal device to instruct the terminal device to send the unrendered follow-up training video stream. The first indication signal is also used to instruct the terminal device to pause the rendering operation.

[0131] To ensure synchronization between the training video stream and the follow-up video stream played on the display device, the controller of the display device directly receives the training video stream from the image acquisition module of the terminal device. This avoids the problem of the training video stream and the follow-up video stream being out of sync due to rendering operations. At this time, the first indication signal that the controller of the display device can send can also be used to instruct the terminal device to pause the rendering operation and reduce the power consumption of the terminal device.

[0132] This disclosure also provides a display device, which includes:

[0133] monitor;

[0134] The controller is configured as follows:

[0135] In response to the received start command, the monitor is triggered to display the training interface, which has a display window and a local window. The display window is used to display the training video stream.

[0136] The system retrieves the training video stream corresponding to the training interface from the server and receives the rendered training video stream sent by the terminal device. The rendered training video stream is used to display the video stream directly in the local window without being rendered by the display device.

[0137] When the action of the first training image corresponding to the first playback time in the training video stream does not match the action of the first follow-up image corresponding to the first playback time in the rendered follow-up video stream, a second training image that matches the action of the first follow-up image is found in the training video stream, wherein the first playback time is greater than the second playback time corresponding to the second training image.

[0138] When a second training image exists in the training video stream, a first indication signal is sent to the terminal device to instruct the terminal device to send the unrendered follow-up training video stream.

[0139] As one possible implementation, the controller is optionally configured to:

[0140] Receive unrendered follow-up video streams sent by terminal devices;

[0141] In the unrendered follow-up video stream, identify and train a second follow-up image that matches the action of the first training image in the video stream;

[0142] After rendering the unrendered follow-up video stream, the rendered follow-up video stream is displayed in a local window on the monitor. The follow-up video stream displayed in the local window starts playing from the playback time corresponding to the second follow-up image.

[0143] As one possible implementation, the controller is optionally configured to:

[0144] If the difference between the first playback time and the second playback time is less than a preset difference, the first indication signal is not sent to the terminal device.

[0145] This disclosure also provides an electronic device, including: a processor, the processor being configured to execute a computer program stored in a memory, the computer program being executed by the processor to implement the steps of the above method embodiments.

[0146] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this disclosure. Figure 5 A block diagram is shown that is suitable for implementing embodiments of the present invention. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0147] like Figure 5 As shown, the electronic device 800 is presented in the form of a general-purpose computing device. The components of the electronic device 800 may include, but are not limited to: one or more processors 810, system memory 820, and bus 830 connecting different system components (including system memory 820 and processors).

[0148] Bus 830 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0149] Electronic device 800 typically includes a variety of computer system readable media. These media can be any media that can be accessed by electronic device 800, including volatile and non-volatile media, removable and non-removable media.

[0150] System memory 820 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 840 and / or cache memory 850. Electronic device 800 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 860 may be used to read and write non-removable, non-volatile magnetic media (commonly referred to as a "hard disk drive"). Disk drives for reading and writing to removable non-volatile disks (e.g., "floppy disks") and optical disk drives for reading and writing to removable non-volatile optical disks (e.g., CD-ROMs, DVD-ROMs, or other optical media) may be provided. In these cases, each drive may be connected to bus 830 via one or more data media interfaces. System memory 820 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0151] A program / utility 880 having a set (at least one) of program modules 870 may be stored, for example, in system memory 820. Such program modules 870 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 870 typically perform the functions and / or methods described in the embodiments of this invention.

[0152] The processor 810 executes various functional applications and information processing by running at least one of a plurality of programs stored in the system memory 820, such as implementing the method embodiments provided in the embodiments of the present invention.

[0153] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method embodiments.

[0154] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0155] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0156] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0157] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or wide area network (WAN) domain—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0158] This disclosure also provides a computer program product that, when run on a computer, causes the computer to perform the steps of the above-described method embodiments.

[0159] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0160] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A screen projection method, characterized by, include: In response to the received start command, the display is triggered to show the training interface, wherein the training interface is provided with a display window and a local window, and the display window is used to display the training video stream; The training video stream corresponding to the training interface is obtained from the server, and the rendered training video stream sent by the terminal device is received. The rendered training video stream is used to be displayed directly in the local window without being rendered by the display device. When the first training image corresponding to the first playback time in the training video stream does not match the action of the first follow-up image corresponding to the first playback time in the rendered follow-up video stream, a second training image that matches the action of the first follow-up image is searched in the training video stream. The first playback time is greater than the second playback time corresponding to the second training image. When the first training image does not match the action of the first follow-up image, it indicates that the training video stream and the follow-up video stream are out of sync. When the second training image exists in the training video stream, a first indication signal is sent to the terminal device to instruct the terminal device to send an unrendered follow-up training video stream; After sending the first indication signal to the terminal device, the method further includes: Receive the unrendered follow-up video stream sent by the terminal device; In the unrendered follow-up video stream, a second follow-up image is determined that matches the action of the first training image in the training video stream; After rendering the unrendered follow-up video stream, the rendered follow-up video stream is displayed in a local window on the monitor. The follow-up video stream displayed in the local window is played from the playback time corresponding to the second follow-up image.

2. The method of claim 1, wherein, When the action of the first training image corresponding to the first playback time in the training video stream does not match the action of the first follow-up image corresponding to the first playback time in the rendered follow-up video stream, searching for a second training image in the training video stream that matches the action of the first follow-up image includes: Obtain the first training image of the training video stream at the first playback moment, and the first follow-up image of the rendered follow-up video stream corresponding to the first playback moment; When the degree of motion matching between the first follow-up image and the first training image is lower than a preset threshold, a second training image that matches the degree of motion matching between the first follow-up image and the first training image in the training video stream is found to meet the preset threshold. When the second training image exists in the training video stream, sending a first indication signal to the terminal device includes: When the difference between the first playback time and the second playback time is greater than or equal to a preset difference, a first indication signal is sent to the terminal device to instruct the terminal device to send an unrendered follow-up video stream.

3. The method of claim 2, wherein, The method further includes: If the difference between the first playback time and the second playback time is less than a preset difference, the first indication signal is not sent to the terminal device.

4. The method of claim 1, wherein, The first indication signal is also used to instruct the terminal device to pause the rendering operation.

5. The method according to claim 1, characterized in that, The step of searching for a second training image in the training video stream that matches the action of the first follow-up image includes: In the training video stream, starting from the first playback moment, candidate training images of the preset moments before the first playback moment of the training video stream are obtained, and the candidate training images and the first follow-up image are sent to the server. Obtain the comparison results between the candidate training image and the first follow-up image obtained by the server; The candidate training image whose matching degree with the first training image meets a preset threshold is determined as the second training image.

6. The method of claim 5, wherein, The step of obtaining the comparison result between the candidate training image and the first follow-up image by the server includes: Obtain the feature point comparison results of the first image feature points of the candidate training image and the second image feature points of the first follow-up training image from the server.

7. A display device, characterized in that, The display device includes: monitor; The controller is configured as follows: In response to the received start command, the display is triggered to show the training interface, wherein the training interface is provided with a display window and a local window, and the display window is used to display the training video stream; The training video stream corresponding to the training interface is obtained from the server, and the rendered training video stream sent by the terminal device is received. The rendered training video stream is used to be displayed directly in the local window without being rendered by the display device. When the first training image corresponding to the first playback time in the training video stream does not match the action of the first follow-up image corresponding to the first playback time in the rendered follow-up video stream, a second training image that matches the action of the first follow-up image is searched in the training video stream. The first playback time is greater than the second playback time corresponding to the second training image. When the first training image does not match the action of the first follow-up image, it indicates that the training video stream and the follow-up video stream are out of sync. When the second training image exists in the training video stream, a first indication signal is sent to the terminal device to instruct the terminal device to send an unrendered follow-up training video stream; The controller is also configured to: Receive the unrendered follow-up video stream sent by the terminal device; In the unrendered follow-up video stream, a second follow-up image is determined that matches the action of the first training image in the training video stream; After rendering the unrendered follow-up video stream, the rendered follow-up video stream is displayed in a local window on the monitor. The follow-up video stream displayed in the local window is played from the playback time corresponding to the second follow-up image.

8. The device according to claim 7, characterized in that, The controller is also configured to: If the difference between the first playback time and the second playback time is less than a preset difference, the first indication signal is not sent to the terminal device.

Images