Display device, terminal device and screen projection method

Abstract

The present application provides a display device, a terminal device, and a screen projection method. After receiving the screen projection instruction from the mobile control terminal, the display device of the present application displays all the terminal devices that sent the screen projection instruction on the display interface, determines the range of the projected screen area and the terminal device to receive the screen projection on the display interface of the display device, performs a screen recording operation on the content displayed within the range on the display interface, and sends the screen recording data stream obtained by the screen recording operation to the terminal device to receive the screen projection, so that the terminal device to receive the screen projection displays the content displayed within the range on the display interface. In the present application, the display device can project the screen projection area to the corresponding terminal device according to the selected screen projection area, so that the content on the display interface of the display device can be projected to multiple terminal devices at the same time, thereby improving the efficiency of screen projection and improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of intelligent interaction technology, and in particular to a display device, a terminal device and a screen projection method. Background Art

[0002] With the development of technology, screen projection technology for projecting content displayed on a terminal device onto a display device for display has been widely used.

[0003] In order to further meet the needs of users, in the prior art, the content displayed on the display interface of the display device can also be projected to the terminal device, and the content displayed on the display interface of the display device can be controlled by manipulating the terminal device.

[0004] However, in the prior art, the content on the display interface of the display device can only be projected to one terminal device at a time. If multiple terminal devices have the need to project the content on the display device, the prior art can only end the projection task of one terminal device and then connect to another terminal device, resulting in low projection efficiency. Summary of the invention

[0005] The present application provides a display device, a terminal device and a screen projection method, which are used to solve the problem that the existing screen projection technology can only project the content on the display interface of the display device to one terminal device at a time, resulting in low screen projection efficiency.

[0006] In a first aspect, the present application provides a display device, the display device comprising:

[0007] A display, wherein the display is used to present a display interface;

[0008] A communicator, the communicator being used for communication connection with a terminal device;

[0009] A controller, the controller being configured to execute:

[0010] In response to a user's selection operation, controlling the display to display a range of an area to be projected and a list of terminal devices, wherein the area to be projected is a partial area of ​​the display interface;

[0011] In response to the determination operation of the target terminal device, a screen recording data stream of the content displayed within the range of the area to be projected is sent to the target terminal device, so that the content displayed within the range of the area to be projected is displayed on the target terminal device.

[0012] In a second aspect, the present application provides a terminal device, the terminal device comprising:

[0013] A communicator, the communicator being used for communication connection with a display device;

[0014] A display, wherein the display is used to present a display interface;

[0015] A controller, the controller being configured to execute:

[0016] The communicator is controlled to receive the screen recording data stream sent by the display device, and the display interface of the display is controlled to display the content corresponding to the screen recording data stream.

[0017] In a third aspect, the present application provides a screen projection method, the method comprising:

[0018] According to the user's selection operation, control the display interface to display the range of the area to be projected and display the terminal device list, wherein the area to be projected is a partial area of ​​the display interface;

[0019] According to the determination operation received from the target terminal device, a screen recording operation is performed on the content displayed within the range of the screen projection area, and the screen recording data stream obtained by the screen recording operation is sent to the target terminal device, so that the target terminal device displays the content displayed within the range on the display interface.

[0020] In a fourth aspect, the present application provides a display device, the display device comprising:

[0021] A display, the display being used to present a display interface, the display interface being divisible so that the display interface includes at least one area to be projected, wherein the area to be projected is a partial area of ​​the display interface;

[0022] A communicator, used for communication connection with at least one terminal device;

[0023] A controller, the controller being configured to execute:

[0024] Send a screen recording data stream of the content displayed within the area to be projected to at least one of the terminal devices, so that the content displayed within the area to be projected is displayed on the terminal device.

[0025] In a fifth aspect, the present application provides an electronic device, comprising a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to implement the steps of the above-mentioned screen projection method when executing the computer program stored in the memory.

[0026] In a sixth aspect, the present application provides a computer-readable storage medium storing a computer program, which implements the steps of the above-mentioned screen projection method when executed by a processor.

[0027] After receiving the screen projection instruction from the mobile control terminal, the display device of the present application displays all the terminal devices that sent the screen projection instruction on the display interface, determines the range of the projected screen area and the terminal device to receive the screen projection on the display interface of the display device, performs a screen recording operation on the content displayed within the range on the display interface, and sends the screen recording data stream obtained by the screen recording operation to the terminal device to receive the screen projection, so that the terminal device to receive the screen projection displays the content displayed within the range on the display interface. In the present application, the display device can project the screen projection area to the corresponding terminal device according to the selected screen projection area, so that the content on the display interface of the display device can be projected to multiple terminal devices at the same time, thereby improving the efficiency of screen projection and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0029] Figure 1 A schematic diagram of a business process of projecting an image of a display interface of a display device to a terminal device provided by the prior art;

[0030] Figure 2 A schematic diagram of the technical route of screen projection provided by existing technologies;

[0031] Figure 3 A schematic diagram of the structure of a display device provided in this application;

[0032] Figure 4 A schematic diagram of a method for sending screen recording data streams of different areas of a display interface of a display device provided by the present application to different terminal devices via different port numbers;

[0033] Figure 5a-Figure 5b A schematic diagram of a method for implementing screen projection by customizing area division provided in this application;

[0034] Figure 6a-6d A schematic diagram of a structure for implementing screen projection in a fixed sub-area division manner provided in this application;

[0035] Figure 7 A schematic diagram of the structure of a terminal device provided in this application;

[0036] Figure 8 A schematic diagram of the structure of a display device provided in this application;

[0037] Fig. 9A schematic diagram of a process of establishing a connection between a controlled terminal and a control terminal provided in this application;

[0038] Fig.10 A schematic diagram of a screen projection process provided for this application;

[0039] Fig.11 A schematic diagram of a screen projection method provided in this application;

[0040] Fig.12 A schematic diagram of a screen projection method provided in this application;

[0041] Fig.13 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0043] Figure 1 A schematic diagram of a process of projecting the content displayed on the display interface of a display device provided by the prior art to a terminal device for display, Figure 2 This is a schematic diagram of the device communication process in the existing local area network. For the convenience of description, the display device is called the controlled end and the terminal device is called the control end. Figure 1 and Figure 2 Describe the screen projection process:

[0044] The control terminal and the controlled terminal are connected to the same local area network, that is, data is exchanged through the same wireless router. The control terminal sends a connection request, and the controlled terminal in the same local area network receives the connection request.

[0045] After receiving the connection request sent by the control end, the controlled end records the content on the display interface to generate streaming media data, and sends the streaming media data to the control end. The control end receives the streaming media data sent by the controlled end and displays the content in the streaming media data on the screen.

[0046] The control end determines the coordinate information of the operation according to the operation performed by the user on the screen, and sends the coordinate information to the controlled end, thereby controlling the content on the display interface of the controlled end.

[0047] In the present application, in order to project the content displayed on the display interface of the display device to multiple terminal devices, the terminal device with the projection demand sends a projection instruction to the display device, and the projection instruction carries the identification information of the terminal device. After receiving the projection instruction, the display device displays the identification symbol of the terminal device on the display interface. By inputting the selection operation of the projection range and the terminal device on the display interface, the display device receives the selection operation, and determines the range of the projection area on the display interface and the target terminal device according to the selection operation. The content displayed within the range of the projection area on the determined display interface is recorded to obtain a screen recording data stream. The display device sends the screen recording data stream to the target terminal device, and the target terminal device displays the content displayed within the range of the projection area on the display interface of the display device. By dividing the display interface of the display device and performing a screen recording operation on the content displayed in each range, the content displayed in different ranges can be projected to multiple smart terminals.

[0048] In order to realize screen projection of the content on the display interface of a display device to multiple terminal devices and improve the screen projection efficiency, the present application provides a display device, a terminal device and a screen projection method.

[0049] Figure 3 A schematic diagram of the structure of a display device provided in the present application, the display device comprising:

[0050] A display 301, which is used to present a display interface;

[0051] A communicator 302, which is used to communicate with a terminal device;

[0052] A controller 303, wherein the controller is used to execute:

[0053] In response to a user's selection operation, controlling the display to display a range of an area to be projected and a list of terminal devices, wherein the area to be projected is a partial area of ​​the display interface;

[0054] In response to the determination operation of the target terminal device, a screen recording data stream of the content displayed within the range of the area to be projected is sent to the target terminal device, so that the content displayed within the range of the area to be projected is displayed on the target terminal device.

[0055] In order to simultaneously project the content displayed on the display interface of the display device to multiple terminal devices, it is first necessary to ensure that the display device performing the projection and the terminal device receiving the projection content are connected to the same local area network.

[0056] When there is a need for screen projection, a screen projection instruction can be sent to the display device. The screen projection instruction can be sent through the terminal device by operating the screen of the terminal device, or it can be sent through the remote control of the display device, or there can be a screen projection button displayed on the display device. When the button is triggered, it is determined that the screen projection instruction has been received. After the display device receives the screen projection instruction, it controls its own display interface to display the terminal device connected to the same local area network as the display device. Or the terminal device displayed on the display interface at this time can be a terminal device that has sent a screen projection instruction to the display device within a set time length.

[0057] In this application, in order to realize the projection of the content displayed on the display interface of the display device to multiple terminal devices, the content displayed on the display interface of the display device can be selected, so as to facilitate the determination of the scope of the projection area. Specifically, a selection operation can be sent to the display device, and when the display device receives the selection operation, the scope of the projection area is determined.

[0058] The selection operation may be an operation of inputting coordinate information of the range of the projection area through a keyboard connected to the display device, or a box selection operation performed on the display interface of the display device through a remote control or a mouse connected to the display device. The display device determines the range of the projection area according to the selection operation.

[0059] In this application, in order to realize the projection of the content displayed on the display interface of the display device to multiple terminal devices, the terminal device displayed on the display interface of the display device can be selected, so as to facilitate the determination of the target terminal device. Specifically, a selection operation can be sent to the display device, and when the display device receives the selection operation, the target terminal device is determined.

[0060] The selection operation may be a click operation performed on a display interface of the display device through a remote controller or a mouse connected to the display device, etc. The display device determines the target terminal device according to the selection operation.

[0061] In the present application, after determining the scope of the screen projection area on the display interface and the target terminal device, the controller 303 performs a screen recording operation on the content displayed within the scope of the screen projection area, and generates a screen recording data stream corresponding to the content displayed within the scope of the screen projection area. After obtaining the screen recording data stream of the content displayed within the scope of the screen projection area, the controller 303 controls the communicator 302 to send the screen recording data stream to the target terminal device. After the communicator 302 sends the screen recording data stream to the target terminal device, the target terminal device displays the content displayed within the scope of the screen projection area.

[0062] In the present application, the display device can project the screen area to the corresponding terminal device according to the selected projection area, so that the content on the display interface of the display device can be projected to multiple terminal devices at the same time, thereby improving the efficiency of projection and improving the user experience.

[0063] In order to accurately send the screen recording data stream to each terminal, based on the above embodiment, in this application, the controller 303 is used to execute:

[0064] Generate a port number corresponding to the target terminal device, and the port number is different from other port numbers generated by the display device; and control the communicator 302 to send the screen recording data stream obtained by the screen recording operation to the target terminal device through the port number.

[0065] In the present application, the content displayed on the display interface of the display device can be projected to multiple terminal devices. In order to send the screen recording data stream recorded for each terminal device to the corresponding terminal device, in the present application, after the target terminal device is determined, the processing module generates a port number for the target terminal device, where the port number is different from the port number that has been generated for other terminal devices. Therefore, the subsequent sending module can send the screen recording data stream to the target terminal through the port number during subsequent sending.

[0066] Figure 4 A schematic diagram of the process of sending screen recording data streams of different areas of the display interface of a display device provided for some embodiments of the present application to different terminal devices through different port numbers, wherein the display device sends the screen recording video stream recorded for the content of projection area A to terminal device a through port number 5000 (UDP PORT=5000), sends the content of area B to terminal device b through port number 5001, and sends the content of area C to terminal device c through port number 5002.

[0067] In order to perform screen recording operations for different terminal devices respectively, thereby generating screen recording video streams for different terminal devices, on the basis of the above embodiments, in the present application, the screen recording module 302 is specifically used to construct a pipeline corresponding to the target terminal device based on the Gstreamer framework, obtain the content displayed within the range on the display interface through the pipeline, and record the obtained content to generate a screen recording data stream.

[0068] The display device in this application is based on the Gstreamer framework, which is a framework for creating streaming media applications. The Gstreamer framework is based on plug-ins, and plug-ins can be easily connected to the pipeline of the Gstreamer framework to achieve code reuse.

[0069] The pipeline of the Gstreamer framework is composed of multiple components. The code implementation of each component can be corresponded to the implementation of the plug-in code. Each component is connected and data flows through pads. Pads can be regarded as interfaces between components. Data flows into and out of components through pads. Pads have special data processing capabilities and can limit the data types that pass through them. Only when the data types allowed to pass through two pads are compatible can they be connected.

[0070] In order to realize screen recording of the content displayed within the projection area, after determining the scope of the projection area and the target terminal device, the display device builds a pipeline corresponding to the target terminal device based on the Gstreamer framework. The content displayed within the scope on the display interface is obtained through the pipeline, and the obtained content is encoded frame by frame to generate a screen recording data stream.

[0071] In the pipeline, the video stream can be recorded through the screen recording command. The screen recording command can refer to the following command:

[0072] gst-launch-1.0 ximagesrc startx=1920 starty=0 endx=2943 endy=768show-pointer=0use-damage=0\

[0073] ! video / x-raw,framerate=30 / 1\

[0074] ! videoconvert! vp8enc! matroskamux! filesink location= / home / nvidia / test / test.mkv

[0075] In the above instructions, the ximagesrc plug-in is used to perform screen recording, and startx, starty, endx, and endy are used to specify the location information of the range of the projection area.

[0076] In order to realize the control of the projected screen area, based on the above embodiments, in this application, the controller 303 is used to execute:

[0077] The control communicator 302 receives control information sent by the target terminal device, wherein the control information includes an operation type of an operation input on a display interface of the target terminal device and location information of the operation;

[0078] Determine an operation instruction and operation content for the content displayed within the range on the display interface according to the operation type and the position information; and adjust the content displayed within the range on the display interface according to the operation instruction and the operation content.

[0079] In the present application, not only can the content displayed in the projection area on the display interface of the display device be projected to the target terminal device, but the content displayed in the projection area on the display interface of the display device can also be adjusted through the target terminal device.

[0080] When there is a need to adjust the content displayed in the projection area on the display interface of the display device, an operation can be input based on the display interface of the terminal device. After receiving the operation, the terminal device determines the operation type and location information of the input operation. The display device and the target terminal device are in the same local area network. The terminal device sends control information carrying the operation type and location information of the operation to the display device through TCP communication or other means. The display device adjusts the content displayed in the projection area on the display interface according to the control information.

[0081] The operation may be a click operation or a drag operation. If the operation is a click operation, the position information is the position of the operation; if the operation is a drag operation, the position information is the position change information between the initial position when the operation starts and the end position when the operation ends.

[0082] The display device determines the operation instruction and operation content of the operation according to the operation type and the position information. When the operation type is click, the display device can determine that the operation instruction is pause, close or open, etc. according to the operation type and the position information of the operation, and determine that the operation content is that the operation instruction is pressed. When the operation type is drag, the display device can determine that the operation instruction is fast forward, rewind and zoom in, etc. according to the operation type and the position information of the operation, and the operation content refers to the degree of adjustment.

[0083] For example, if the input operation is a drag operation, when the display device receives the drag operation and the position information of the drag operation in the control information, the display device determines that the operation instruction corresponding to the drag operation is fast forward, and the operation content, that is, the progress of the fast forward, is determined according to the position information. The display device fast-forwards the content displayed within the projection area according to the operation instruction and the operation content, so that the display device plays the content to be played after the fast forward.

[0084] In order to facilitate the determination of the scope of the projection area, based on the above embodiments, in this application, the controller is used to execute:

[0085] If the received selection operation is a frame selection operation, the frame selection area is determined as the range of the screen projection area.

[0086] When a selection operation is input for the content displayed on the display interface of the display device, so that the display device determines the range of the projection area, the selection operation can be a frame selection operation. When the display device receives the frame selection operation, the frame selection area determined by the frame selection operation is determined as the range of the projection area.

[0087] In the present application, the way to input the frame selection operation can be to directly input the coordinate information of each vertex of the projection area through a keyboard connected to the display device, or to perform a frame selection operation on the display interface of the display device through a remote control or a mouse connected to the display device, or it can also be a frame selection operation sent by a terminal device to the display device. The display device determines the coordinates of each vertex of the projection area according to the frame selection operation, and determines the range of the projection area according to the coordinates of each vertex.

[0088] Figure 5a and Figure 5b A schematic diagram of a structure for implementing screen projection by customizing area division provided in this application. In this embodiment, the terminal device is illustrated as a mobile phone:

[0089] exist Figure 5a In the example, document A and document B are displayed on the display interface of the display device.

[0090] The display device and Phone X, Phone Y, and Phone Z are connected to the same network. When screen projection is required, a screen projection command is sent to the display device.

[0091] exist Figure 5b In the example, a selection operation is input on the display interface of the display device to select the document A area. After receiving the selection operation, the display device determines the display area of ​​the selected document A as the range of the projection area. After determining the range of the projection area, the display interface of the display device displays the terminal devices to be connected: mobile phone Phone X, mobile phone Phone Y, and mobile phone Phone Z. The display device determines mobile phone Phone X as the target terminal device according to the received selection operation of selecting mobile phone Phone X.

[0092] The display device builds a pipeline corresponding to the target terminal device Phone X based on the Gstreamer framework, and realizes the screen recording operation of document A displayed within the scope of the screen projection area on the display interface through the pipeline, and generates a first screen recording data stream. The first screen recording data stream is sent to the target terminal device Phone X, so that Phone X displays document A.

[0093] The pipeline can be recorded by referring to the following command:

[0094] gst-launch-1.0 ximagesrc startx=640 starty=480 endx=1280 endy=960 show-pointer=0 use-damage=0! video / x-raw,framerate=30 / 1! videoconvert! x264enc! h264parse! rtph264pay pt=96! udpsink host=127.0.0.1port=5000

[0095] Specifically, this command implements the screen recording of the content of document A in the projection area, and determines the range of the projection area to be (640, 480) to (1280, 960), i.e., the area where document A is located, by setting the attribute values ​​of the ximagesrc plug-in: startx, starty, endx, endy, and at the same time, performs H.264 encoding on the acquired data stream within the range of the projection area to generate the first screen recording data stream. After determining the mobile phone Phone X to be connected, a port number 5000 corresponding to the mobile phone Phone X is generated, wherein the port number 5000 is different from other port numbers generated by the display device. The display device sends the first screen recording data stream to the mobile phone Phone X in UDP mode through the port number 5000.

[0096] After projecting the content of the area where document A is located to phone X, the display device can continue to determine whether to project the screen to phone Y or phone Z according to the selection operation.

[0097] At this time, you can input a selection operation on the display interface of the display device to select the area where document B is located. The display device determines that the area where document B is located is the range of the screen projection area based on the selection operation. After determining the range of the screen projection area, the display interface of the display device displays the mobile phone Phone Y and mobile phone Phone Z to be connected. The display device determines that the target terminal device is mobile phone Phone Y based on the input selection operation of selecting the target terminal device. The display device builds a pipeline corresponding to the target terminal device mobile phone Phone Y based on the Gstreamer framework, and implements the screen recording operation of document B displayed within the range of the screen projection area on the display interface through the pipeline, generating a second screen recording data stream. The second screen recording data stream is sent to the target terminal device mobile phone Phone Y, so that mobile phone Phone Y displays document B.

[0098] In order to realize screen projection of a designated area of ​​a display device to a terminal device, based on the above embodiments, in this application, the controller 303 is used to execute:

[0099] If the selection operation is a click operation, determine the sub-area where the click operation is located, and determine the sub-area as the range of the screen projection area.

[0100] In order to facilitate the projection of the content displayed on the display device to the terminal device, the display device in this application can be displayed in partitions when displaying content, that is, the display interface is pre-divided into at least two sub-areas, and different content is displayed in each sub-area. For example, there are three sub-areas on the display interface, namely sub-area A, sub-area B and sub-area C, where sub-area A displays a document, sub-area B displays a photo, and sub-area C is playing a video.

[0101] If the display interface of the display device is divided into at least two sub-areas, when determining the range of the projection area based on the content displayed on the display interface of the display device, a selection operation can be sent to the display device. At this time, the selection operation can be a click operation. The display device determines the sub-area where the location of the click operation is located based on the location of the click operation, and determines the sub-area as the range of the projection area.

[0102] Figure 6a-6d A structural diagram of a fixed sub-area division method for implementing screen projection provided by the present application. In this embodiment, the terminal device is illustrated as a mobile phone:

[0103] exist Figure 6a In the embodiment, the display interface of the display device is divided into four sub-areas, and each sub-area displays different contents. Figure 6b As shown, each sub-area of ​​the display interface of the display device displays different documents A, B, C and D. Because the size of the display screen of the display device is fixed, the size of the display interface is also fixed, and the size of each sub-area is also fixed. With a certain point of the display interface as the origin, the range corresponding to each sub-area in the determined coordinate system is determined. Figure 6c As shown, a rectangular coordinate system is established with the vertex in the upper left corner of the display interface (left, right, top and bottom as shown in the figure) as the origin, wherein the coordinate range of the sub-area A in the upper left corner is (0, 0) to (960, 540), the coordinate range of the sub-area B in the upper right corner is (960, 0) to (1920, 540), the coordinate range of the sub-area C in the lower left corner is (0, 540) to (960, 1080), and the coordinate range of the sub-area D in the lower right corner is (960, 540) to (1920, 1080).

[0104] The display device and Phone X, Phone Y, and Phone Z are connected to the same network, and the display device receives a screen projection command.

[0105] The display device receives a click operation on the display interface, and determines that the click operation is applied to the upper left sub-area according to the position of the click operation and the range of each sub-area. Therefore, the range of the upper left sub-area is determined as the range of the projection area. Figure 6d As shown, since document A is displayed in the upper left corner sub-area of ​​the current display interface, it can be considered that the sub-area where document A is located is determined as the range of the projection area.

[0106] After determining that the sub-area where document A is located is the projection area, the display interface of the display device displays the terminal devices to be connected: mobile phone Phone X, mobile phone Phone Y and mobile phone Phone Z. The display device determines that mobile phone Phone X is the target terminal device based on the received selection operation of selecting mobile phone Phone X.

[0107] The display device builds a pipeline corresponding to the target terminal device Phone X based on the Gstreamer framework, and realizes the screen recording operation of document A displayed within the scope of the screen projection area on the display interface through the pipeline, and generates a first screen recording data stream. The first screen recording data stream is sent to the target terminal device Phone X, so that Phone X displays document A.

[0108] The pipeline can be recorded by referring to the following command:

[0109] gst-launch-1.0 ximagesrc startx=0 starty=0 endx=960 endy=540 show-pointer=0 use-damage=0! video / x-raw,framerate=30 / 1! videoconvert! x264enc! h264parse! rtph264pay pt=96! udpsink host=127.0.0.1port=5000

[0110] Specifically, this command implements the screen recording of the content of document A in the projection area, and determines the range of the projection area to be (0, 0) to (960, 540), i.e., the area where document A is located, by setting the attribute values ​​of the ximagesrc plug-in: startx, starty, endx, endy, and at the same time, performs H.264 encoding on the data stream within the range of the projection area to generate the first screen recording data stream. After determining the mobile phone Phone X to be connected, a port number 5000 corresponding to the mobile phone Phone X is generated, wherein the port number 5000 is different from other port numbers generated by the display device. The display device sends the first screen recording data stream to the mobile phone Phone X in UDP mode through the port number 5000.

[0111] After projecting the content of sub-area A in the upper left corner to phone X, the display device can continue to determine whether to project the screen to phone Y or phone Z according to the selection operation.

[0112] At this time, you can input a click operation on the display interface of the display device, click the area where document B is located, and the display device determines that the sub-area B in the upper right corner is the range of the screen projection area based on the click operation. After determining the range of the screen projection area, the display interface of the display device displays the mobile phones Phone Y and Phone Z to be connected. The display device determines that mobile phone Phone Y is the target terminal device based on the received selection operation of selecting mobile phone Phone Y. The display device is based on the Gstreamer framework, and constructs a pipeline corresponding to the target terminal device mobile phone Phone Y. Through the pipeline, the screen recording operation of the sub-area B in the upper right corner of the display interface is implemented to generate a second screen recording data stream. The second screen recording data stream is sent to the target terminal device mobile phone Phone Y, so that mobile phone Phone Y displays document B.

[0113] In order to realize the projection of the content displayed on the display interface of the display device to multiple terminal devices, the present application provides a terminal device, Figure 7 This is a schematic diagram of the structure of the terminal device provided in this application, and the terminal device includes:

[0114] A communicator 701, which is used to communicate with a display device;

[0115] A display 702, the display is used to present a display interface;

[0116] A controller 703, wherein the controller is configured to execute:

[0117] The communicator is controlled to receive the screen recording data stream sent by the display device, and the display interface of the display is controlled to display the content corresponding to the screen recording data stream.

[0118] In order to simultaneously project the content displayed on the display interface of the display device to multiple terminal devices, it is first necessary to ensure that the display device performing the projection and the terminal device receiving the projection content are connected to the same local area network.

[0119] When there is a need for screen projection, a trigger operation for screen projection can be sent to the terminal device. The trigger operation can be a screen projection button displayed on the terminal device. When the button is triggered, the terminal device receives the trigger operation and sends a screen projection instruction to the display device. In addition, the terminal device can also use the solution in the prior art to send a screen projection instruction to the display device.

[0120] According to the description of the above embodiment, after the display device receives the screen projection instruction, it will display the terminal device to be connected on the display interface. After determining the range of the screen projection area and the target terminal device, the display device will perform a screen recording operation on the content displayed within the range of the screen projection area, thereby generating a screen recording data stream and sending the screen recording data stream to the terminal device.

[0121] After receiving the screen recording data stream, the receiving module of the terminal device sends the screen recording data stream to the display module of the terminal device. The display module decodes the screen recording data stream, adjusts each frame of the decoded image according to the size of the terminal device display interface, and displays the adjusted content on the terminal device.

[0122] In order to realize the control of the projected screen area, based on the above embodiments, in this application, the controller 703 is used to execute:

[0123] Receiving an operation input on a display interface of a terminal device, and determining an operation type and first position information of the operation;

[0124] Determining second position information of the operation effect on the display interface of the display device according to the first position information;

[0125] The communicator 701 is controlled to send control information carrying the operation type and the second position information to the display device, so that the display device is adjusted according to the control information.

[0126] In the present application, not only can the content displayed in the projection area on the display interface of the display device be projected to the target terminal device, but the content displayed in the projection area on the display interface of the display device can also be adjusted through the target terminal device.

[0127] When the user needs to adjust the content displayed in the projection area on the display interface of the display device, the user can input an operation based on the terminal device, where the operation can be a click, drag, slide, etc. After the target terminal device receives the operation, it determines the operation type and the first position information of the operation. The operation types include: click operation, drag operation and slide operation, etc.; the first position information is the position information of the operation on the display screen of the terminal device when the operation acts on the target terminal device.

[0128] The first position information is the coordinate information of the operation on the display screen of the terminal device. In order to let the display device know what control is performed on which area, the terminal device determines the second position information when the operation is performed on the display interface of the display device according to the first position information. The terminal device stores the coordinate information of the displayed content on the display interface of the display device. When the terminal device obtains the first position information, it determines the displayed content corresponding to the first position information according to the first position information, and then determines the second position information corresponding to the displayed content on the display interface of the display device.

[0129] The display device and the target terminal device are in the same local area network. The target terminal device sends control information containing the operation type and the second location information to the display device through TCP communication or other means. After receiving the control information, the display device adjusts the content displayed in the projection area on the display interface according to the control information.

[0130] In order to realize screen projection of the content displayed on the display interface of the display device to multiple terminal devices, the present application provides a display device, Figure 8 This is a schematic diagram of the structure of a display device provided in the present application, and the display device includes:

[0131] A display 801, the display is used to present a display interface, the display interface can be divided so that the display interface includes at least one area to be projected, wherein the area to be projected is a partial area of ​​the display interface;

[0132] A communicator 802, configured to communicate with at least one terminal device;

[0133] A controller 803, wherein the controller is used to execute:

[0134] Send a screen recording data stream of the content displayed within the area to be projected to at least one of the terminal devices, so that the content displayed within the area to be projected is displayed on the terminal device.

[0135] In the present application, in order to realize the simultaneous projection of the content displayed on the display interface of the display device to multiple terminal devices, the display interface of the display device can be divided into at least one area to be projected, wherein the method for determining the area to be projected is the same as the method in the above embodiment, and will not be repeated here.

[0136] In the present application, the display device can establish connections with multiple terminal devices, and send screen recording data streams corresponding to the areas to be projected to the multiple terminal devices respectively, so that the multiple terminal devices play the contents in the corresponding screen recording data streams.

[0137] Fig. 9 The schematic diagram of the process of establishing a connection between the controlled end and the control end provided in this application, for the convenience of description, the display device is called the controlled end, and the terminal device is called the control end. The process includes:

[0138] S901: The control end searches for UPNP devices and establishes a connection.

[0139] In the present application, the control end searches for a controlled end in the same local area network as itself through UPNP, and establishes a connection communication with the searched controlled end.

[0140] S902: The control end sends a screen projection instruction to the controlled end.

[0141] After the controlled end receives the screen projection command sent by the control end, the control end to be connected is displayed on the display interface of the controlled end. The controlled end determines the range of the screen projection area and the target control end according to the input selection operation. The screen recording operation is performed on the content displayed within the range of the screen projection area on the display interface to obtain the screen recording data stream.

[0142] S903: The controlled end accesses the controlling end through RTSP.

[0143] The controlled end accesses the control end through RTSP, communicates with the control end, and sends the screen recording data stream to the control end.

[0144] S904: The control end establishes TCP communication with the controlled end and sends control information to the controlled end.

[0145] The control end sends control information to the controlled end through TCP communication.

[0146] Fig.10 A schematic diagram of a screen projection process provided in this application. For the convenience of description, the display device is referred to as the TV end, and the terminal device is referred to as the mobile phone end. The process includes:

[0147] S1001: The TV records the screen and sends the recorded data stream to the mobile phone.

[0148] In this application, after determining the target mobile phone, the TV generates a port number corresponding to the target mobile phone, which is different from the port number generated by the TV. The TV sends the screen recording data stream to the target mobile phone through UPD according to the port number.

[0149] S1002: The mobile phone receives the screen recording data stream and displays the content in the screen recording data stream on the display interface of the mobile phone.

[0150] The mobile phone receives the screen recording data stream sent by the TV according to the port number and decodes the screen recording data stream. The mobile phone adjusts the proportion of the content in the decoded screen recording data stream to adapt to the size of the mobile phone display interface.

[0151] S1003: The mobile phone receives the operation input on the display interface, performs coordinate conversion on the operation, and sends control information carrying the control type and the second position information to the TV side.

[0152] The user inputs an operation on the display interface of the mobile phone, and after receiving the operation, the mobile phone determines the operation type and the first position information of the operation on the display interface of the mobile phone. The mobile phone converts the first position information into the second position information when the operation acts on the display interface of the TV.

[0153] The mobile phone sends the control information to the TV terminal via TCP.

[0154] S1004: The TV adjusts the content displayed within the projection area according to the operation type and the second location information.

[0155] The TV receives the control information sent by the mobile phone, and determines the operation instruction and operation content for the content displayed within the projection area on the display interface according to the operation type and the second location information carried in the control information. The TV adjusts the content displayed within the projection area on the display interface according to the operation instruction and the operation content.

[0156] Fig.11 A schematic diagram of a screen projection method provided in this application, the process includes the following steps:

[0157] S1101: According to the user's selection operation, control the range of the area to be projected on the display interface and display the terminal device list, wherein the area to be projected is a partial area of ​​the display interface.

[0158] S1102: Based on the received determination operation of the target terminal device, a screen recording operation is performed on the content displayed within the range of the screen projection area, and a screen recording data stream obtained by the screen recording operation is sent to the target terminal device, so that the target terminal device displays the content displayed within the range on the display interface.

[0159] Furthermore, after controlling the display interface to display the range of the area to be projected and the list of terminal devices according to the user's selection operation, the method further includes:

[0160] Generate a port number corresponding to the target terminal device, where the port number is different from other port numbers generated by the display device;

[0161] The step of sending the screen recording data stream obtained by the screen recording operation to the target terminal device includes:

[0162] The screen recording data stream obtained by the screen recording operation is sent to the target terminal device through the port number.

[0163] Furthermore, the screen recording operation of the content displayed within the area to be projected includes:

[0164] Based on the Gstreamer framework, a pipeline corresponding to the target terminal device is constructed, the content displayed within the range on the display interface is obtained through the pipeline, and the obtained content is recorded to generate a screen recording data stream.

[0165] Furthermore, the method further comprises:

[0166] receiving control information sent by the target terminal device, the control information including an operation type of an operation input on a display interface of the target terminal device and location information of the operation;

[0167] Determine, according to the operation type and the location information, an operation instruction and operation content for the content displayed within the range on the display interface;

[0168] The content displayed within the range on the display interface is adjusted according to the operation instruction and the operation content.

[0169] The method is applied to a display device. The specific process of the display device executing the screen projection can be referred to the other embodiments mentioned above, and the specific content will not be repeated here.

[0170] Fig.12 A schematic diagram of a screen projection method provided in this application, the process includes the following steps:

[0171] S1201: If a trigger operation for screen projection is received, a screen projection instruction is sent to the display device.

[0172] S1202: Receive the screen recording data stream sent by the display device, and display the content corresponding to the screen recording data stream.

[0173] Furthermore, the method further comprises:

[0174] Receiving an operation input on a display interface of a terminal device, and determining an operation type and first position information of the operation;

[0175] Determining second position information of the operation effect on the display interface of the display device according to the first position information;

[0176] The control information carrying the operation type and the second position information is sent to the display device, so that the display device is adjusted according to the control information.

[0177] Fig.13 This is a schematic diagram of the structure of an electronic device provided by the present application. Based on the above embodiments, the present application embodiment further provides an electronic device, such as Fig.13 As shown, it includes: a processor 131, a communication interface 132, a memory 133 and a communication bus 134, wherein the processor 131, the communication interface 132, and the memory 133 communicate with each other through the communication bus 134;

[0178] The memory 133 stores a computer program. When the program is executed by the processor 131, the processor 131 performs the following steps:

[0179] According to the user's selection operation, control the display interface to display the range of the area to be projected and display the terminal device list, wherein the area to be projected is a partial area of ​​the display interface;

[0180] According to the determination operation received from the target terminal device, a screen recording operation is performed on the content displayed within the range of the screen projection area, and the screen recording data stream obtained by the screen recording operation is sent to the target terminal device, so that the target terminal device displays the content displayed within the range on the display interface.

[0181] Furthermore, after controlling the display interface to display the range of the area to be projected and the list of terminal devices according to the user's selection operation, the method further includes:

[0182] Generate a port number corresponding to the target terminal device, where the port number is different from other port numbers generated by the display device;

[0183] The step of sending the screen recording data stream obtained by the screen recording operation to the target terminal device includes:

[0184] The screen recording data stream obtained by the screen recording operation is sent to the target terminal device through the port number.

[0185] Furthermore, the screen recording operation of the content displayed within the area to be projected includes:

[0186] Based on the Gstreamer framework, a pipeline corresponding to the target terminal device is constructed, the content displayed within the range on the display interface is obtained through the pipeline, and the obtained content is recorded to generate a screen recording data stream.

[0187] Furthermore, the method further comprises:

[0188] receiving control information sent by the target terminal device, the control information including an operation type of an operation input on a display interface of the target terminal device and location information of the operation;

[0189] Determine, according to the operation type and the location information, an operation instruction and operation content for the content displayed within the range on the display interface;

[0190] The content displayed within the range on the display interface is adjusted according to the operation instruction and the operation content.

[0191] The communication interface 132 is used for communication between the above electronic device and other devices.

[0192] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0193] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0194] On the basis of the above embodiments, the present application further provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0195] The memory stores a computer program, and when the program is executed by the processor, the processor performs the following steps:

[0196] If a trigger operation for screen projection is received, a screen projection instruction is sent to the display device;

[0197] Receive the screen recording data stream sent by the display device, and display the content corresponding to the screen recording data stream.

[0198] Furthermore, the method further comprises:

[0199] Receiving an operation input on a display interface of a terminal device, and determining an operation type and first position information of the operation;

[0200] Determining second position information of the operation effect on the display interface of the display device according to the first position information;

[0201] The control information carrying the operation type and the second position information is sent to the display device, so that the display device is adjusted according to the control information.

[0202] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0203] The communication interface is used for communication between the above electronic device and other devices.

[0204] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0205] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0206] On the basis of the above embodiments, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by a processor, and when the program runs on the processor, the processor implements the following steps when executing:

[0207] According to the user's selection operation, control the display interface to display the range of the area to be projected and display the terminal device list, wherein the area to be projected is a partial area of ​​the display interface;

[0208] According to the determination operation received from the target terminal device, a screen recording operation is performed on the content displayed within the range of the screen projection area, and the screen recording data stream obtained by the screen recording operation is sent to the target terminal device, so that the target terminal device displays the content displayed within the range on the display interface.

[0209] Furthermore, after controlling the display interface to display the range of the area to be projected and the list of terminal devices according to the user's selection operation, the method further includes:

[0210] Generate a port number corresponding to the target terminal device, where the port number is different from other port numbers generated by the display device;

[0211] The step of sending the screen recording data stream obtained by the screen recording operation to the target terminal device includes:

[0212] The screen recording data stream obtained by the screen recording operation is sent to the target terminal device through the port number.

[0213] Furthermore, the screen recording operation of the content displayed within the area to be projected includes:

[0214] Based on the Gstreamer framework, a pipeline corresponding to the target terminal device is constructed, the content displayed within the range on the display interface is obtained through the pipeline, and the obtained content is recorded to generate a screen recording data stream.

[0215] Furthermore, the method further comprises:

[0216] receiving control information sent by the target terminal device, the control information including an operation type of an operation input on a display interface of the target terminal device and location information of the operation;

[0217] Determine, according to the operation type and the location information, an operation instruction and operation content for the content displayed within the range on the display interface;

[0218] The content displayed within the range on the display interface is adjusted according to the operation instruction and the operation content.

[0219] Since the principle of solving the problem by the computer-readable medium provided above is similar to that of the screen projection method, after the processor executes the computer program in the computer-readable medium above, the steps to be implemented can refer to the other embodiments above, and the repeated parts will not be repeated.

[0220] The embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by a processor, and when the program is executed on the processor, the processor implements the following steps:

[0221] If a trigger operation for screen projection is received, a screen projection instruction is sent to the display device;

[0222] Receive the screen recording data stream sent by the display device, and display the content corresponding to the screen recording data stream.

[0223] Furthermore, the method further comprises:

[0224] Receiving an operation input on a display interface of a terminal device, and determining an operation type and first position information of the operation;

[0225] Determining second position information of the operation effect on the display interface of the display device according to the first position information;

[0226] The control information carrying the operation type and the second position information is sent to the display device, so that the display device is adjusted according to the control information.

[0227] Since the principle of solving the problem by the computer-readable medium provided above is similar to that of the screen projection method, after the processor executes the computer program in the computer-readable medium above, the steps to be implemented can refer to the other embodiments above, and the repeated parts will not be repeated.

[0228] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0229] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0230] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0231] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0232] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A display device, characterized in that: The display device comprises: A display, wherein the display is used to present a display interface; A communicator, the communicator being used for communication connection with a terminal device; A controller, the controller being configured to execute: In response to a user's selection operation, controlling the display to display a range of an area to be projected and a list of terminal devices, wherein the area to be projected is a partial area of ​​the display interface; In response to a determination operation of the target terminal device, a screen recording data stream of the content displayed within the area to be projected is sent to the target terminal device, so that the content displayed within the area to be projected is displayed on the target terminal device; There are multiple target terminal devices, and each target terminal corresponds to a different screen projection area; Wherein, the controller is used to execute: Generate a port number corresponding to the target terminal device, where the port number is different from other port numbers generated by the display device; and control the communicator to send the screen recording data stream to the target terminal device through the port number; Wherein, the controller is used to execute: The control communicator receives control information sent by the target terminal device, wherein the control information includes an operation type of an operation input on a display interface of the target terminal device and location information of the operation; Determine an operation instruction and operation content for the content displayed within the range on the display interface according to the operation type and the position information; and adjust the content displayed within the range on the display interface according to the operation instruction and the operation content.

2. The display device according to claim 1, characterized in that The controller is also used to execute: It is specifically used to build a pipeline corresponding to the target terminal device based on the Gstreamer framework, obtain the content displayed within the range on the display interface through the pipeline, and record the obtained content to generate a screen recording data stream.

3. The display device according to claim 1, characterized in that The controller is used to perform: If the received selection operation is a frame selection operation, the frame selection area is determined as the range of the screen projection area.

4. The display device according to claim 1, characterized in that The controller is used to perform: If the selection operation is a click operation, determine the sub-area where the click operation is located, and determine the sub-area as the range of the screen projection area.

5. A screen projection method, characterized in that: The method comprises: According to the user's selection operation, control the display interface to display the range of the area to be projected and the list of terminal devices, wherein the area to be projected is a partial area of ​​the display interface; According to the received determination operation of the target terminal device, a screen recording operation is performed on the content displayed within the range of the screen projection area, and a screen recording data stream obtained by the screen recording operation is sent to the target terminal device, so that the target terminal device displays the content displayed within the range on the display interface; There are multiple target terminal devices, and each target terminal corresponds to a different screen projection area; Wherein, after controlling the display interface to display the range of the area to be projected and the list of terminal devices according to the user's selection operation, the method further includes: Generate a port number corresponding to the target terminal device, where the port number is different from other port numbers generated by the display device; The step of sending the screen recording data stream obtained by the screen recording operation to the target terminal device includes: Sending the screen recording data stream obtained by the screen recording operation to the target terminal device through the port number; Wherein, the method further comprises: receiving control information sent by the target terminal device, the control information including an operation type of an operation input on a display interface of the target terminal device and location information of the operation; Determine, according to the operation type and the location information, an operation instruction and operation content for the content displayed within the range on the display interface; The content displayed within the range on the display interface is adjusted according to the operation instruction and the operation content.

Images