Projection screen display control system, method and equipment

Abstract

The embodiment of the invention provides a projection screen display control system, method and device, the system comprises a projection screen device and at least one projected device, and the projection screen device is in communication connection with the projected device; the screen projection equipment is used for sending a screen projection picture; the projected equipment is used for receiving and displaying a screen projection picture; the screen projection equipment is also used for acquiring fixation point information of the user; the screen projection equipment or the projected equipment is also used for determining first content needing to be highlighted in the screen projection picture according to the fixation point information; the projected equipment is used for highlighting the first content. The highlighted display of the first content is realized in the projected device based on the fixation point information, and the display effect and the user experience are improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of display technology, and specifically to a screen projection display control system, method and device. Background Art

[0002] With the continuous development of technology, users are using screen projection applications more and more frequently. At the same time, with the rapid development of large-screen display devices, large-size conference whiteboard devices, commercial display devices, projectors, digital advertising signs, etc. are also being used more and more in conference, education and exhibition scenarios. Therefore, it is often necessary to project the content on small-screen display devices such as laptops, mobile phones, and tablets onto large-screen display devices for display and explanation, or to project the content on large-screen display devices onto multiple small-screen display devices for operation.

[0003] When the contents of a laptop computer, mobile phone, tablet, etc. are projected onto a large-screen display device, the lecturer is usually far away from the large-screen display device and it is not convenient for him to gesture the specific location of the content being explained, making it difficult for the audience to quickly locate the key areas of the explanation screen; when the contents of a large-screen display device are projected onto multiple mobile phones or tablets, more content needs to be displayed on a smaller screen, which makes it difficult for users to see the projected content clearly, affecting the reading and control experience.

[0004] In summary, whether projecting the content on a large-screen display device to multiple small-screen display devices, or projecting the content on a small-screen display device to a large-screen display device, the existing methods cannot achieve ideal display effects and the user experience is poor. Summary of the invention

[0005] Embodiments of the present invention provide a screen projection display control system, method and device to solve the problems of poor display effect and poor user experience in existing screen projection methods.

[0006] In a first aspect, an embodiment of the present invention provides a screen projection display control system, comprising: a screen projection device and at least one projected device, wherein the screen projection device is communicatively connected with the projected device;

[0007] The screen projection device is used to send the projection screen;

[0008] The projected device is used to receive and display the projected screen;

[0009] The screen projection device is also used to obtain the user's gaze point information;

[0010] The projection device or the projection device is further used to determine the first content that needs to be highlighted in the projection screen according to the gaze point information;

[0011] The projected device is used to highlight the first content.

[0012] In one embodiment, when the physical size of the screen of the projection device is larger than the physical size of the screen of the projection device, the projection device is used to highlight the first content, including: the projection device is used to enlarge and display the first content on the screen of the projection device.

[0013] In one embodiment, when the physical size of the screen of the projection device is smaller than the physical size of the screen of the projected device, the projected device is used to highlight the first content, including: the projected device is used to highlight the first content on the screen of the projected device by at least one of highlighting, cursor and box selection.

[0014] In a second aspect, an embodiment of the present invention provides a screen projection display control method, which is applied to a projected device, comprising:

[0015] Obtain screen projection data and gaze point information. The gaze point information is used to indicate the location information of the user's visual focus on the screen of the projection device.

[0016] The projection data is highlighted based on the gaze point information.

[0017] In one embodiment, before obtaining the screen projection data and the gaze point information, the method further includes: obtaining the resolution of the screen projection device;

[0018] The projection data is highlighted based on the gaze point information, including:

[0019] Determine the first content to be highlighted from the projection data according to the gaze point information, the resolution of the projection device, and the resolution of the projection device;

[0020] The first content is highlighted.

[0021] In one embodiment, when the physical size of the screen of the projection device is larger than the physical size of the screen of the projection device, highlighting the first content includes:

[0022] The first content is displayed in an enlarged manner on the screen of the projected device.

[0023] In one embodiment, when the physical size of the screen of the projection device is smaller than the physical size of the screen of the projection device, highlighting the first content includes:

[0024] The first content is highlighted on the screen of the projected device by at least one of highlighting, cursor and box selection.

[0025] In one embodiment, the method further comprises:

[0026] In response to a user's touch operation on the screen of the projected device, acquiring the touch point coordinates;

[0027] Update the touch point coordinates to the target coordinates.

[0028] In one embodiment, the method further comprises:

[0029] Generate click events based on touch operations;

[0030] The generated click event is sent to the projection device so that the projection device can control the projection device according to the click event.

[0031] In one embodiment, obtaining the resolution of the screen projection device includes:

[0032] Receive the response frame of the projection device to the search broadcast frame sent by the projection device;

[0033] Get the resolution of the projection device from the response frame.

[0034] In a third aspect, an embodiment of the present invention provides a screen projection display control method, which is applied to a screen projection device, comprising:

[0035] Get the user's gaze point information;

[0036] Acquire the first content that needs to be highlighted in the projection screen according to the user's gaze point information;

[0037] The first content is highlighted and sent to the projected device.

[0038] In a fourth aspect, an embodiment of the present invention provides an electronic device, including:

[0039] Processor and memory;

[0040] Memory stores computer-executable instructions;

[0041] The processor executes the computer execution instructions stored in the memory, so that the processor executes the screen projection display control method as described in any one of the second aspect or the third aspect.

[0042] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the screen projection display control method as described in any one of the second aspect or the third aspect.

[0043] The screen projection display control system, method and device provided by the embodiment of the present invention include: a screen projection device and at least one projected device, the screen projection device and the projected device are in communication connection; the screen projection device is used to send the projected screen; the projected device is used to receive the projected screen and display it; the screen projection device is also used to obtain the user's gaze point information; the screen projection device or the projected device is also used to determine the first content that needs to be highlighted in the projected screen according to the gaze point information; the projected device is used to highlight the first content. In the projected device, the first content is highlighted based on the gaze point information, which helps to improve the display effect and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0045] Figure 1 A flowchart of a screen projection display control method provided by an embodiment of the present invention;

[0046] Figure 2 A flowchart of a screen projection display control method provided by another embodiment of the present invention;

[0047] Figure 3 A schematic diagram of determining a display ratio according to an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of screen projection display provided by an embodiment of the present invention;

[0049] Figure 5 A schematic diagram of screen projection display provided by another embodiment of the present invention;

[0050] Figure 6 A flowchart of a screen projection display control method provided by another embodiment of the present invention;

[0051] Figure 7 A flowchart of a screen projection display control method provided by another embodiment of the present invention;

[0052] Figure 8 A schematic diagram of the structure of a screen projection display control system provided by an embodiment of the present invention;

[0053] Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

[0054] The above drawings have shown clear embodiments of the present invention, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present invention in any way, but to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0055] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.

[0056] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.

[0057] The serial numbers of the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).

[0058] In the embodiment described in the present application, when the interface of device A is projected onto device X, device A is the projection device and device X is the projected device. The projection display control method provided in the present application is described below from the perspectives of the projected device and the projection device.

[0059] Figure 1 Flow chart of a screen projection display control method provided by an embodiment of the present invention. Figure 1 As shown, the screen projection display control method provided in this embodiment is applied to the projected device and may specifically include:

[0060] S101. Obtain screen projection data and gaze point information, where the gaze point information is used to indicate the position information of the user's visual focus on the screen of the projection device.

[0061] After turning on the screen projection, the interface of the projection device will be projected onto the projected device. The projection data in this embodiment can be understood as data related to the projection screen. The gaze point information in this embodiment is used to indicate the position information of the user's line of sight focus in the projection device screen, for example, it can be the position coordinates of the user's line of sight focus in the projection device screen.

[0062] During the projection process, the commentator looks at the projection device and forms different gaze points by browsing different positions of the screen with his eyes. In this embodiment, the gaze point coordinates can be obtained by positioning the eyeballs based on eye tracking technology. The gaze point coordinates can represent the position of the user's visual focus in the projection device screen. Specifically, when a person sees an object clearly, there is always a rotation of the eyeball. Therefore, the position of a person's eyeballs provides a good indication-the position of the eyeballs can reflect which part of the scene the object he is observing is. Usually, the movement of the human eyeball occurs almost instantaneously and quickly. And then there will be a relatively static staring process of 200ms to 600ms so that the object can be seen clearly. At this time, the eyeball is relatively completely still, accompanied by a slight shake, but in this staring process, this shake is generally less than 1 degree. If the eyeball rotates steadily, this situation only occurs when a person observes a moving object. Other eyeball activities, such as rotation, dispersion, etc., are relatively unimportant in human-computer dialogue and can be ignored. On the whole, when a person sits in front of a computer and operates, he uses the steady staring direction of the eyeball as input. The eye positioning device and the sight tracking device can be fixed in front of the computer screen. When the infrared auxiliary light source (LED array) is used to illuminate the face of a person, a reflection image is formed on the surface of the cornea of ​​the eye. This image is called the Purkinje point. As the human eye stares at different positions on the computer screen, the eyeball will rotate accordingly. The eye positioning device records the initial position of the eyeball. Assuming that the experimenter's head is motionless, the position of the infrared light-emitting diode is fixed, and the eyeball is an approximate sphere, when the eyeball rotates, it can be considered that the absolute position of the Purkinje point does not change; while the position of the iris and the pupil will change accordingly, so the Purkinje point formed by the infrared auxiliary light source on the iris and the relative position relationship between the pupil and the iris will also change. The infrared camera can determine its position in real time. This relative position relationship can be determined by image processing. Then, the direction of the sight input is obtained from the relative position relationship between them to determine the gaze point on the screen and the gaze point coordinates. Then the projection device sends the projection data and the gaze point coordinates to the projected device in real time.

[0063] S102: Highlight the projection data according to the gaze point information.

[0064] After obtaining the projection data and the gaze point information, the projected device can highlight the projection data based on the gaze point information. Specifically, the data that needs to be highlighted can be obtained from the projection data according to the gaze point information. For example, the user's gaze area can be determined according to the gaze point information, and the data in the gaze area is the data that needs to be highlighted. Among them, the gaze area can be determined according to the gaze point coordinates. Specifically, the area that the human eye can gaze at is a circular area. A circular area can be determined by determining the coordinates of the gaze point of the human eye, and then the gaze area can be determined based on the circular area. Therefore, the gaze area may include a rectangular area circumscribed to the circular area on the display panel, or other areas, such as a triangular area.

[0065] The screen projection display control method provided in this embodiment obtains the projection data and gaze point information, and the gaze point information is used to indicate the position information of the user's visual focus in the screen of the projection device. The projection data is highlighted according to the gaze point information. The projection data is highlighted in the projected device based on the gaze point information, which helps to improve the display effect and user experience.

[0066] Normally, the projected device and the projection device have different resolutions, which will cause the same position to have different coordinate information in the projected device and the projection device. Therefore, it is necessary to convert the position coordinates according to the resolutions of the projected device and the projection device, so as to accurately highlight the content that the user is looking at. In other words, in addition to obtaining the projection data and gaze point information, it is also necessary to obtain the resolution of the projection device; then, the first content that needs to be highlighted can be determined from the projection data based on the gaze point information, the resolution of the projection device, and the resolution of the projected device; and the first content is highlighted. For details, please refer to Figure 2 , Figure 2 FIG. 1 is a flowchart of a screen projection display control method provided by another embodiment of the present invention. Figure 2 As shown, the screen projection display control method provided in this embodiment is applied to the projected device and may specifically include:

[0067] S201. Obtain the resolution of the screen projection device.

[0068] The screen projection device and the projected device can communicate wirelessly or wired, for example, through Wifi, Bluetooth, mobile network, etc. Before the screen projection control is turned on, the two will exchange relevant information during the negotiation stage, discover other devices within the communication range through the search broadcast frame, and obtain relevant information from the response frame returned by other devices. In this embodiment, the projected device can send a search broadcast frame through the User Datagram Protocol (UDP), and the content of the search broadcast frame may include its own Internet Protocol (IP) address and Media Access Control Address (MAC) information; when the screen projection device listens to the search broadcast frame sent by the projected device, it sends a response frame including its own IP, MAC and resolution information through UDP.

[0069] In an optional implementation, obtaining the resolution of the screen projection device may specifically include: receiving a response frame of the screen projection device to a search broadcast frame sent by the projected device; and obtaining the resolution of the screen projection device from the response frame.

[0070] S202. Obtain screen projection data and gaze point information, where the gaze point information is used to indicate the position information of the user's visual focus on the screen of the projection device.

[0071] Refer to step S101 in the above embodiment, which will not be described again here.

[0072] S203: Determine the first content that needs to be highlighted from the projection data according to the gaze point information, the resolution of the projection device, and the resolution of the projection device.

[0073] After obtaining the resolution of the projection device and the resolution of the projected device, the display ratio can be determined according to the resolution of the projected device and the resolution of the projection device, and then the coordinate conversion can be performed between the projection device and the projected device according to the display ratio. For example, the position coordinates in the projection device can be converted into the corresponding position coordinates in the projected device.

[0074] After obtaining the resolution of the projection device, the projected device can determine the display ratio based on its own resolution and the obtained resolution of the projection device. Figure 3For example, in the process of establishing a communication connection, the projected device (receiving device) X first sends its own information through broadcasting. After receiving the information of device X, the projection device (sending device) A sends its own resolution (Wa, Ha) to device X in the form of a response frame. After receiving the resolution of device A, device X will calculate it with its own resolution (Wx, Hx) to obtain the display ratio (Wd=Wx / Wa, Hd=Hx / Ha), which can be used for subsequent coordinate conversion.

[0075] The display ratio can be used for coordinate conversion. The coordinates in the projection device can be converted to the coordinates in the projection device according to the display ratio. Figure 3 For example, assuming that device A obtains the gaze point coordinates (Xa, Ya) through the eye tracking module and sends them to device X, device X performs coordinate conversion calculation according to the display ratio (Xb=Xa*Wd, Yb=Ya*Hd), that is, obtains the target coordinates that meet the resolution of device X. At this time, the target coordinates represent the corresponding position of the gaze point coordinates in the projected device.

[0076] According to the determined display ratio, the position information indicating the user's visual focus in the projection device screen can be converted into the position information that needs to be highlighted in the projection device. Further, according to the position information that needs to be highlighted in the projection device, the first content that needs to be highlighted is determined from the projection data.

[0077] S204: Highlight the first content.

[0078] In an optional implementation, the first content may be highlighted according to the needs of the user. For example, a specific implementation method for highlighting may be provided in the interface of the projected device for the user to choose, and the specific implementation method for highlighting includes but is not limited to magnified display, highlighted display, box selection prompt, changing font color, etc.

[0079] In another optional implementation, the specific implementation method for highlighting can also be determined based on the size relationship between the physical sizes of the screens of the projection device and the projection device. Specifically, when the physical size of the screen of the projection device is larger than the physical size of the screen of the projection device, such as projecting from a laptop to a mobile phone, from a smart TV to a tablet device, from a large commercial display screen to a laptop, etc., the first content is highlighted, including: enlarging and displaying the first content on the screen of the projection device. The display effect can be referred to Figure 4As shown, by magnifying the display, only the content of the gaze area is displayed in the small-screen projected device, avoiding the situation where more content is displayed in a smaller screen, resulting in the inability to see the projected content clearly. It should be noted that in this embodiment, the projected screen data can be magnified and displayed according to a preset magnification factor, and the magnification factor can also be determined according to the ratio between the resolution of the projecting device and the user's gaze area, so that the gaze area after magnification covers the entire screen of the projected device.

[0080] When the screen physical size of the projection device is smaller than the screen physical size of the projection device, such as projection from a mobile phone to a laptop, projection from a tablet device to a smart TV, projection from a laptop to a large commercial display screen, etc., the first content is highlighted, including: highlighting the first content on the screen of the projection device by at least one of highlighting, cursor and box selection. The display effect can be referred to Figure 5 As shown, in Figure 5 The target area centered on the target coordinates is highlighted by frame selection, so that the audience can quickly locate the key area of ​​the explanation screen, improving the user experience.

[0081] The screen projection display control method provided in this embodiment, based on the above embodiment, can accurately highlight the content that the user is looking at by obtaining the resolution, projection data and gaze point information of the projection device, and further determining the first content that needs to be highlighted from the projection data according to the gaze point information, the resolution of the projection device and the resolution of the projected device. In addition, for the two situations where the screen physical size of the projection device is larger than the screen physical size of the projected device and the screen physical size of the projection device is smaller than the screen physical size of the projected device, a specific implementation method for highlighting the first content is provided, which significantly improves the user experience.

[0082] It is understandable that the user's operation on the projected device has a higher priority than the control of the projection device's gaze point coordinates on the projected device. Therefore, when a finger clicks on the screen of the projected device, the display area anchor point position will be recalculated based on the finger click position coordinates, and the coordinates of the remote (projection device) gaze point fed back by the communication module will be ignored. In other words, the finger click position coordinates are used as the target coordinates to determine the first content that needs to be highlighted. In other words, the projection display control method provided in this embodiment can also include, on the basis of any of the above embodiments: in response to the user's touch operation on the screen of the projected device, obtaining the touch point coordinates; updating the touch point coordinates to the target coordinates. Determine the first content that needs to be highlighted from the projection data according to the target coordinates.

[0083] In some application scenarios, the projection device and the projected device need to interact with each other. For example, the speaker initiated a voting activity and projected it from his laptop to the smartphones of the audience in the venue. The audience voted by clicking on the smartphone. At this time, the projection device (i.e., the laptop) needs to perform corresponding operations according to the click operations of the audience. The projection display control method provided in this embodiment can also include, based on any of the above embodiments: generating a click event according to a touch operation; sending the generated click event to the projection device, so that the projection device controls the projection device according to the click event.

[0084] Combining the above embodiments, another embodiment of the screen projection display control method is provided. Figure 6 , Figure 6 Flow chart of a screen projection display control method provided by another embodiment of the present invention. Figure 6 As shown, the screen projection display control method provided in this embodiment is applied to the projected device and may specifically include:

[0085] S601. Obtain the resolution and device type of the screen projection device.

[0086] The screen projection device and the projected device can communicate wirelessly or wired, for example, through Wifi, Bluetooth, mobile network, etc. Before the screen projection control is turned on, the two will exchange relevant information during the negotiation stage, discover other devices within the communication range through the search broadcast frame, and obtain relevant information from the response frame returned by other devices. In this embodiment, the projected device can send a search broadcast frame through the User Datagram Protocol (UDP), and the content of the search broadcast frame may include its own Internet Protocol (IP) address and Media Access Control Address (MAC) information; when the screen projection device listens to the search broadcast frame sent by the projected device, it sends a response frame including its own IP, MAC, resolution information and device type information through UDP.

[0087] In an optional implementation, obtaining the resolution and device type of the screen projection device may specifically include: receiving a response frame of the screen projection device to the search broadcast frame sent by the projected device; obtaining the resolution and device type of the screen projection device from the response frame. It should be noted that the device type in this embodiment may be, for example, a type identifier for indicating that the device is a mobile phone, tablet, notebook, all-in-one conference machine, smart TV, large-size commercial display screen, projector, digital advertising sign, etc., and may further include the physical screen size of the device.

[0088] S602: Determine a display ratio according to a resolution of the projected device and a resolution of the projection device.

[0089] After obtaining the resolution of the projection device, the projected device can determine the display ratio based on its own resolution and the obtained resolution of the projection device. Figure 3 For example, in the process of establishing a communication connection, the projected device (receiving device) X first sends its own information through broadcasting. After receiving the information of device X, the projection device (sending device) A sends its own resolution (Wa, Ha) to device X in the form of a response frame. After receiving the resolution of device A, device X will calculate it with its own resolution (Wx, Hx) to obtain the display ratio (Wd=Wx / Wa, Hd=Hx / Ha), which can be used for subsequent coordinate conversion.

[0090] S603: Determine the screen projection type according to the device type of the projected device and the device type of the screen projection device. The screen projection type is used to indicate the size relationship between the physical sizes of the screens of the screen projection device and the projected device.

[0091] After receiving the device type of the projection device, the projected device compares it with its own device type to obtain the projection type. The projection type in this embodiment may include a first projection type and a second projection type, wherein the first projection type is used to indicate that the screen physical size of the projection device is larger than the screen physical size of the projected device, and the second projection type is used to indicate that the screen physical size of the projection device is smaller than the screen physical size of the projected device.

[0092] For example, if the device type of the projected device indicates that the projected device is a mobile phone, and the device type of the projection device indicates that the projection device is an all-in-one conference machine, then the projection type is the first projection type; if the device type of the projected device indicates that the projected device is a smart TV, and the device type of the projection device indicates that the projection device is a tablet, then the projection type is the second projection type.

[0093] S604: Obtain projection data and gaze point coordinates, where the gaze point coordinates represent the coordinates of the user's visual focus on the projection device screen.

[0094] After the screen projection is turned on, the interface of the projection device is projected onto the projected device. At this time, the interpreter looks at the projection device and forms different gaze points by browsing different positions of the screen with his eyes. In this embodiment, the gaze point coordinates can be obtained by positioning the eyeballs based on eye tracking technology. The gaze point coordinates can represent the position of the user's visual focus in the screen of the projection device. Specifically, when a person sees an object clearly, there is always a rotation of the eyeball. Therefore, the position of a person's eyeballs provides a good indication-the position of the eyeballs can reflect which part of the scene the object he is observing is. Usually, the movement of the human eyeball occurs almost instantaneously and quickly. And then there will be a relatively static staring process of 200ms to 600ms in order to see the object clearly. At this time, the eyeball is relatively completely still, accompanied by a slight shake, but in this staring process, this shake is generally less than 1 degree. If the eyeball rotates smoothly, this situation only occurs when a person observes a moving object. Other eye activities, such as rotation, dispersion, etc., are relatively unimportant in human-computer dialogue and can be ignored. Generally speaking, when a person sits in front of a computer and operates, he or she uses the stable gaze direction of the eyeball as input. An eye positioning device and a sight tracking device can be fixed in front of the computer screen. When the infrared auxiliary light source (LED array) illuminates the face of a person, a reflection image is formed on the surface of the cornea of ​​the eye. This image is called the Purkinje point. As the human eye stares at different positions on the computer screen, the eyeball will rotate accordingly. The eye positioning device records the initial position of the eyeball. Assuming that the experimenter's head is motionless, the position of the infrared light-emitting diode is fixed, and the eyeball is an approximate sphere, when the eyeball rotates, it can be assumed that the absolute position of the Purkinje point does not change; while the position of the iris and the pupil will change accordingly, so the Purkinje point formed by the infrared auxiliary light source on the iris and the relative position relationship between the pupil and the iris will also change. The infrared camera can determine their position in real time. This relative position relationship can be determined through image processing. Then, the direction of the sight input is obtained from the relative position relationship between them to determine the gaze point on the screen and the gaze point coordinates. The projection device then sends the projection data and gaze point coordinates to the projected device in real time.

[0095] S605: Determine the target coordinates according to the display ratio and the gaze point coordinates.

[0096] The display ratio can be used for coordinate conversion. The coordinates in the projection device can be converted to the coordinates in the projection device according to the display ratio. Figure 3For example, assuming that device A obtains the gaze point coordinates (Xa, Ya) through the eye tracking module and sends them to device X, device X performs coordinate conversion calculation according to the display ratio (Xb=Xa*Wd, Yb=Ya*Hd), that is, obtains the target coordinates that meet the resolution of device X. At this time, the target coordinates represent the corresponding position of the gaze point coordinates in the projected device.

[0097] S606: Highlight the projection data on the screen of the projected device according to the projection type and target coordinates.

[0098] After obtaining the projection type and target coordinates, the gaze area can be highlighted on the screen of the projected device with the target coordinates as the center for different projection types. Among them, the gaze area can be determined according to the gaze point coordinates. Specifically, the area that the human eye can gaze at is a circular area. A circular area can be determined by determining the coordinates of the gaze point of the human eye, and then the gaze area can be determined based on the circular area. Therefore, the gaze area may include the circumscribed rectangular area of ​​the circular area on the display panel, or other areas, such as triangular areas, etc. By using the display ratio for coordinate conversion, the area corresponding to the gaze area in the projected device can be obtained, and this part of the area can be highlighted. For example, when projecting from a large-screen device to a small-screen device, the gaze area is enlarged and displayed; when projecting from a small-screen device to a large-screen device, the gaze area is highlighted, etc. In this way, the display effect is improved, thereby improving the user experience.

[0099] The screen projection display control method provided in this embodiment obtains the resolution and device type of the screen projection device; determines the display ratio according to the resolution of the projected device and the resolution of the screen projection device; determines the screen projection type according to the device type of the projected device and the device type of the screen projection device, and the screen projection type is used to indicate the size relationship between the physical sizes of the screens of the projecting device and the projected device; obtains the screen projection data and the gaze point coordinates, and the gaze point coordinates indicate the coordinates of the user's line of sight focus in the screen of the screen projection device; determines the target coordinates according to the display ratio and the gaze point coordinates; and highlights the screen projection data in the screen of the projected device according to the screen projection type and the target coordinates, thereby realizing highlighting based on the gaze point coordinates in the projected device, which helps to improve the display effect and user experience.

[0100] On the basis of the above embodiments, the screen projection display control method provided by the present application will be further elaborated for different screen projection types. In an optional implementation, when the screen projection type indicates that the physical size of the screen of the projection device is larger than the physical size of the screen of the projected device, such as projecting from a laptop to a mobile phone, from a smart TV to a tablet device, from a large-size commercial display screen to a laptop, etc., the projection data is highlighted on the screen of the projected device according to the projection type and the target coordinates, which may specifically include: determining the target coordinates as the center point of the display, and enlarging the projection data on the screen of the projected device. The display effect can be referred to Figure 4 As shown, by magnifying the display, only the content of the gaze area is displayed in the small-screen projected device, avoiding the situation where more content is displayed in a smaller screen, resulting in the inability to see the projected content clearly. It should be noted that in this embodiment, the projected screen data can be magnified and displayed according to a preset magnification factor, and the magnification factor can also be determined according to the ratio between the resolution of the projecting device and the user's gaze area, so that the gaze area after magnification covers the entire screen of the projected device.

[0101] The following takes the Android system as an example to illustrate the specific implementation process. Through the SurfaceComposerClient component of the Android system of device A, a display device client is created to connect to SurfaceFlinger, thereby obtaining a Surface object, which, as a consumer, can obtain the data rendered by the current display interface. The data is encoded through the MediaCodec module to obtain a video stream recorded on the screen, which is then sent through the following communication module so that multiple devices X can decode and display the corresponding system interface. For the zoom display of the video area, one way is to move and zoom on the player side based on TextureView. TextureView was introduced in Android4.0 (APIlevel 14). It can project the content stream directly into the View and can be used to implement functions such as Live preview. Unlike SurfaceView, it does not create a separate window in WMS, but as a normal View in the View hierachy, so it can be moved, rotated, scaled, animated, and other changes like other normal Views. Based on the characteristics of TextureView that can display content streams, move and scale, it can realize the dynamic adjustment of the projection area on the screen of the mobile device. Start TextureView on the screen projection receiving end, send the data decoded by MediaCodec to the Surface of TextureView, and then render and display the display interface content of the screen projection sending end.

[0102] In the process of establishing a link through the communication module, the receiving device X first sends its own information through broadcasting. After receiving the information of device X, the sending device A sends its own resolution (Wa, Ha) to device X in the form of a response frame. After receiving the resolution of device A, device X will calculate it with its own resolution (Wx, Hx) to obtain the ratio (Wd = Wx / Wa, Hd = Hx / Ha) for subsequent coordinate conversion. When device A generates the gaze point coordinates (Xa, Ya) through the eye tracking module, it is sent to device X for calculation (Xb = Xa*Wd, Yb = Ya*Hd), and the gaze point coordinate value that meets the resolution of device X can be obtained. When device X displays the projection image of device A, the default anchor point is the center point of View. At this time, the anchor point can be set to (Xb, Yb) through the View methods setPivotX and setPivotY, and then scaled through the setScaleX and setScaleY methods. The scaling values ​​Dx and Dy are based on the original display View area size. The default setting is 2, so as to realize the function of magnifying the local area of ​​the gaze point for display.

[0103] In another optional implementation, when the projection type indicates that the physical size of the projection device is smaller than the physical size of the projection device, such as projection from a mobile phone to a laptop, from a tablet device to a smart TV, from a laptop to a large commercial display screen, etc., the projection data is highlighted on the screen of the projection device according to the projection type and the target coordinates, which may include: highlighting the target area centered on the target coordinates in the projection data on the screen of the projection device by at least one of highlighting, cursor and box selection. The display effect can be referred to Figure 5 As shown, in Figure 5 The target area centered on the target coordinates is highlighted by frame selection, so that the audience can quickly locate the key area of ​​the explanation screen, improving the user experience.

[0104] In order to prevent the screen from moving when the gaze point is slightly offset, causing discomfort to the viewer, the projection display control method provided in this embodiment may further include, on the basis of any of the above embodiments: when a new gaze point coordinate is received, a new target coordinate is determined according to the display ratio and the new gaze point coordinate; the distance between the new target coordinate and the current target coordinate is calculated; when the distance is greater than a preset threshold, the current target coordinate is updated using the new target coordinate. At the projected device end, by setting a threshold for the distance between the new target coordinate and the current target coordinate, the target coordinate is updated only when the distance is greater than the preset threshold, which can effectively avoid frequent movement of the screen and improve the user's viewing experience. Of course, in order to further reduce the communication overhead between the projection device and the projected device, a threshold can also be set for the distance between the new gaze point coordinate and the current gaze point coordinate at the projection device end, and the new gaze point coordinate is sent to the projection device end only when the distance is greater than the preset threshold.

[0105] It is understandable that the user's operation on the projected device has a higher priority than the control of the projection device's gaze point coordinates on the projected device. Therefore, when a finger clicks on the screen of the projected device, the display area anchor point position will be recalculated based on the finger click position coordinates, and the coordinates of the remote (projection device) gaze point fed back by the communication module will be ignored. At this time, the display area adjustment method is consistent with the above scheme, except that the anchor point coordinates are obtained in a different way. In other words, the projection display control method provided in this embodiment can also include, on the basis of any of the above embodiments: obtaining touch point coordinates in response to the user's touch operation on the screen of the projected device; and updating the touch point coordinates to the target coordinates.

[0106] In some application scenarios, the projection device and the projected device need to interact with each other. For example, the speaker initiated a voting activity and projected it from his laptop to the smartphones of the audience in the venue. The audience voted by clicking on the smartphone. At this time, the projection device (i.e., the laptop) needs to perform corresponding operations according to the click operation of the audience. The projection display control method provided in this embodiment can also include, on the basis of any of the above embodiments: generating a click event according to a touch operation; sending the generated click event to the projection device, so that the projection device controls the projection device according to the click event. Taking the Android system as an example, based on the touch event framework and the main class MotionEvent of the Android system, the projected device X can collect its own click events and send them to the projection device A, thereby converting the click operation into a click operation of device A for remote control.

[0107] The above-mentioned embodiment explains the screen projection display control method provided by the present application from the perspective of the projected device. The following will further explain the screen projection display control method provided by the present application from the perspective of the screen projection device.

[0108] Figure 7 FIG. 1 is a flowchart of a screen projection display control method provided by another embodiment of the present invention. Figure 7 As shown, the screen projection display control method provided in this embodiment is applied to a screen projection device and may specifically include:

[0109] S701. Obtain user's gaze point information.

[0110] The user's gaze point information reflects the position information of the user's visual focus on the projection device screen, for example, it can be the position coordinates of the user's visual focus on the projection device screen. During the projection process, the user gazes at the projection device and forms different gaze points by browsing different positions of the screen with his eyes. Based on the eye tracking technology, the user's gaze point information can be obtained by locating the eyeballs.

[0111] S702: Obtain the first content that needs to be highlighted in the projection image according to the user's gaze point information.

[0112] The user's gaze area can be determined based on the user's gaze point information, and the content in the gaze area in the projection screen can be determined as the first content to be highlighted. For example, the content in an area of ​​a preset size in the projection screen can be determined as the first content to be highlighted, with the position indicated by the gaze point information as the center. The area of ​​the preset size can be a circular area, a rectangular area, a triangular area, etc.

[0113] S703: Perform highlighting processing on the first content, and send the highlighted first content to the projected device.

[0114] The first content is highlighted on the projection device, including but not limited to amplifying, highlighting, and selecting the first content. The highlighted first content is sent to the projected device so that the projected device can highlight the first content. It is understandable that the projection device has already highlighted the first content, and the projected device does not need to highlight it again, and only needs to display the highlighted first content. At this time, there is no need to send the gaze point information to the projected device, which reduces the amount of data transmission.

[0115] The screen projection display control method provided in this embodiment obtains the user's gaze point information; obtains the first content that needs to be highlighted in the projection screen according to the user's gaze point information; performs highlighting processing on the first content, and sends the highlighted first content to the projection device. Not only can the first content be highlighted in the projection device to improve the display effect and user experience, but the processing process is concentrated on the projection device end, which can reduce the processing load of the projection device and reduce the data transmission volume to a certain extent (no need to send the gaze point information to the projection device).

[0116] Figure 8 This is a schematic diagram of the structure of a screen projection display control system provided by an embodiment of the present invention. Figure 8 As shown, the screen projection display control system provided by this embodiment includes: a screen projection device 801 and at least one projected device 802, the screen projection device 801 is connected to the projected device 802 in communication; the screen projection device 801 is used to send the projection screen; the projected device 802 is used to receive the projection screen and display it; the screen projection device 801 is also used to obtain the user's gaze point information; the screen projection device 801 or the projected device 802 is also used to determine the first content that needs to be highlighted in the projection screen according to the gaze point information; the projected device 802 is used to highlight the first content. The highlighting of the first content is achieved in the projected device based on the gaze point information, which helps to improve the display effect and user experience.

[0117] It should be noted that there can be multiple projected devices in the screen projection display control system. For example, when the speaker initiates a screen projection from his laptop to the smartphones of the audience in the venue, the laptop is the screen projection device in the system, and all smartphones in the venue that receive the projection data are the projected devices of the system. The screen projection device 801 in the screen projection display control system can also implement screen projection using the screen projection display control method applied to the screen projection device provided in any of the above embodiments. The projected device 802 in the screen projection display control system can also implement screen projection using the screen projection display control method applied to the projected device provided in any of the above embodiments.

[0118] In an optional implementation, when the screen physical size of the projection device 801 is larger than the screen physical size of the projection device 802, the projection device 802 is used to enlarge and display the first content on the screen of the projection device 802. This allows the projection device 802 with a smaller screen to clearly display the content that the user is interested in.

[0119] In an optional implementation, when the screen physical size of the projection device 801 is smaller than the screen physical size of the projection device 802, the projection device 802 is used to highlight the first content on the screen of the projection device 802 by at least one of highlighting, cursor and box selection. This allows the user to quickly locate the key area of ​​the explanation screen in the projection device 802 with a larger screen, thereby improving the user experience.

[0120] The present invention also provides an electronic device, see Fig. 9 As shown, the embodiment of the present invention is only Fig. 9 This is just an example for explanation, and it does not mean that the present invention is limited to this. Fig. 9 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Fig. 9 As shown, the electronic device 90 provided in this embodiment may include: a memory 901, a processor 902 and a bus 903. The bus 903 is used to realize the connection between various components.

[0121] The memory 901 stores a computer program, and when the computer program is executed by the processor 902, the technical solution of any of the above method embodiments can be implemented.

[0122] Among them, the memory 901 and the processor 902 are electrically connected directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines, such as through bus 903. The memory 901 stores a computer program for implementing the projection display control method, including at least one software function module that can be stored in the memory 901 in the form of software or firmware. The processor 902 executes various functional applications and data processing by running the software programs and modules stored in the memory 901.

[0123] The memory 901 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electric erasable programmable read-only memory (EEPROM), etc. The memory 901 is used to store programs, and the processor 902 executes the programs after receiving the execution instruction. Furthermore, the software programs and modules in the above-mentioned memory 901 may also include an operating system, which may include various software components and / or drivers for managing system tasks (such as memory management, storage device control, power management, etc.), and may communicate with various hardware or software components to provide an operating environment for other software components.

[0124] The processor 902 may be an integrated circuit chip with signal processing capability. The processor 902 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. The methods, steps, and logic diagrams disclosed in the embodiments of the present invention may be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor. It is understood that Fig. 9 The structure is only for illustration and may also include Fig. 9 More or fewer components as shown, or with Fig. 9 Different configurations are shown. Fig. 9 Each component shown in can be implemented by hardware and / or software. It should be noted that the electronic device provided in this embodiment can be any electronic device that can display the projected screen, including but not limited to mobile phones, tablets, computers, televisions, large-size commercial display screens, etc. It can be a projection device or a projected device, and can also be a projection device in certain periods of time and a projected device in other periods of time. For example, when casting content from a mobile phone to a TV, the mobile phone is the projection device; and when casting content from a computer to a mobile phone, the mobile phone becomes the projected device.

[0125] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the technical solution of any of the above method embodiments.

[0126] The various embodiments in the present disclosure are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0127] The protection scope of the present disclosure is not limited to the above-mentioned embodiments. Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the scope and spirit of the present disclosure. If these changes and modifications fall within the scope of the claims of the present disclosure and their equivalents, the intention of the present disclosure also includes these changes and modifications.

Claims

1. A screen projection display control system, characterized in that: include: A screen projection device and at least one projected device, wherein the screen projection device is communicatively connected with the projected device; The screen projection device is used to send the projection screen; The projected device is used to receive and display the projected screen; The screen projection device is also used to obtain the user's gaze point information; The screen projection device or the projected device is further used to determine the first content that needs to be highlighted in the projected screen according to the gaze point information; The projected device is used to highlight the first content.

2. The system according to claim 1, characterized in that When the physical size of the screen of the projection device is larger than the physical size of the screen of the projection device, the projection device is used to highlight the first content, including: the projection device is used to enlarge and display the first content on the screen of the projection device.

3. The system according to claim 1, characterized in that When the physical size of the screen of the projection device is smaller than the physical size of the screen of the projected device, the projected device is used to highlight the first content, including: the projected device is used to highlight the first content on the screen of the projected device by at least one of highlighting, cursor and box selection.

4. A screen projection display control method, applied to a projected device, characterized in that: include: Obtaining screen projection data and gaze point information, where the gaze point information is used to indicate the position information of the user's visual focus on the screen of the projection device; The projection data is highlighted according to the gaze point information.

5. The method according to claim 4, characterized in that Before obtaining the projection data and the gaze point information, the method further includes: obtaining the resolution of the projection device; The highlighting of the projection data according to the gaze point information includes: Determining first content to be highlighted from the projection data according to the gaze point information, the resolution of the projection device, and the resolution of the projection device; The first content is highlighted.

6. The method according to claim 5, characterized in that When the physical size of the screen of the screen projection device is larger than the physical size of the screen of the projected device, highlighting the first content includes: The first content is enlarged and displayed on the screen of the projected device.

7. The method according to claim 5, characterized in that When the physical size of the screen of the screen projection device is smaller than the physical size of the screen of the projected device, highlighting the first content includes: The first content is highlighted on the screen of the projected device by at least one of highlighting, cursor and box selection.

8. The method according to claim 4, characterized in that The method further comprises: In response to a touch operation of a user on the screen of the projected device, acquiring touch point coordinates; The touch point coordinates are updated to target coordinates.

9. The method according to claim 8, characterized in that The method further comprises: Generate a click event according to the touch operation; The generated click event is sent to the screen projection device so that the screen projection device controls the screen projection device according to the click event.

10. The method according to any one of claims 5 to 9, characterized in that: The obtaining of the resolution of the screen projection device includes: Receive the response frame of the projection device to the search broadcast frame sent by the projection device; The resolution of the screen projection device is obtained from the response frame.

11. A screen projection display control method, applied to a screen projection device, characterized in that: include: Get the user's gaze point information; Acquire the first content that needs to be highlighted in the projection screen according to the user's gaze point information; The first content is highlighted and the highlighted first content is sent to the projected device.

12. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer execution instructions stored in the memory, so that the processor executes the screen projection display control method as described in any one of claims 4-11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer execution instructions, which, when executed by a processor, are used to implement the screen projection display control method as described in any one of claims 4-11.

Images