Data display method, device, electronic device and computer storage medium
By identifying and prioritizing multiple video sources and displaying them in the same screen in the form of video and snapshots, the problems of many devices and cumbersome operations in the prior art are solved, and the user experience and data management efficiency are improved.
Patent Information
- Application Number
- CN202210171024.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-23
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-02-23
AI Technical Summary
In the prior art, when users view data from different video sources at the same time, there are many devices, high costs, cumbersome operations, poor user experience, and low data viewing efficiency.
By identifying multiple target video sources, determining their priority, and displaying high priority video source data in the same screen area in the form of video, and displaying low priority video source data in the form of snapshots to achieve the difference between different video source data Showcase.
It reduces the number of playback devices, saves costs, simplifies user operations, improves user experience, and facilitates the management and operation of multiple video source information.
Smart Images

Figure CN114567732B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer display technology, and in particular to a data display method, device, electronic equipment and computer storage medium. Background Art
[0002] With the continuous development of science and technology and society, users receive more and more data. Among them, the data provided by various video sources has greatly enriched the work and life of users. In order to improve the efficiency of data acquisition, some users usually have the need to view data from different video sources at the same time. For example, some users want to watch different events at the same time.
[0003] Currently, in order to meet the needs of users to watch data from different video sources at the same time, data from different video sources are usually played on different playback devices. For example, event A is played on a TV, while event B is played on a mobile terminal, and so on.
[0004] However, the data display method in the prior art requires a large number of devices and is costly; and users need to operate different playback devices, which is cumbersome, inefficient in data viewing, and poor in user experience. Summary of the invention
[0005] In view of the above problems, the present invention is proposed to provide a data display method, device, electronic device and computer storage medium that overcome the above problems or at least partially solve the above problems.
[0006] According to one aspect of the present invention, there is provided a data display method, comprising:
[0007] Identify multiple target video sources;
[0008] Determining priorities of the multiple target video sources;
[0009] According to the priorities of the multiple target video sources, identifying a first video source and a second video source from the multiple target video sources; wherein the priority of the first video source is higher than the priority of the second video source;
[0010] Obtain video data corresponding to a first video source, and display the video data corresponding to the first video source in a first screen area; and obtain snapshot data corresponding to a second video source, and display the snapshot data corresponding to the second video source in a second screen area.
[0011] In an optional implementation, the identifying multiple target video sources further comprises: identifying multiple target video sources according to user operations and / or user data;
[0012] Then the determining the priorities of the multiple target video sources further includes:
[0013] Obtain user data and video source information of each target video source;
[0014] For any target video source, calculating the matching degree between the video source information of the target video source and the user data;
[0015] The priorities of the multiple target video sources are determined according to the matching degree between the video source information corresponding to each target video source and the user data.
[0016] In an optional implementation, the identifying multiple target video sources further includes: determining multiple data sources corresponding to the data displayed on the current screen as multiple target video sources;
[0017] Then the determining the priorities of the multiple target video sources further includes:
[0018] Identify the user's visual focus on the screen;
[0019] The priority of the target data sources is determined according to the positional relationship between the screen area corresponding to each target data source and the visual focus.
[0020] In an optional implementation, identifying the user's visual focus on the screen further includes:
[0021] Collect the user image in front of the screen;
[0022] Determining a projection area of the user's face on the screen according to the user image;
[0023] The visual focus of the user on the screen is determined according to the projection area.
[0024] In an optional embodiment, the method further includes:
[0025] Collecting the vertical distance between the user and the screen;
[0026] Determining the position of the user's face according to the vertical distance and the projection area;
[0027] Identifying the user's sight direction according to the user image;
[0028] Then determining the visual focus of the user on the screen according to the projection area further includes: determining the visual focus of the user on the screen according to the position of the user's face and the direction of the user's sight line.
[0029] In an optional implementation, identifying the user's sight direction according to the user image further includes:
[0030] Pre-collecting user reference images of the user at different reference sight lines;
[0031] respectively calculating the similarity between the user image and each of the user reference images;
[0032] The user sight direction is determined according to the similarity between the user image and each of the user reference images.
[0033] In an optional implementation, determining the priority of the target data sources according to the positional relationship between the screen area corresponding to each target data source and the visual focus further includes:
[0034] If the visual focus is only in one screen area, the target data source corresponding to the screen area where the visual focus is located is used as the target data source with the highest priority;
[0035] If the visual focus is located in multiple screen areas, the target data source corresponding to the screen area with the largest intersection area with the visual focus is used as the target data source with the highest priority.
[0036] According to another aspect of the present invention, there is provided a data display device, comprising:
[0037] A first identification module, used for identifying multiple target video sources;
[0038] A priority determination module, used to determine the priorities of the multiple target video sources;
[0039] A second identification module, configured to identify a first video source and a second video source from the multiple target video sources according to the priorities of the multiple target video sources; wherein the priority of the first video source is higher than the priority of the second video source;
[0040] An acquisition module, used to acquire video data corresponding to the first video source, and acquire snapshot data corresponding to the second video source;
[0041] A display module is used to display the video data corresponding to the first video source in the first screen area; and to display the snapshot data corresponding to the second video source in the second screen area.
[0042] In an optional implementation, the first identification module is further used to: identify multiple target video sources according to user operations and / or user data;
[0043] Then the priority determination module is further used for:
[0044] Obtain user data and video source information of each target video source;
[0045] For any target video source, calculating the matching degree between the video source information of the target video source and the user data;
[0046] The priorities of the multiple target video sources are determined according to the matching degree between the video source information corresponding to each target video source and the user data.
[0047] In an optional implementation, the first identification module is further used to: determine multiple data sources corresponding to the data displayed on the current screen as multiple target video sources;
[0048] Then the priority determination module is further used for:
[0049] Identify the user's visual focus on the screen;
[0050] The priority of the target data sources is determined according to the positional relationship between the screen area corresponding to each target data source and the visual focus.
[0051] In an optional implementation, the priority determination module is further used to: collect an image of the user in front of the screen;
[0052] Determining a projection area of the user's face on the screen according to the user image;
[0053] The visual focus of the user on the screen is determined according to the projection area.
[0054] In an optional implementation, the priority determination module is further used to:
[0055] Collecting the vertical distance between the user and the screen;
[0056] Determining the position of the user's face according to the vertical distance and the projection area;
[0057] Identifying the user's sight direction according to the user image;
[0058] The visual focus of the user on the screen is determined according to the position of the user's face and the direction of the user's sight.
[0059] In an optional implementation, the priority determination module is further used to:
[0060] Pre-collecting user reference images of the user at different reference sight lines;
[0061] respectively calculating the similarity between the user image and each of the user reference images;
[0062] The user sight direction is determined according to the similarity between the user image and each of the user reference images.
[0063] In an optional implementation, the priority determination module is further used to:
[0064] If the visual focus is only in one screen area, the target data source corresponding to the screen area where the visual focus is located is used as the target data source with the highest priority;
[0065] If the visual focus is located in multiple screen areas, the target data source corresponding to the screen area with the largest intersection area with the visual focus is used as the target data source with the highest priority.
[0066] According to another aspect of the present invention, there is provided an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;
[0067] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned data display method.
[0068] According to another aspect of the present invention, a computer storage medium is provided, wherein the storage medium stores at least one executable instruction, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned data display method.
[0069] The data display method, device, electronic device and computer storage medium disclosed in the present invention: identify multiple target video sources; determine the priorities of multiple target video sources; identify a first video source and a second video source from the multiple target video sources according to the priorities of the multiple target video sources; the priority of the first video source is higher than the priority of the second video source; display the video data corresponding to the first video source in the first screen area; display the snapshot data corresponding to the second video source in the second screen area. This solution can view data from different video sources on the same screen at the same time, reduce the number of playback devices, save costs, and facilitate users to manage information from multiple video sources; and display data from video sources with high priority in the form of video, and display data from video sources with low priority in the form of snapshots, so as to achieve differentiated display of data from different video sources, facilitate highlighting key video sources, and improve user experience.
[0070] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0072] Figure 1 A schematic diagram of a flow chart of a data display method provided in Embodiment 1 of the present invention is shown;
[0073] Figure 2 A screen display schematic diagram provided by the first embodiment of the present invention is shown;
[0074] Figure 3 A schematic diagram of a data display method according to a second embodiment of the present invention is shown;
[0075] Figure 4 A schematic diagram of a flow chart of a data display method provided in Embodiment 3 of the present invention is shown;
[0076] Figure 5 A schematic diagram showing a flow chart of a method for determining a visual focus provided by Embodiment 3 of the present invention;
[0077] Figure 6 A schematic diagram of a user image provided by Embodiment 3 of the present invention is shown;
[0078] Figure 7 A schematic diagram showing the positional relationship between a screen and an image acquisition device provided in a third embodiment of the present invention is shown;
[0079] Figure 8 A schematic diagram showing a flow chart of another method for determining visual focus provided in Embodiment 3 of the present invention;
[0080] Fig. 9 A schematic diagram showing a reference sight line orientation provided by Embodiment 3 of the present invention is shown;
[0081] Fig.10 A schematic diagram of a user's sight line orientation provided by a third embodiment of the present invention is shown;
[0082] Fig.11 A screen display schematic diagram provided by Embodiment 3 of the present invention is shown;
[0083] Fig.12 Another screen display schematic diagram provided by the third embodiment of the present invention is shown;
[0084] Fig.13 A schematic diagram of the structure of a data display device provided by a fourth embodiment of the present invention is shown;
[0085] Fig.14 A schematic structural diagram of an electronic device provided in Embodiment 6 of the present invention is shown. DETAILED DESCRIPTION
[0086] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.
[0087] Embodiment 1
[0088] Figure 1 The flowchart of a data display method provided by the first embodiment of the present invention is shown. The data display method provided by the embodiment of the present invention can be executed by a preset client, a plug-in (such as a browser plug-in or a desktop plug-in) in the electronic device.
[0089] like Figure 1 As shown, the method comprises the following steps:
[0090] Step S110: identifying multiple target video sources.
[0091] The data that the user currently wants to view simultaneously comes from multiple video sources, which are target video sources. Each video source can provide corresponding video data for the user to watch. In order to improve the subsequent data acquisition efficiency, the address of each target video source can be recorded after identifying multiple target video sources.
[0092] Step S120: Determine the priorities of multiple target video sources.
[0093] In order to facilitate the subsequent display of data corresponding to different target video sources on the same screen in a differentiated manner and improve the user experience, the embodiment of the present invention further determines the priority of each target video source after identifying multiple target video sources. The embodiment of the present invention does not limit the specific method for determining the priority of the target video source.
[0094] In an optional implementation, in order to facilitate the display of data in the target video source, the embodiment of the present invention can store the relevant information of the target video source in the video source queue according to the priority of the target video source. The higher the priority, the closer the target video source is to the head of the queue.
[0095] Step S130: identifying a first video source and a second video source from the multiple target video sources according to the priorities of the multiple target video sources; wherein the priority of the first video source is higher than the priority of the second video source.
[0096] According to the priorities of the target video sources, multiple target video sources are divided into a first video source and a second video source. Specifically, a target video source with a high priority is used as the first video source, and a target video source with a low priority is used as the second video source. For example, the target video source with the highest priority can be used as the first video source, and the other target video sources can be used as the second video source.
[0097] Step S140: obtaining video data corresponding to the first video source, and displaying the video data corresponding to the first video source in the first screen area; and obtaining snapshot data corresponding to the second video source, and displaying the snapshot data corresponding to the second video source in the second screen area.
[0098] In an embodiment of the present invention, data corresponding to multiple target data sources are simultaneously displayed on the same device screen. Specifically, for the first video source, the corresponding video data is obtained, and the video data is displayed in the first screen area of the device screen, that is, the data in the target video source with a high priority is displayed in video form; for the second video source, the corresponding snapshot data is obtained, and the snapshot data is displayed in the second screen area of the device screen. Optionally, the snapshot data corresponding to the second video source can be obtained in the following manner, but not limited to: performing frame extraction processing on the video data corresponding to the second video source to obtain the snapshot data corresponding to the second video source; or, pre-generating the snapshot data corresponding to the second video source (for example, some video sources will automatically generate snapshot data such as exciting replays and gold-winning moments), and obtaining the snapshot data corresponding to the second video source through the corresponding data interface of the second video source.
[0099] In an optional embodiment, to enhance the user's viewing experience, the first screen area may be located in the center of the screen, and / or the first screen area may be larger than any second screen area, etc., so that the user can clearly and conveniently watch the video through the first screen area.
[0100] by Figure 2 For example, multiple target video sources are S1, S2, S3, S4, and S5, and the priority of the multiple target video sources is S1>S2>S3>S4>S5. Then the target video source S1 is used as the first video source, and the video data corresponding to the target video source S1 (corresponding to Figure 2 "Video A" in the first screen area ( Figure 2 The target video sources S2, S3, S4, and S5 are respectively used as the second video source, and the snapshot data corresponding to the target video sources S2, S3, S4, and S5 (corresponding to Figure 2 The "snapshot B", "snapshot C", "snapshot D", "snapshot E" in the same screen are displayed in four second screen areas (corresponding to Figure 2 the upper left, upper right, lower left, and lower right areas in the image).
[0101] The following is a detailed description of the implementation process of the present invention using a specific application scenario:
[0102] Multi-event viewing scenario: The user currently wants to watch Event 1, Event 2, Event 3, Event 4, and Event 5 at the same time. Among them, the video source corresponding to Event 1 has the highest priority, so the video source corresponding to Event 1 is used as the first video source, and the video sources corresponding to Event 2, Event 3, Event 4, and Event 5 are used as the second video source. In this way, Event 1 is displayed in the form of a video in the first screen area of the same screen, and Event 2, Event 3, Event 4, and Event 5 are displayed in the form of snapshots in each second screen area.
[0103] Multi-screen monitoring scenario: The user currently wants to watch the monitoring images of Region 1, Region 2, Region 3, and Region 4 at the same time. Among them, the video source corresponding to Region 1 has the highest priority, so the video source corresponding to Region 1 is used as the first video source, and the video sources corresponding to Region 2, Region 3, and Region 4 are used as the second video source. In this way, the monitoring images of Region 1 are displayed in the form of video in the first screen area of the same screen, and the monitoring images of Region 2, Region 3, and Region 4 are displayed in the form of snapshots in each second screen area.
[0104] Multi-person video conference scenario: User 4 holds a video conference with User 1, User 2, and User 3. User 1 has the highest priority, so the video source corresponding to User 1 is used as the first video source, and the video sources corresponding to User 2 and User 3 are used as the second video source. Thus, the screen corresponding to User 1 is displayed in the first screen area of the same screen in the form of a video, and the screens corresponding to User 2 and User 3 are displayed in the form of snapshots in each second screen area.
[0105] It should be understood that the data display method provided by the embodiment of the present invention includes but is not limited to the above application scenarios, and the above screen areas may correspond to respective screen windows, so the embodiment of the present invention may also be applied to multi-window management scenarios of desktops, etc. Those skilled in the art may apply the data display method provided by the embodiment of the present invention to a scenario where a user simultaneously views data from multiple video sources according to actual application requirements.
[0106] It can be seen that the embodiment of the present invention can simultaneously display the data of multiple video sources in different areas of the same screen, thereby reducing the number of playback devices and saving costs while ensuring that users can view data in different video sources at the same time; and the data of multiple video sources are simultaneously displayed on the same screen, which is convenient for users to manage and operate the information of multiple video sources and improve user experience; and the embodiment of the present invention determines the first video source and the second video source according to the priority of the target video source, and displays the data in the first video source with a high priority in the form of video, and displays the data in the second video source with a low priority in the form of snapshots, thereby realizing differentiated display of data in different video sources, highlighting key video sources, and improving user experience.
[0107] Embodiment 2
[0108] Figure 3 The flowchart of a data display method provided by the second embodiment of the present invention is shown. The data display method provided by the embodiment of the present invention can be executed by a preset client or plug-in in the electronic device.
[0109] like Figure 3 As shown, the method comprises the following steps:
[0110] Step S310: identifying multiple target video sources according to user operations and / or user data.
[0111] In an optional implementation, multiple target video sources are identified according to user operations. In this implementation, the user can select multiple video sources by himself through corresponding operations, and the multiple video sources selected by the user are used as target video sources. This method can improve the flexibility of user operations and enhance user experience.
[0112] In another optional implementation, multiple target video sources are identified based on user data. In this implementation, user data such as user historical playback records and user portrait data can be obtained to determine multiple target video sources. In this way, data in the video source can be automatically displayed without user operation, simplifying user operations and improving user experience.
[0113] In addition, the above two implementations may be combined, that is, obtaining part of the target video source according to user operations and obtaining part of the target data source according to user data, and so on.
[0114] Step S320: determining the priority of multiple target video sources according to the matching degree between each target video source and the user data.
[0115] Specifically, user data and video source information of each target video source are obtained. The user data includes but is not limited to: user historical playback data and / or user portrait data, etc. The video source information includes but is not limited to: video source type and / or video source rating, etc.
[0116] Furthermore, for any target video source, the matching degree between the video source information of the target video source and the user data is calculated. The embodiment of the present invention does not limit the specific matching degree calculation method. For example, the similarity between the video source information and the user data can be calculated, and the higher the similarity, the higher the matching degree; or, a machine learning algorithm can be used to predict the matching degree between the video source information of the target video source and the user data, etc.
[0117] The priority of multiple target video sources is determined according to the matching degree between the video source information corresponding to each target video source and the user data. The priority of a target video source is positively correlated to the matching degree between the video source information of the target video source and the user data, that is, the higher the matching degree, the higher the priority of the target video source.
[0118] Step S330: identifying a first video source and a second video source from the multiple target video sources according to the priorities of the multiple target video sources; wherein the priority of the first video source is higher than the priority of the second video source.
[0119] Step S340, obtaining video data corresponding to the first video source, and displaying the video data corresponding to the first video source in the first screen area; and obtaining snapshot data corresponding to the second video source, and displaying the snapshot data corresponding to the second video source in the second screen area.
[0120] The specific implementation process of step S330 and step S340 can refer to the description of the corresponding parts in embodiment 1, which will not be repeated here.
[0121] It can be seen that the embodiments of the present invention identify multiple target video sources according to user operations and / or user data, thereby improving user operation flexibility and improving user experience; and determine the priorities of multiple target video sources according to the matching degree between the target video sources and the user data, so that the priorities of the target video sources are adapted to the user preferences, and then the data in the video source that the user is most interested in can be displayed in the form of video, and the data in the video source that the user is less interested in can be displayed in the form of snapshots, thereby improving the user experience.
[0122] Embodiment 3
[0123] Figure 4A flow chart of a data display method provided in Embodiment 3 of the present invention is shown. The data display method provided in the embodiment of the present invention can be executed by a preset client and / or plug-in in an electronic device. The execution subject of the embodiment of the present invention can obtain data from relevant sensors (such as cameras, voice acquisition or playback devices and / or gyroscopes, etc.) with the authorization of the user, and execute this solution based on the acquired data.
[0124] like Figure 4 As shown, the method comprises the following steps:
[0125] Step S410: multiple data sources corresponding to the data displayed on the current screen are determined as multiple target video sources.
[0126] The data display method provided by the embodiment of the present invention is applied to a scenario where the current screen has displayed data from multiple data sources, and the multiple data sources corresponding to the data displayed on the current screen are determined as multiple target video sources. And through subsequent steps, the layout of the screen and / or the type of data displayed on the screen are dynamically changed according to the change of the user's visual focus on the screen.
[0127] Step S420, identifying the user's visual focus on the screen.
[0128] The embodiment of the present invention can use real-time recognition or periodic recognition to identify the user's visual focus on the screen. The visual focus reflects the screen position that the user is currently focusing on. The embodiment of the present invention specifically captures the user image in front of the screen through a corresponding image capture device, and then identifies the user's visual focus on the screen based on the captured user image.
[0129] In an optional embodiment, it is possible to use Figure 5 The steps S421 to S423 are shown to determine the visual focus. The specific implementation process is as follows:
[0130] Step S421, capturing an image of the user in front of the screen.
[0131] An image acquisition device (such as a camera, etc.) may be arranged near the screen, and the image acquisition device may acquire an image of the user in front of the screen with the authorization of the user.
[0132] Step S422: Determine the projection area of the user's face on the screen based on the user image.
[0133] After obtaining the user image, the user image reference point is determined. The user image reference point is the vertical projection point of the image acquisition device in the user image. Usually, the user image reference point is the center point of the user image.
[0134] The face recognition technology is used to identify the user face area in the user image, and the embodiment of the present invention does not limit the specific face recognition algorithm. And the coordinates of multiple boundary points of the user face area relative to the user image reference point are calculated.
[0135] The coordinates of the image acquisition device relative to the screen reference point are further determined, and the coordinates of the projection points of the multiple boundary points of the user's face area on the screen are determined based on the coordinates of the image acquisition device relative to the screen reference point and the coordinates of the multiple boundary points of the user's face area relative to the user's image reference point. Finally, the projection area of the user's face on the screen is determined based on the coordinates of the projection points of the multiple boundary points of the user's face area on the screen.
[0136] The following combination Figure 6 and Figure 7 The process of determining the projection area of the user's face on the screen in this embodiment is explained in detail: Figure 6 FIG. 1 shows a schematic diagram of a user image provided by Embodiment 3 of the present invention. Figure 6 As shown in the figure, P is the user image reference point, the shaded part is the user face area in the user image, and Q is a boundary point of the user face area. With the horizontal right direction as the positive direction of the X axis and the vertical upward direction as the positive direction of the Y axis, the coordinates of the boundary point Q relative to the user image reference point P are (x1, y1).
[0137] Figure 7 FIG. 1 shows a schematic diagram of the positional relationship between a screen and an image acquisition device provided by Embodiment 3 of the present invention. Figure 7 As shown, Figure 7 The center O of the middle rectangle is the screen reference point of the screen 71. The length of the screen 71 is L and the height is H. The image acquisition device 72 is set at the top center of the screen. With the horizontal right direction as the positive direction of the X axis and the vertical upward direction as the positive direction of the Y axis, the coordinates of the image acquisition device 72 relative to the screen reference point O are (0, H / 2).
[0138] The coordinate origin conversion is then performed to determine the coordinates of the boundary point Q when the screen reference point O is the coordinate origin, that is, according to the coordinates (x1, y1) of the boundary point Q relative to the user image reference point P and the coordinates (0, H / 2) of the image acquisition device 72 relative to the screen reference point O, the coordinates of the projection point of the boundary point Q on the screen are determined to be (x1, y1+H / 2). In this way, the coordinates of the projection points of each boundary point of the user's face area on the screen can be determined, so that the area surrounded by the projection points is determined as the projection area of the user's face on the screen.
[0139] Step S423: Determine the visual focus of the user on the screen according to the projection area.
[0140] Specifically, the projection area of the user's face on the screen can be determined as the visual focus of the user on the screen.
[0141] Figure 5 The visual focus determination method shown determines the visual focus based on the projection of the user's face on the screen, which can improve the recognition efficiency of the visual focus and improve the overall execution efficiency of the method.
[0142] In another optional implementation, in order to further improve the recognition accuracy of the visual focus, it is possible to use Figure 8 The steps S424 to S427 are shown to determine the visual focus. The specific implementation process is as follows:
[0143] Step S424, capturing the user image in front of the screen.
[0144] Step S425: Determine the projection area of the user's face on the screen based on the user image.
[0145] The specific implementation process of step S424 and step S425 can refer to the description of the corresponding parts in step S421-step S423, which will not be repeated here.
[0146] Step S426, collecting the vertical distance between the user and the screen.
[0147] The vertical distance between the user and the screen is collected through a preset distance sensor.
[0148] Step S427: Determine the position of the user's face according to the vertical distance and the projection area.
[0149] The actual position of the user's face can be determined based on the vertical distance between the user and the screen and the projection area of the user's face on the screen. For example, the coordinates of the user's face boundary point M relative to the screen reference point O are (x2, y2), and the vertical distance between the user and the screen is z2, then the coordinates of the user's face boundary point M are (x2, y2, z2). In this way, the three-dimensional coordinates of each boundary point of the user's face can be determined, and then the user's face position can be determined.
[0150] Step S428: Identify the user's sight direction based on the user image.
[0151] Because the user's line of sight is not always perpendicular to the screen when the user is viewing the screen, in order to improve the recognition accuracy of the visual focus, this embodiment needs to identify the current direction of the user's line of sight.
[0152] As an optional method for identifying the user's line of sight direction, user benchmark images of the user under different benchmark line of sight directions are collected in advance, and then the similarities between the user image and each user benchmark image are calculated respectively, and the user's line of sight direction is determined based on the similarities between the user image and each user benchmark image.
[0153] by Fig. 9 For example, the user can be prompted to stand at a specific position U in front of the screen 71 through voice prompts or text prompts, the line connecting the specific position U and the screen reference point O is perpendicular to the screen, and the distance between the specific position U and the screen reference point O is d1; and the user is guided to look at the screen reference point O and the guide points P1, P2, P3, P4, P5, P6, P7, and P8 in the screen (the guide points are the four vertices of the screen and the midpoints of the four edges). And collect 9 user reference images of the user looking at O, P1, P2, P3, P4, P5, P6, P7, and P8 at the specific position U. And calculate the reference sight direction corresponding to each user reference image. Take the user reference image of the user looking at P3 at a specific position U as an example. The angle between the reference sight line and the screen normal is θ3 (tanθ3 = (H / 2) / d1), and the direction angle of the reference sight line in the plane of the screen is 90 degrees (with O pointing to P1 as 0 degrees and the angle increasing counterclockwise). Then establish the mapping relationship between each user reference image and the corresponding reference sight line.
[0154] It should be understood that the above Fig. 9 This is only a feasible implementation scheme of the embodiment of the present invention. In actual application scenarios, the number of guide points can be determined according to the accuracy requirement. The higher the accuracy requirement, the more guide points there are. Moreover, the embodiment of the present invention does not limit the specific position of the guide points.
[0155] Furthermore, the similarity between the user image and each user reference image is calculated respectively. Specifically, the similarity between the face region in the user image and the face region in each user reference image is calculated. The embodiment of the present invention does not limit the specific similarity algorithm.
[0156] Finally, the user's sight direction is determined based on the similarity between the user image and each user's benchmark image. Among them, since the user is in the same sight direction, the similarity of the corresponding user face area is close to 100%. In this embodiment, one or more user benchmark images with the highest similarity to the user image can be obtained, and the user's sight direction is determined based on the benchmark sight direction of one or more user benchmark images with the highest similarity to the user image. For example, if the user image has the highest similarity with the user benchmark image F3, the benchmark sight direction corresponding to the user benchmark image F3 is used as the user's sight direction; or, if the user image has a high similarity with the user benchmark images F3 and F4, the average of the benchmark sight directions corresponding to the user benchmark images F3 and F4 is used as the user's sight direction.
[0157] Step S429, determining the visual focus of the user on the screen according to the position of the user's face and the direction of the user's line of sight.
[0158] Specifically, the position of the user's face is the three-dimensional coordinates of the boundary points of the user's face. For any boundary point of the user's face, according to the position of the boundary point and the direction of the user's line of sight, the user's line of sight passing through the boundary point can be determined, and then the intersection points of each user's line of sight and the screen can be determined. The area surrounded by the intersection points of each user's line of sight and the screen is the user's visual focus on the screen.
[0159] by Fig.10 For example, the coordinates of the user face boundary point M are (x2, y2, z2), and the user line of sight is: the angle with the screen normal is θ, and the direction angle in the plane where the screen is located is γ. The user line of sight passing through the boundary point M is MM2, M2 is the intersection of the user line of sight MM2 and the screen, and M1 is the projection point of the user face boundary point M on the screen. Then the coordinates of M1 are (x2, y2, 0), and the distance between M1 and M2 is z2*tanθ. Since the direction angle of the user line of sight MM2 in the plane where the screen is located is γ, the coordinates of M2 are (x2+z2*tanθ*cosγ, y2+z2*tanθ*sinγ). In this way, the intersection of the user line of sight passing through each user face boundary point on the screen can be calculated, and the area surrounded by the intersection is the user's visual focus on the screen.
[0160] Step S430 , determining the priorities of the multiple target data sources according to the positional relationship between the screen area corresponding to each target data source and the visual focus.
[0161] Specifically, if the visual focus is located in only one screen area, the target data source corresponding to the screen area where the visual focus is located is used as the target data source with the highest priority; if the visual focus is located in multiple screen areas, the target data source corresponding to the screen area with the largest intersection area with the visual focus is used as the target data source with the highest priority, and the target data source corresponding to the screen area with the second largest intersection area with the visual focus is used as the second highest priority target data source.
[0162] Step S440: identifying a first video source and a second video source from the multiple target video sources according to the priorities of the multiple target video sources; wherein the priority of the first video source is higher than the priority of the second video source.
[0163] Step S450, obtaining video data corresponding to the first video source, and displaying the video data corresponding to the first video source in the first screen area; and obtaining snapshot data corresponding to the second video source, and displaying the snapshot data corresponding to the second video source in the second screen area.
[0164] Among them, if the first video source and the second video source determined in step S440 are consistent with the first video source and the second video source determined last time, the original data acquisition and display logic is maintained; if the first video source and the second video source determined in step S440 are inconsistent with the first video source and the second video source determined last time, the current data acquisition and display logic needs to be changed.
[0165] by Figure 2 and Fig.11 For example, if the first video source determined at the first moment is the target video source S1, and the second video sources are S2, S3, S4, and S5, the screen diagram is as follows: Figure 2 If the first video source determined at the second moment is the target video source S2, and the second video sources are S1, S3, S4, and S5, then the data acquisition and display logic of video sources S1 and S2 need to be exchanged to obtain Fig.11 That is, the video data corresponding to the target video source S2 is displayed in the first screen area (corresponding to Fig.11 The snapshot data of the current video source S1 is displayed in the screen area of the original video source S2 (corresponding to Fig.11 Snapshot A).
[0166] In addition, as an optional implementation, when it is determined in step S440 that a target video source is changed from the second video source to the first video source, the video data of the target video source is not directly displayed in the first screen area, but the screen area currently corresponding to the target video source is first expanded. If it is determined after a preset time that the target video source still needs to be switched to the first video source, step S450 is executed to display the video data of the target video source in the first screen area; if it is determined after the preset time that the target video source no longer needs to be switched to the first video source, the original data acquisition and display logic is maintained.
[0167] by Figure 2 , Fig.11 and Fig.12 For example, if the first video source determined at the first moment is the target video source S1, and the second video sources are S2, S3, S4, and S5, the screen diagram is as follows: Figure 2 If the first video source determined at the second moment is the target video source S2, and the second video sources are S1, S3, S4, and S5, then the area of the screen region where the target video source S2 is located is enlarged, thereby obtaining Fig.12 If it is determined at the third moment that the target video S2 is still the first video source, then Fig.11 If it is determined at the third moment that the target video S2 is no longer the first video source, then the Figure 2 This method can reduce the misjudgment rate, avoid frequent switching of the screen layout, improve user experience and save system resources.
[0168] It can be seen that the embodiment of the present invention determines the priority of the target data source according to the user's visual focus on the screen, so that the display layout and the displayed data format in the screen can be changed without user operation, so that the target data source that the user is most interested in is displayed in the first screen area in the form of video, thereby improving the user experience, improving the efficiency of display layout switching and data format switching, and improving the user experience.
[0169] In addition, as an optional implementation of the present invention, the above-mentioned Example 2 and Example 3 can be combined. Specifically, in the initial stage of implementation, that is, in normal mode, the data corresponding to multiple target video sources can be displayed on the screen at the same time according to the method provided in Example 2, thereby entering the immersive mode. In the immersive mode, while the user is watching the data, the method provided in Example 3 can be used to change the priority of the target video source, and then change the content displayed on the screen. In this way, the user experience can be improved and the user retention rate can be increased. In addition, the embodiment of the present invention can also realize the switching between normal mode and silent mode through corresponding operations (such as gesture operation, shortcut key operation and / or voice operation).
[0170] Embodiment 4
[0171] Fig.13 FIG. 4 shows a schematic diagram of the structure of a data display device provided by Embodiment 4 of the present invention. Fig.13 As shown, the device 1300 includes: a first identification module 1310 , a priority determination module 1320 , a second identification module 1330 , an acquisition module 1340 , and a display module 1350 .
[0172] A first identification module 1310, configured to identify multiple target video sources;
[0173] A priority determination module 1320, configured to determine the priorities of the multiple target video sources;
[0174] A second identification module 1330 is used to identify a first video source and a second video source from the multiple target video sources according to the priorities of the multiple target video sources; wherein the priority of the first video source is higher than the priority of the second video source;
[0175] An acquisition module 1340 is used to acquire video data corresponding to the first video source and acquire snapshot data corresponding to the second video source;
[0176] The display module 1350 is used to display the video data corresponding to the first video source in the first screen area; and to display the snapshot data corresponding to the second video source in the second screen area.
[0177] In an optional implementation, the first identification module is further used to: identify multiple target video sources according to user operations and / or user data;
[0178] Then the priority determination module is further used to: obtain user data and video source information of each target video source;
[0179] For any target video source, calculating the matching degree between the video source information of the target video source and the user data;
[0180] The priorities of the multiple target video sources are determined according to the matching degree between the video source information corresponding to each target video source and the user data.
[0181] In an optional implementation, the first identification module is further used to: determine multiple data sources corresponding to the data displayed on the current screen as multiple target video sources;
[0182] Then the priority determination module is further used to: identify the user's visual focus on the screen;
[0183] The priority of the target data sources is determined according to the positional relationship between the screen area corresponding to each target data source and the visual focus.
[0184] In an optional embodiment, the priority determination module is further used to: capture a user image in front of the screen; determine a projection area of the user's face on the screen based on the user image; and determine the user's visual focus on the screen based on the projection area.
[0185] In an optional embodiment, the priority determination module is further used to: collect the vertical distance between the user and the screen; determine the position of the user's face based on the vertical distance and the projection area; identify the user's line of sight based on the user image; and determine the user's visual focus on the screen based on the position of the user's face and the user's line of sight.
[0186] In an optional implementation, the priority determination module is further used to:
[0187] Pre-collecting user reference images of the user at different reference sight lines;
[0188] respectively calculating the similarity between the user image and each of the user reference images;
[0189] The user sight direction is determined according to the similarity between the user image and each of the user reference images.
[0190] In an optional implementation, the priority determination module is further used to:
[0191] If the visual focus is only in one screen area, the target data source corresponding to the screen area where the visual focus is located is used as the target data source with the highest priority;
[0192] If the visual focus is located in multiple screen areas, the target data source corresponding to the screen area with the largest intersection area with the visual focus is used as the target data source with the highest priority.
[0193] It can be seen that the embodiment of the present invention can simultaneously display the data of multiple video sources in different areas of the same screen, thereby reducing the number of playback devices and saving costs while ensuring that users can view data in different video sources at the same time; and the data of multiple video sources are simultaneously displayed on the same screen, which is convenient for users to manage and operate the information of multiple video sources and improve user experience; and the embodiment of the present invention determines the first video source and the second video source according to the priority of the target video source, and displays the data in the first video source with a high priority in the form of video, and displays the data in the second video source with a low priority in the form of snapshots, thereby realizing differentiated display of data in different video sources, highlighting key video sources, and improving user experience.
[0194] Embodiment 5
[0195] Embodiment 5 of the present invention provides a non-volatile computer storage medium, wherein the computer storage medium stores at least one executable instruction, and the computer executable instruction can execute the data display method in any of the above method embodiments.
[0196] Embodiment 6
[0197] Fig.14 The sixth embodiment of the present invention provides a schematic diagram of the structure of an electronic device. The specific embodiment of the present invention does not limit the specific implementation of the electronic device.
[0198] like Fig.14 As shown, the electronic device may include: a processor (processor) 1402 , a communication interface (Communications Interface) 1404 , a memory (memory) 1406 , and a communication bus 1408 .
[0199] The processor 1402, the communication interface 1404, and the memory 1406 communicate with each other via the communication bus 1408. The communication interface 1404 is used to communicate with other devices such as a client or other server network elements. The processor 1402 is used to execute the program 1410, which can specifically execute the relevant steps in the above method embodiment.
[0200] Specifically, the program 1410 may include program codes, which include computer operation instructions.
[0201] The processor 1402 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the electronic device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.
[0202] The memory 1406 is used to store the program 1410. The memory 1406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The program 1410 may be specifically used to enable the processor 1402 to execute the method in the above method embodiment.
[0203] The algorithm or display provided herein is not inherently related to any particular computer, virtual system or other equipment. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the embodiment of the present invention is not directed to any specific programming language yet. It should be understood that various programming languages can be utilized to realize the content of the present invention described herein, and the description made to specific languages above is for disclosing the best mode of the present invention.
[0204] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.
[0205] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, the various features of the embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed invention requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.
[0206] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.
[0207] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.
[0208] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components according to embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.
[0209] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.
Claims
1. A data display method, characterized in that: include: Determine multiple data sources corresponding to the data displayed on the current screen as multiple target video sources; The projection area of the user's face on the screen is determined as the visual focus of the user on the screen, and the priorities of the multiple target video sources are determined according to the positional relationship between the screen area corresponding to each target data source and the visual focus; wherein, if the visual focus is located in only one screen area, the target data source corresponding to the screen area where the visual focus is located is taken as the target data source with the highest priority; if the visual focus is located in multiple screen areas, the target data source corresponding to the screen area with the largest intersection area with the visual focus is taken as the target data source with the highest priority; According to the priorities of the multiple target video sources, identifying a first video source and a second video source from the multiple target video sources; wherein the priority of the first video source is higher than the priority of the second video source; Obtain video data corresponding to a first video source, and display the video data corresponding to the first video source in a first screen area; and obtain snapshot data corresponding to a second video source, and display the snapshot data corresponding to the second video source in a second screen area.
2. The method according to claim 1, characterized in that Before determining the projection area of the user's face on the screen as the visual focus of the user on the screen, the method further includes: Collect the user image in front of the screen; A projection area of the user's face on the screen is determined according to the user image.
3. The method according to claim 2, characterized in that The method further includes: collecting the vertical distance between the user and the screen; determining the position of the user's face according to the vertical distance and the projection area; and identifying the user's sight direction according to the user image; Then determining the projection area of the user's face on the screen as the visual focus of the user on the screen further includes: determining the visual focus of the user on the screen according to the position of the user's face and the direction of the user's sight.
4. The method according to claim 3, characterized in that The identifying the user's sight direction according to the user image further comprises: Pre-collecting user reference images of the user at different reference sight lines; respectively calculating the similarity between the user image and each of the user reference images; The user sight direction is determined according to the similarity between the user image and each of the user reference images.
5. A data display device, characterized in that: include: A first identification module, used to determine multiple data sources corresponding to the data displayed on the current screen as multiple target video sources; a priority determination module, used to determine the projection area of the user's face on the screen as the user's visual focus on the screen, and determine the priorities of the multiple target video sources according to the positional relationship between the screen area corresponding to each target data source and the visual focus; wherein, if the visual focus is located in only one screen area, the target data source corresponding to the screen area where the visual focus is located is used as the target data source with the highest priority; if the visual focus is located in multiple screen areas, the target data source corresponding to the screen area with the largest intersection area with the visual focus is used as the target data source with the highest priority; A second identification module, configured to identify a first video source and a second video source from the multiple target video sources according to the priorities of the multiple target video sources; wherein the priority of the first video source is higher than the priority of the second video source; An acquisition module, used to acquire video data corresponding to the first video source, and acquire snapshot data corresponding to the second video source; A display module is used to display the video data corresponding to the first video source in the first screen area; and to display the snapshot data corresponding to the second video source in the second screen area.
6. An electronic device, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the data display method according to any one of claims 1-4.
7. A computer storage medium, characterized in that: The storage medium stores at least one executable instruction, and the executable instruction enables the processor to execute an operation corresponding to the data display method according to any one of claims 1 to 4.
