Information display method and apparatus for head-mounted display device

Abstract

This disclosure provides an information display method and apparatus for a head-mounted display device. The method includes: in response to receiving an access request for a target resource page from the head-mounted display device, acquiring source data for multiple functional areas of the target resource page; generating multiple functional interface windows based on the source data of the multiple functional areas; determining a first target window among the multiple functional interface windows and acquiring recommendation information associated with the content displayed in the first target window to generate a corresponding recommendation window; and determining a window to be displayed so that the head-mounted display device displays the window to be displayed, wherein the window to be displayed includes a recommendation window and at least the first target window among the multiple functional interface windows. This disclosure allows for the generation and display of functional interface windows and recommendation windows based on the resource page accessed by the user when using the head-mounted display device, providing an immersive experience while offering reasonable information recommendations to the user.

Description

Technical Field

[0001] This disclosure relates to the field of information technology, and in particular to an information display method and apparatus for a head-mounted display device. Background Technology

[0002] In scenarios such as Virtual Reality (VR), Augmented Reality (AR), or Mixed Reality (MR), the terminal provides users with an interactive and immersive experience by constructing a virtual environment.

[0003] Enhancing the user's immersive experience and providing appropriate information recommendations when accessing a website on a terminal is a problem that urgently needs to be solved. Summary of the Invention

[0004] Embodiments of this disclosure provide an information display method and apparatus for a head-mounted display device.

[0005] According to a first aspect of the present disclosure, an information display method for a head-mounted display device is provided, comprising: in response to receiving an access request for a target resource page of the head-mounted display device, acquiring source data of multiple functional areas in the target resource page; and generating multiple functional interface windows based on the source data of the multiple functional areas;

[0006] A first target window is determined among multiple functional interface windows, and recommendation information associated with the content displayed in the first target window is obtained to generate a corresponding recommendation window; a window to be displayed is determined so that the head-mounted display device displays the window to be displayed, wherein the window to be displayed includes at least the recommendation window and the first target window.

[0007] According to a second aspect of the present disclosure, an information display apparatus for a head-mounted display device is provided, comprising: a source data acquisition module, configured to acquire source data of multiple functional areas in a target resource page in response to receiving an access request from a target resource page of the head-mounted display device; a functional interface window generation module, configured to generate multiple functional interface windows based on the source data of the multiple functional areas; a recommendation window generation module, configured to determine a first target window among the multiple functional interface windows and acquire recommendation information associated with the content displayed in the first target window to generate a corresponding recommendation window; and a display module, configured to determine a window to be displayed so that the head-mounted display device displays the window to be displayed, wherein the window to be displayed includes at least a recommendation window and a first target window.

[0008] According to a third aspect of the present disclosure, a head-mounted display device is provided, the head-mounted display device comprising: a processor; a memory for storing executable instructions of the processor; and a processor for reading the executable instructions from the memory and executing the instructions to implement the information display method for the head-mounted display device described in the first aspect.

[0009] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the information display method for a head-mounted display device described in the first aspect.

[0010] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0011] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0012] Figure 1 This is an exemplary system architecture that can be applied to embodiments of the information display method or apparatus for head-mounted display devices in this application;

[0013] Figure 2 This is a schematic flowchart of an information display method for a head-mounted display device according to one embodiment of the present disclosure;

[0014] Figure 3 This is a schematic diagram showing a functional interface window and a recommendation window in one example of this disclosure;

[0015] Figure 4 This is a flowchart illustrating an information display method for a head-mounted display device according to another embodiment of this disclosure;

[0016] Figure 5 Is Figure 3 An illustration showing the updated display interface window and recommendation window;

[0017] Figure 6 This is a flowchart illustrating step S2 in one embodiment of the present disclosure;

[0018] Figure 7 This is a schematic diagram showing the addition of window labels to the functional interface window and the recommendation window in one of the examples disclosed herein;

[0019] Figure 8This is a schematic diagram showing a specified window magnified according to a user operation command in one of the examples disclosed herein;

[0020] Figure 9 This is a structural block diagram of an information display device for a head-mounted display device according to an embodiment of the present disclosure;

[0021] Figure 10 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0022] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0023] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0024] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0025] It should also be understood that in the embodiments disclosed herein, "multiple" can refer to two or more, and "at least one" can refer to one, two or more.

[0026] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0027] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0028] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0029] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0030] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0031] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0032] This disclosure can be applied to electronic devices such as head-mounted displays and servers, which can operate in conjunction with a wide range of other general-purpose or special-purpose computing system environments or configurations. The electronic devices, such as head-mounted displays and servers, can be described in the general context of computer-executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. The computer system / server can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can reside on local or remote computing system storage media, including storage devices.

[0033] Figure 1 This is an exemplary system architecture that can be applied to embodiments of the information display method or apparatus for head-mounted display devices in this application.

[0034] like Figure 1 As shown, the system architecture may include a head-mounted display device 1, a network 2, and a server 3. Network 2 serves as the medium for communication links between the head-mounted display device 1 and the server 3. Network 2 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0035] The aforementioned head-mounted display device 1 can be an electronic device with image display capabilities, including but not limited to AR smart glasses, VR smart glasses, etc. The head-mounted display device 1 can be an all-in-one device, meaning it can install various client applications. Alternatively, the head-mounted display device 1 can also be a separate device used in conjunction with a terminal device to achieve image display capabilities, in which case the terminal device can perform the computing functions of the head-mounted display device. Here, for ease of representation, both all-in-one devices and separate devices can be collectively referred to as head-mounted display devices.

[0036] The aforementioned head-mounted display device 1 can provide various services, such as obtaining source data of multiple functional areas in the target resource page when receiving an access request for the target resource page, analyzing and processing the source data to generate multiple functional interface windows and recommendation windows, and displaying the windows to be displayed after determining them.

[0037] It should be noted that the method for displaying information provided in the embodiments of this application is generally executed by a head-mounted electronic device 1, and correspondingly, the information display device for the head-mounted display device is generally disposed in the head-mounted electronic device 1.

[0038] It should also be noted that although the solution disclosed in this application can be applied to head-mounted display devices, this does not preclude the possibility that the solution can also be applied to server 3, which can be a backend page server. When the solution disclosed in this application is applied to server 3, upon receiving an access request for a target resource page from the head-mounted display device, the server can obtain source data for multiple functional areas of the target resource page. The server then analyzes and processes the obtained source data to generate multiple functional interface windows and recommendation windows. Finally, the server can determine the window to be displayed and send it to the head-mounted display device so that the head-mounted display device can display the window. In this case, the information display method for the head-mounted display device can be executed by server 3, and correspondingly, the information display device for the head-mounted display device can also be located in server 3.

[0039] Exemplary methods

[0040] Figure 2 This is a flowchart illustrating an information display method for a head-mounted display device according to one embodiment of this disclosure. Embodiments of this disclosure can be applied to electronic devices, such as… Figure 2 As shown, it includes the following steps:

[0041] S2: In response to receiving an access request for the target resource page from the head-mounted display device, obtain source data for multiple functional areas in the target resource page.

[0042] The entity performing the information display method for a head-mounted display device (e.g., Figure 1When a head-mounted display device (such as the one shown) receives an access request for a target resource page, it can obtain the source data of each functional area within the target resource page. This access request can be generated in various ways. For example, it could be generated by a user wearing the head-mounted display device clicking a link to access the target resource. Upon receiving this access request, the executing entity can obtain the source data of each functional area within the target resource page from a backend page server. The target resource page can be various types of interfaces, such as an app interface or a webpage.

[0043] In an optional embodiment of this disclosure, the target resource page can be an AR page, VR page, or MR page, and the target resource page can be pre-divided according to functional areas. When a user accesses the target resource page through a head-mounted display device, the source data of each pre-divided functional area can be directly obtained. For example, when the target resource page includes N functional areas, it can be pre-divided according to N functional areas, and the aforementioned execution entity can directly obtain the source data corresponding to the N functional areas. Here, N is an integer greater than or equal to 1.

[0044] In another optional embodiment of this disclosure, the target resource page can be a conventional resource page, in which each functional area is displayed in a single window without being divided. When the server of the target resource page receives an access request for the target resource page, the backend page server or the head-mounted display device can detect whether the user's access request for the target resource page was issued through the head-mounted display device in various ways. For example, the backend page server can determine whether the user's access request for the target resource page was issued through the head-mounted display device by detecting whether the device ID in the access request is the device ID of the head-mounted display device. When an access request for the target resource page is detected by the user through the head-mounted display device, the aforementioned execution entity can extract the source data of multiple functional areas in the target resource page. For example, when the target resource page includes N functional areas, the aforementioned execution entity can obtain the source data corresponding to the N functional areas divided by the backend page server, or the aforementioned execution entity can also obtain the target resource interface and then obtain the source data of each functional area according to the functional division. Wherein, N is an integer greater than or equal to 1. S4: Generate multiple functional interface windows based on the source data of multiple functional areas.

[0045] In the embodiments of this disclosure, the source data of each functional area from the acquired source data of multiple functional areas is filled into a functional interface window, thereby generating multiple functional interface windows. Specifically, for the source data of any functional area among the multiple functional areas, the source data of that functional area can be filled into a pre-built window, thereby generating the functional interface window for that functional area. For example, for the source data of N functional areas, according to the layout of each functional area in the N functional areas, the source data of each functional area is filled into a functional interface window, thereby generating N functional interface windows.

[0046] S6: Determine the first target window among multiple functional interface windows, and obtain the recommendation information associated with the content displayed in the first target window to generate the corresponding recommendation window.

[0047] In one optional embodiment of this disclosure, a window can be randomly selected from multiple functional interface windows as the first target window. Alternatively, the window that the user is focusing on can be determined as the first target window by identifying the user's gaze point.

[0048] In another optional embodiment of this disclosure, the window with the highest popularity among multiple functional interface windows can be determined as the first target window. The popularity of the functional interface window can be determined based on the number of online viewers or by other methods (such as clicks, comments, etc.).

[0049] In another optional embodiment of this disclosure, the first target window can be determined based on the degree of correspondence between the access request of the target resource page and the themes of multiple functional interface windows. For example, the access request of the target resource page is a video access request for a specified ball game, and the multiple functional pages include a video window for the specified ball game, lineup windows for both teams in the ball game, match score windows for both teams in the ball game, historical match results windows for both teams in the ball game, and news windows about the specified ball game, etc. The video window for the specified ball game can be used as the first target window.

[0050] After determining the first target window, the aforementioned execution entity can process the display content of the first target window in various ways to generate recommendation information associated with the display content, thereby generating a recommendation window that includes the recommendation information.

[0051] S8: Determine the window to be displayed so that the head-mounted display device can display the window to be displayed.

[0052] The aforementioned execution entity can determine the windows to be displayed from all windows through various methods, so that the head-mounted display device can display the acquired windows. The windows to be displayed can include at least the generated recommended window and the first target window among the generated multiple functional interfaces. For example, the execution entity can determine all functional interface windows and recommended windows as the windows to be displayed, in which case the head-mounted display device can display all functional interface windows and recommended windows. Alternatively, the execution entity can determine only the first target window and recommended window as the windows to be displayed, in which case the head-mounted display device can display the first target window and recommended window. As an example, the execution entity can first determine the first target window and recommended window as the windows to be displayed, and then determine the windows to be displayed from other functional interface windows besides the first target window through random methods. It is understood that in the actual use of the head-mounted display device, the determined windows to be displayed are variable; therefore, the windows displayed by the head-mounted display device are also variable.

[0053] Typically, the pose information of a head-mounted display device changes as the user's head moves while wearing the device. This pose information characterizes the head-mounted display device's position and orientation in space. The aforementioned head-mounted display device can adjust its current display based on its current pose information. For example, if the pose information meets preset conditions, the head-mounted display device can select only the first target window and the recommended window for display. These preset conditions could be that the angle between the user's line of sight and the horizontal plane is greater than a preset angle, or that the pose information of the head-mounted display device changes within a threshold range over a preset time period. In this example, the pose information meeting the preset conditions indicates that the user needs to view the first target window for an extended period; therefore, to reduce computational load and power consumption, windows that the user is not interested in can be omitted.

[0054] Figure 3 This is a schematic diagram illustrating the functional interface window and recommendation window in one example of this disclosure. For example... Figure 3 As shown, after a user accesses a target resource page using a head-mounted display device, multiple functional interface windows are generated based on the source data of multiple functional areas of the target resource page. In this example, the target resource page is a football match X, and the multiple functional interface windows include functional interface window A, functional interface window B, and functional interface window C, etc. For example, functional interface window A is a video window, functional interface window B is a lineup introduction window for the participating teams in football match X, and functional interface window C is a performance window for the participating teams (e.g., a group stage points window). In this example, the first target window is functional interface window A, and the recommended window is generated based on the content displayed in functional interface window A. The recommended window can recommend information about the participating teams' jerseys for sale. Figure 3 In this process, functional interface window A, functional interface window B, and the recommended window can be determined from the above-mentioned recommended window, functional interface window A, functional interface window B, and functional interface window C as windows to be displayed, so that the head-mounted display device can display functional interface window A, functional interface window B, and the recommended window.

[0055] In this embodiment, when a user accesses a target resource page using a head-mounted display device, the target resource page can be split into multiple functional interface windows. Information is recommended to the first target window through a recommendation window. Then, the head-mounted display device can selectively display each window to avoid interference from functional information that the user does not care about. This provides the user with an immersive experience while making reasonable information recommendations, resulting in a good user experience.

[0056] Figure 4 This is a flowchart illustrating an information display method for a head-mounted display device according to another embodiment of this disclosure, as shown below. Figure 4 As shown, after step S8, the following steps are also included:

[0057] S10: Based on the pose change information of the head-mounted display device, adjust the window to be displayed so that the head-mounted display device can display the adjusted window.

[0058] As the user's head moves while wearing a head-mounted display (HUD), the HUD's pose information changes. The HUD can then adjust the displayed windows in real time based on this changed pose information, thereby updating the rendered image. This ensures that the window seen by the user changes with head movement. For example, when the HUD is in the pose indicated by the first pose information, it displays the recommended window and all functional interface windows. When the user's head moves, changing the HUD's pose, the adjusted displayed windows can become the recommended window and the first target window. Rendering these windows allows the user to see both the recommended window and the first target window through the HUD.

[0059] Figure 5 Is Figure 3 This is a diagram illustrating the user's change in the pose of the head-mounted display device. The head-mounted display device currently displays function interface window A, function interface window B, and a recommendation window, as shown below. Figure 3 As shown. When the user updates the pose information of the head-mounted display device through head movements, the windows to be displayed can be redefined as the functional interface window A and the recommended window. The head-mounted display device can then render the redefined windows and update the windows displayed on the device to the functional interface window A and the recommended window, as shown. Figure 5 As shown. The content of the recommendation window may not change with updates to the display window; for example, the recommendation window may continue to display recommendations for functional interface window A. Alternatively, the content of the recommendation window may change with updates to the display window; for example, updating the content of the recommendation window for functional interface window B, or recommending historical match lineups for teams in football match X, or recommending the win rates of each lineup for teams in football match X, and so on.

[0060] In this embodiment, the current display window can be quickly and accurately adjusted based on the pose change information of the head-mounted display device, thereby meeting the user's intelligent control needs for the display window.

[0061] Figure 6 This is a flowchart illustrating step S2 in one embodiment of this disclosure. Figure 6 As shown, step S2 includes:

[0062] S2-1: Obtain multiple target functional interface window templates from a pre-built functional interface window template library.

[0063] In this embodiment, multiple functional interface window templates can be pre-stored in a functional interface window template library. Each functional interface window template has a corresponding window frame. For example, the window frame specifies how many unit interfaces the template includes, the display size of each unit interface, and parameters such as the font type and font size in each unit interface. Then, based on the source data of multiple functional areas in the target resource page, suitable templates are matched to obtain multiple target functional interface window templates.

[0064] S2-2: Fill the source data of multiple functional areas into multiple target functional interface templates to generate multiple functional interface windows.

[0065] In this embodiment, by pre-setting a functional interface window template library with functional interface window templates, the generation speed of functional interface windows can be greatly improved by reusing templates.

[0066] In an optional embodiment of this disclosure, step S2-2 may include:

[0067] S2-2-1: Based on the source data of multiple functional areas, generate multiple window labels that correspond one-to-one with multiple functional interface windows.

[0068] Specifically, corresponding window labels can be generated based on the classification results of the displayed content of the source data for each functional area. For example, such as... Figure 7As shown, when the displayed content of the source data in the functional area is live streaming, the corresponding window label is Live Streaming; when the displayed content of the source data in the functional area is the lineup of the participating teams, the corresponding window label is Lineup.

[0069] S2-2-2: Based on source data from multiple functional areas, generate multiple sets of window display content that correspond one-to-one with multiple functional interface windows.

[0070] For example, such as Figure 7 As shown, video data is used as the display content of the live stream window, lineup information is used as the display content of the lineup window, and so on.

[0071] S2-2-3: Fill multiple sets of window display content and multiple window labels into multiple target functional interface templates to generate multiple functional interface windows, such as... Figure 7 As shown.

[0072] In this embodiment, corresponding window labels can be added to each functional interface window, allowing users to quickly determine the type of content displayed in each functional interface window based on the window labels. This facilitates users in quickly finding the functional interface window they need and improves user operation efficiency.

[0073] In an optional embodiment of this disclosure, after step S8, the method further includes: for one of the currently displayed functional interface window and the recommendation window, in response to receiving a user operation instruction for that window, controlling the window to perform a corresponding action according to the user operation instruction. The user operation instruction may include instructions such as window zoom-in, window zoom-out, window push-away, window zoom-in, window drag, and window close.

[0074] In one example of this disclosure, a user can input user operation commands through other devices used to sense user behavior. For example, after the user performs a predetermined gesture, the server collects the data through the device, finds the corresponding window operation behavior, and then performs the corresponding action. For instance, the window operation behavior corresponding to the predetermined gesture might be that when a window is zoomed in, the selected window can be zoomed in.

[0075] In another example of this disclosure, the user can input user operation commands through the head-mounted display device, such as by inputting user operation commands through the function buttons on the head-mounted display device. First, the window to be operated is determined, for example, the currently selected window of the head-mounted display device is selected as the selected window, and then the selected window is operated according to the user operation command.

[0076] Figure 8 This is a schematic diagram showing a specified window magnified according to a user operation command, as an example of this disclosure. For example... Figure 8 As shown, when a user enters a window magnification command on a selected window, the selected window can be magnified.

[0077] In related technologies, various functional areas of a target resource page are displayed in the same window, or although the functional areas are not in the same window, they are interconnected. Users often need to operate on the target resource page as a whole, and cannot zoom in or out on the content of a single functional area. In this embodiment, after splitting the target resource page into different functional interface windows, each functional interface window and recommendation window can be flexibly operated on according to the user's operation instructions, resulting in a better user experience.

[0078] In an optional embodiment of this disclosure, obtaining recommendation information associated with the content displayed in the first target window includes: obtaining source data of the content displayed in the first target window; determining target points of interest based on the source data of the content displayed in the first target window; and generating recommendation information based on the target points of interest.

[0079] Specifically, interest point analysis can be performed on the source data of the content displayed in the first target window in various ways. For example, target interest points can be determined by word segmentation and analysis of the text content in the first target window; target interest points can also be determined by object recognition and analysis of image content; target interest points can be determined by converting audio content into text, performing word segmentation, and then analyzing the text, and so on. Based on the interest point analysis results, target interest points can be determined, and then recommendation information associated with the target interest points can be generated.

[0080] In this embodiment, points of interest can be determined based on the content displayed in the window, thereby generating recommended information that matches the user's interests, making the information recommendations more targeted.

[0081] In an optional embodiment of this disclosure, when the content displayed in the first target window is a target video, the step of determining the target point of interest based on the source data of the content displayed in the first target window may include: obtaining user comment data associated with the first target video frame image from the source data of the target video; determining the first point of interest associated with the first target video frame image based on the user comment data associated with the first target video frame image, and using the first point of interest as the target point of interest.

[0082] Specifically, firstly, image data of a first target video frame image is obtained from the target video. This first target video frame image can be a keyframe image or a specific frame image of the target video. For example, the target video frame image could be a frame image showing a close-up of a specific player. Then, user comment data associated with the first target video frame image is obtained. A first point of interest is determined based on this user comment data; for example, the first point of interest might be the style, material, and price of the specified player's shoes, etc., and this first point of interest is used as the target point of interest.

[0083] In this embodiment, user comments associated with image frames in a video can be used to quickly determine points of interest, so as to recommend appropriate information to users.

[0084] In one embodiment of this disclosure, the step of determining a first point of interest associated with a target video frame image based on user comment data associated with the first target video frame image includes: acquiring all user comment data associated with the first target video frame image; determining multiple points of interest based on all user comment data; counting the number of people interested in the multiple points of interest, and selecting the point of interest with the largest number of interested people as the first point of interest.

[0085] Specifically, analyzing all user comment data associated with the first target video frame image may identify multiple points of interest. For example, if the target video frame image is a close-up of a specific player, the points of interest analyzed from all user comment data may include points of interest specifically about that player, the shoes the player is wearing, the wristband the player is wearing, and so on. For each point of interest, the point of interest with the largest number of interested users is selected as the first point of interest.

[0086] In this embodiment, multiple points of interest can be determined based on the comment information of all users in the video frame image, and then corresponding recommendation information can be generated based on the points of interest with the most interested users, so that the generated recommendation information can satisfy the interests of most viewers.

[0087] In one embodiment of this disclosure, the step of determining a first point of interest associated with a target video frame image based on user comment data associated with the first target video frame image includes: acquiring target comment data associated with the first target video frame image, wherein the target comment data is comment data of a target user; and determining the first point of interest based on the target comment data.

[0088] In this embodiment, points of interest can be determined based on the comment information of the target user in the video frame image, and then corresponding recommendation information can be generated so that the generated recommendation information meets the personalized needs of the target user.

[0089] In one embodiment of this disclosure, when the content displayed in the first target window is a live video, the step of determining a target point of interest based on the source data of the content displayed in the first target window includes: acquiring multiple video frame images from the source data of the live video; performing video prediction based on the multiple video frame images to generate image data of the predicted frame images; determining a second point of interest based on the image data of the predicted frame images, and using the second point of interest as the target point of interest.

[0090] Specifically, because it's difficult to accurately determine the image content of un-broadcast video frames in a live stream, video prediction is needed from multiple video frames to ensure the timeliness of generating recommendations. For example, if a popular athlete appears in a frame of a live stream, and the athlete's dribbling action appears in M ​​consecutive frames starting from that frame, then that frame and the subsequent M frames can be treated as an image sequence. Video prediction can then be performed using this sequence to generate image data for the predicted frame image. Based on this predicted frame image data, a second point of interest is determined. For instance, if the predicted frame image clearly shows the athlete's shoes, the information related to those shoes can be used as the target point of interest.

[0091] Here, video prediction can be performed in various ways. For example, a deep neural network can be used to calculate the motion information of each pixel in the current video frame image, thereby predicting each pixel in the future frame image and obtaining the predicted frame image.

[0092] In one embodiment of this disclosure, video prediction can be performed using a pre-trained adversarial network. Inputting multiple consecutive video frame images into the pre-trained generative adversarial network outputs predicted frame images. The aforementioned generative adversarial network can consist of a generator and two discriminators. The generator can generate future image frames based on the input video frame images, and the discriminators can determine the authenticity of the future image frames generated by the generator. Here, the future image frames generated by the generator can be input into a first discriminator so that the first discriminator can determine whether the future image frame is generated or real. Then, the consecutive video frames and the future image frames determined to be real by the first discriminator are input as a video frame sequence into a second discriminator so that the second discriminator can determine whether the input sequence contains generated image frames. Therefore, for the aforementioned generative adversarial network (GAN), if the first discriminator determines that the future image frame output by the generator is real, and the second discriminator determines that the sequence of the input continuous video frame images and the future frame images does not contain the generated image frame, that is, the second discriminator determines that all images in the input sequence are real, then the future image frame generated by the generator can be identified as the predicted frame image. The aforementioned GAN improves the accuracy of the output image by setting two discriminators: one to judge the authenticity of the future image frame generated by the generator itself, and the other to judge the image sequence containing the future image frame. Those skilled in the art will understand that the generator and discriminator are both neural networks; they only need to fit the corresponding generation and discrimination functions. For example, the discriminator can be composed of five convolutional layers plus an activation function.

[0093] In this embodiment, prediction frame images can be generated by performing video prediction on live video, and then target interest points can be determined in the prediction frame images, ensuring the timeliness of generating recommendation information for live video.

[0094] In one embodiment of this disclosure, for live video, step S8, the step of displaying a recommendation window, includes: obtaining the estimated appearance time of the predicted frame image; and displaying the recommendation window at the estimated appearance time. For example, if video prediction determines that the predicted frame image will appear 5 seconds after the current time of the live video, then 5 seconds after the current time, a recommendation window generated for the target point of interest in the predicted frame image is displayed. Here, the appearance time of the predicted frame image can be estimated based on parameters such as the current video frame image's position in the overall video and the video's refresh rate. In this embodiment, when it is determined that a target point of interest exists in the obtained predicted frame image, a corresponding recommendation window can be generated when the predicted frame image arrives, thereby enabling the head-mounted display device to display the corresponding recommendation window simultaneously with the predicted frame image, ensuring the timeliness of generating recommendation information for the live video. In one embodiment of this disclosure, when the content displayed in the first target window is a recorded video, the step of determining the target point of interest based on the source data of the content displayed in the first target window includes: obtaining a second target video frame image from the source data of the recorded video; determining the target point of interest based on the second target video frame image; wherein, displaying the recommendation window includes: obtaining the expected playback time point of the second target video frame image; and displaying the recommendation window at the expected playback time point.

[0095] In this embodiment, since the recorded video can accurately determine the image content of all video frames, the point of interest can be determined for the second target video frame image that appears in the recorded video. When the second target video frame image is played, the corresponding recommendation window is displayed, which can realize the real-time correspondence between video content and recommendation window.

[0096] Any of the information display methods for head-mounted display devices provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the information display methods for head-mounted display devices provided in this disclosure can be executed by a processor, such as by a processor executing any of the information display methods for head-mounted display devices mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated upon below.

[0097] Exemplary device

[0098] Figure 9 This is a structural block diagram of an information display device for a head-mounted display device according to an embodiment of the present disclosure. Figure 9 As shown, the information display device for a head-mounted display device according to an embodiment of this disclosure includes: a source data acquisition module 100, a function interface window generation module 200, a recommendation window generation module 300, and a display module 400.

[0099] The source data acquisition module 100 is used to acquire source data of multiple functional areas in the target resource page in response to receiving an access request from the head-mounted display device. The functional interface window generation module 200 is used to generate multiple functional interface windows based on the source data of the multiple functional areas. The recommendation window generation module 300 is used to determine a first target window among the multiple functional interface windows and acquire recommendation information associated with the content displayed in the first target window to generate a corresponding recommendation window. The display module 400 is used to determine the window to be displayed so that the head-mounted display device displays the window to be displayed, wherein the window to be displayed includes at least the recommendation window and the first target window.

[0100] In one embodiment of this disclosure, the display module 400 is further configured to adjust the window to be displayed based on the pose change information of the head-mounted display device, so that the head-mounted display device displays the adjusted window to be displayed.

[0101] In one embodiment of this disclosure, the functional interface window generation module 300 includes:

[0102] The template acquisition unit is used to acquire multiple target functional interface window templates from a pre-built functional interface window template library.

[0103] The function interface window generation unit is used to fill the source data of multiple function areas into multiple target function interface templates to generate multiple function interface windows.

[0104] In one embodiment of this disclosure, the functional interface window generation unit 300 is used to generate multiple window labels corresponding one-to-one with multiple functional interface windows based on source data of multiple functional areas; the functional interface window generation unit is also used to generate multiple sets of window display content corresponding one-to-one with multiple functional interface windows based on source data of multiple functional areas; the functional interface window generation unit is also used to fill multiple sets of window display content and multiple window labels into multiple target functional interface templates respectively to generate multiple functional interface windows.

[0105] In one embodiment of this disclosure, the information display device for a head-mounted display device further includes a human-computer interaction module. The human-computer interaction module, in response to receiving a user operation command for one of the currently displayed functional interface windows and recommendation windows, controls the window to perform corresponding actions according to the user operation command. The user operation commands include commands to zoom in, zoom out, push away, zoom in, drag, and close the window.

[0106] In one embodiment of this disclosure, the recommendation window generation module 300 includes:

[0107] The source data acquisition unit is used to acquire the source data of the content displayed in the first target window;

[0108] The target interest point determination unit is used to determine the target interest point based on the source data of the content displayed in the first target window;

[0109] The recommendation information generation unit is used to generate recommendation information based on target points of interest.

[0110] In one embodiment of this disclosure, the first target window displays a target video. The target interest point determination unit is used to obtain user comment data associated with the first target video frame image from the source data of the target video. The target interest point determination unit is further used to determine a first interest point associated with the first target video frame image based on the user comment data associated with the first target video frame image, and to use the first interest point as the target interest point.

[0111] In one embodiment of this disclosure, the target interest point determination unit is used to acquire all user comment data associated with the first target video frame image. The target interest point determination unit is further used to determine multiple interest points based on all user comment data. The target interest point determination unit is also used to perform a count of the number of people interested in the multiple interest points, and to select the interest point with the largest number of interested people as the first interest point.

[0112] In one embodiment of this disclosure, the target interest point determination unit is used to acquire target comment data associated with a first target video frame image. The target comment data is comment data from a target user. The target interest point determination unit is further used to determine a first interest point based on the target comment data.

[0113] In one embodiment of this disclosure, the first target window displays live video. The target interest point determination unit is used to acquire multiple video frame images from the source data of the live video. The target interest point determination unit is also used to perform video prediction based on the multiple video frame images, generating image data of predicted frame images. The target interest point determination unit is further used to determine a second interest point based on the image data of the predicted frame images, and use the second interest point as the target interest point.

[0114] In one embodiment of this disclosure, the display module 400 is used to obtain the expected occurrence time of the predicted frame image; and to display a recommendation window at the expected occurrence time.

[0115] In one embodiment of this disclosure, the first target window displays a recorded video. The target interest point determination unit is used to obtain a second target video frame image from the source data of the recorded video, and then determine target interest points based on the second target video frame image. The display module 400 is used to obtain the expected playback time point of the second target video frame image; and display a recommendation window at the expected playback time point.

[0116] It should be noted that the specific implementation of the information display method for a head-mounted display device in this disclosure is similar to the specific implementation of the information display device for a head-mounted display device in this disclosure. For details, please refer to the section on the information display method for a head-mounted display device. To reduce redundancy, further details will not be provided.

[0117] Exemplary head-mounted display device

[0118] Below, for reference Figure 10 To describe an electronic device according to embodiments of this disclosure. For example... Figure 10 As shown, the electronic device includes one or more processors 10 and memory 20.

[0119] The processor 10 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0120] The memory 20 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 10 may execute the program instructions to implement the information display methods for head-mounted display devices and / or other desired functions described in the various embodiments of this disclosure above. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0121] In one example, the electronic device may also include an input device 30 and an output device 40, these components being interconnected via a bus system and / or other forms of connection mechanism (not shown). The input device 30 may be, for example, a keyboard, a mouse, etc. The output device 40 may include, for example, a display, speakers, a printer, and a communication network and its connected remote output devices, etc.

[0122] Of course, for the sake of simplicity, Figure 10 Only some of the components of the head-mounted display device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0123] Exemplary computer-readable storage media

[0124] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0125] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0126] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0127] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0128] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0129] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0130] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0131] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. An information display method for a head-mounted display device, comprising: In response to receiving an access request for a target resource page from a head-mounted display device, source data of multiple functional areas in the target resource page is obtained; The target resource page is split into functional areas, and the source data of each functional area is filled into a pre-built window to generate multiple functional interface windows. The data included in each functional interface window are source data from different functional dimensions corresponding to the same main event in the same resource page. A first target window is determined from the plurality of functional interface windows, and recommendation information associated with the content displayed in the first target window is obtained to generate a corresponding recommendation window; A window to be displayed is determined so that the head-mounted display device displays the window to be displayed, wherein the window to be displayed includes at least the recommended window and the first target window, and the window to be displayed can be determined from each of the functional interface windows by selective display.

2. The method of claim 1, wherein, After determining the window to be displayed so that the head-mounted display device displays the window to be displayed, the method further includes: Based on the pose change information of the head-mounted display device, the window to be displayed is adjusted so that the head-mounted display device displays the adjusted window.

3. The method of claim 1, wherein, The process involves filling the source data of each functional area from the source data of the multiple functional areas obtained from the splitting into a pre-built window, generating multiple functional interface windows, including: Obtain multiple target functional interface templates from a pre-built functional interface window template library; The source data of the multiple functional areas are respectively filled into the multiple target functional interface templates to generate the multiple functional interface windows.

4. The method of claim 3, wherein, The step of filling the source data of the multiple functional areas into the multiple target functional interface templates to generate the multiple functional interface windows includes: Based on the source data of the multiple functional areas, generate multiple window labels that correspond one-to-one with the multiple functional interface windows; Based on the source data of the multiple functional areas, generate multiple sets of window display content that correspond one-to-one with the multiple functional interface windows; The content displayed in the multiple sets of windows and the multiple window labels are respectively filled into the multiple target function interface templates to generate the multiple function interface windows.

5. The method of claim 1, wherein, After determining the window to be displayed so that the head-mounted display device displays the window to be displayed, the method further includes: For one of the currently displayed functional interface window and the recommendation window, in response to receiving a user operation command for that window, control that window to perform the corresponding action according to the user operation command, wherein the user operation command includes a window zoom-in command, a window zoom-out command, a window push-away command, a window zoom-in command, a window drag command, and a window close command.

6. The method of claim 1, wherein, The step of obtaining recommendation information associated with the content displayed in the first target window includes: Obtain the source data of the content displayed in the first target window; Based on the source data of the content displayed in the first target window, the target point of interest is determined; The recommendation information is generated based on the target interest points.

7. The method of claim 6, wherein, The first target window displays the target video; The step of determining the target point of interest based on the source data of the content displayed in the first target window includes: Obtain user comment data associated with the first target video frame image from the source data of the target video; Based on the user comment data associated with the first target video frame image, a first point of interest associated with the first target video frame image is determined, and the first point of interest is used as the target point of interest.

8. The method of claim 7, wherein, The step of determining the first point of interest associated with the first target video frame image based on the user comment data associated with the first target video frame image includes: Obtain all user comment data associated with the first target video frame image; Based on all the user comment data, multiple points of interest were identified; The number of people interested in the multiple points of interest is counted, and the point of interest with the largest number of interested people is selected as the first point of interest.

9. The method of claim 7, wherein, The step of determining the first point of interest associated with the first target video frame image based on the user comment data associated with the first target video frame image includes: Obtain target comment data associated with the first target video frame image, wherein the target comment data is the comment data of the target user; Based on the target comment data, the first point of interest is determined.

10. The method of claim 6, wherein, The first target window displays live video content; The step of determining the target point of interest based on the source data of the content displayed in the first target window includes: Obtain multiple video frame images from the source data of the live video; Based on the multiple video frame images, video prediction is performed to generate image data of the predicted frame images; Based on the image data of the predicted frame image, a second point of interest is determined, and the second point of interest is used as the target point of interest.

11. The method of claim 10, wherein, The step of determining the window to be displayed, so that the head-mounted display device displays the window to be displayed, includes: Obtain the estimated occurrence time of the predicted frame image; The recommendation window will be displayed at the expected time.

12. The method of claim 6, wherein, The first target window displays a recorded video. The step of determining the target point of interest based on the source data of the content displayed in the first target window includes: obtaining a second target video frame image from the source data of the recorded video; and determining the target point of interest based on the second target video frame image. The step of displaying the recommendation window includes: obtaining the expected playback time of the second target video frame image; and displaying the recommendation window at the expected playback time.

13. An information display device for a head-mounted display device, comprising: The source data acquisition module is used to acquire source data of multiple functional areas in the target resource page in response to receiving an access request for the target resource page of the head-mounted display device; The functional interface window generation module is used to split the functional areas of the target resource page, fill the source data of each functional area in the source data of the multiple functional areas obtained by splitting into a pre-built window, and generate multiple functional interface windows. The data included in each functional interface window are source data from different functional dimensions corresponding to the same main event in the same resource page. The recommendation window generation module is used to determine a first target window among the multiple functional interface windows and obtain recommendation information associated with the content displayed in the first target window in order to generate a corresponding recommendation window; A display module is used to determine a window to be displayed so that the head-mounted display device displays the window to be displayed, wherein the window to be displayed includes at least the recommended window and the first target window, and the window to be displayed is determined from each of the functional interface windows by selective display.

14. A head-mounted display device, the head-mounted display device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the information display method for a head-mounted display device according to any one of claims 1-12.

15. A computer-readable storage medium storing a computer program for performing the information display method for a head-mounted display device according to any one of claims 1-12.

Images