Image display method, interactive device and computer readable storage medium

Abstract

The application discloses an image display method, an interactive device and a computer readable storage medium. The method comprises the following steps: receiving a first position of a gaze point on a screen of a head-mounted display device; calculating a second position of the gaze point in an image frame according to the first position and a size of the image frame; rendering the image frame according to the second position; and sending the rendered image frame to the screen of the head-mounted display device for display. The application renders the image by the interactive device, sends the rendered image frame to the head-mounted display device for display, and does not need to render the image by the head-mounted display device, thereby reducing the delay.

Description

Technical Field

[0001] This disclosure relates to the field of image display technology, and more specifically, to an image display method, an interactive device, and a computer-readable storage medium. Background Technology

[0002] With the development of Augmented Reality (AR) technology, major manufacturers have launched more and more AR products and applications. Many manufacturers support streaming functionality, which allows content from a mobile phone to be projected onto AR glasses via Wi-Fi or USB connection.

[0003] In streaming scenarios, real-time display is crucial, and users expect the lowest possible latency during operation. Current technologies involve sending image frames from the mobile phone to the AR device, which then renders and displays them, resulting in significant latency. Summary of the Invention

[0004] One object of this disclosure is to provide a new technical solution for image display.

[0005] According to a first aspect of this disclosure, an image display method is provided, the method comprising:

[0006] The gaze point is received at its first position on the screen of the head-mounted display device;

[0007] Based on the first position and the size of the image frame, the second position of the gaze point in the image frame is calculated;

[0008] The image frame is rendered according to the second position;

[0009] The rendered image frame is sent to the screen of the head-mounted display device for display.

[0010] Optionally, before rendering the image frame according to the second position, the method further includes:

[0011] Obtain the positions of the gaze points in the first N frames of images, where N is a positive integer; and

[0012] The third position is calculated based on the position of the gaze point in the previous N frames and the second position;

[0013] The step of rendering the image frame according to the second position includes:

[0014] The image frame is rendered based on the third position.

[0015] Optionally, calculating the third position based on the position of the gaze point in the previous N frames and the second position includes:

[0016] Based on the position of the gaze point in the first N frames of the image and the second position, a fitting function is obtained, wherein the fitting function represents the relationship between the position of the gaze point in the image frame and the change over time;

[0017] The delay between the interactive device and the head-mounted display device is obtained; and

[0018] The third position is calculated based on the delay and the fitting function.

[0019] Optionally, rendering the image frame based on the second position includes:

[0020] Determine the first region and the second region in the image frame based on the second position; and

[0021] Determine whether to send the image of the second region based on preset rules;

[0022] The step of sending the rendered image frame to the head-mounted display device for display includes:

[0023] If it is determined that the image of the second region will be transmitted, the encoded image frame of the image of the second region will be transmitted to the head-mounted display device for display after encoding the image frame; and

[0024] If it is determined that the image of the second region will not be sent, the encoded image frame of the image of the first region will be sent to the head-mounted display device for display after encoding the image frame.

[0025] Optionally, determining whether to send the image of the second region according to a preset rule includes:

[0026] The color of each pixel in the second region is compared with the color of the corresponding pixel in the previous frame to obtain the number of pixels with the same color;

[0027] Determine whether the ratio between the number of pixels with the same color and the total number of pixels in the second region exceeds a first threshold.

[0028] If the limit is exceeded, determine that the image of the second region will not be sent;

[0029] If the value does not exceed the specified limit, then confirm the sending of the image for the second region.

[0030] Optionally, before determining whether to send the image of the second region according to a preset rule, the method further includes:

[0031] The image of the second region is blurred.

[0032] Optionally, the method further includes:

[0033] Acquire the first position of the gaze point on the screen of the head-mounted display device;

[0034] Send the first location to the interactive device;

[0035] Receive an image frame rendered by the interactive device, the image frame being rendered by the interactive device according to the first position;

[0036] Displays the rendered image frame.

[0037] Optionally, the display of the rendered image frame includes:

[0038] Determine whether the rendered image frame is complete;

[0039] If the rendered image frame is determined to be incomplete, the third region corresponding to the image frame in the previous image frame is determined.

[0040] Display the image frame and a fourth region in the previous frame image, wherein the fourth region is any region in the previous frame image other than the third region; and

[0041] If the rendered image frame is determined to be complete, the rendered image frame is displayed.

[0042] According to a second aspect of this disclosure, an interactive device is provided, the device comprising an interactive apparatus and a head-mounted display device;

[0043] The interactive device is configured to receive a first position of the gaze point on the screen of the head-mounted display device; calculate a second position of the gaze point in the image frame based on the first position and the size of the image frame; render the image frame based on the second position; and send the rendered image frame to the screen of the head-mounted display device for display.

[0044] According to a third aspect of this disclosure, a computer-readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps of the image display method as described in the first aspect of the invention.

[0045] According to embodiments of this disclosure, the present invention allows the interactive device to render the image and then sends the rendered image frame to the head-mounted display device for display, eliminating the need for the head-mounted display device to render the image, thus reducing latency. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the present disclosure and, together with their description, serve to explain the principles of the present disclosure.

[0047] Figure 1 This is a flowchart of the image display method in an embodiment of this application.

[0048] Figure 2 This is a schematic diagram of the division of the first and second regions in an embodiment of this application.

[0049] Figure 3 This is a schematic diagram illustrating the further division of the second region in an embodiment of this application.

[0050] Figure 4 This is a schematic diagram of the interaction process between the mobile phone and the AR glasses in an embodiment of this application.

[0051] Figure 5 This is a schematic diagram of the interactive device in an embodiment of this application. Detailed Implementation

[0052] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0053] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0054] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0055] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0056] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0057] like Figure 1 As shown in the figure, an embodiment of the present invention introduces an image display method, which is executed by an interactive device and includes steps S1100-S1400.

[0058] S1100: Receives the first position of the gaze point on the screen of the head-mounted display device.

[0059] When using a head-mounted display device, users can view the content displayed on the screen of the head-mounted display device. Head-mounted display devices can be AR glasses, MR (Mediated Reality) glasses, VR (Virtual Reality) glasses, and other similar devices.

[0060] A fixation point is a point on the screen where a user's gaze falls. When viewing content, users tend to focus on the area surrounding the fixation point. For example, if the fixation point is on the left side of the screen, the user will focus on the content displayed on the left side.

[0061] Head-mounted displays contain sensors that can calculate the initial position of the gaze point on the screen based on the data detected by the sensors. For example, head-mounted displays can use eye tracing algorithms to calculate the initial position of the gaze point on the screen.

[0062] After calculating the initial position of the gaze point on the screen, the head-mounted display device sends this initial position to the interactive device. The interactive device receives the initial position of the gaze point on the head-mounted display device's screen. The interactive device can be a mobile phone, computer, tablet, etc. A communication connection is established between the head-mounted display device and the interactive device, such as a Wi-Fi connection or a USB connection.

[0063] The first position can be a relative position. For example, if the screen resolution is 1920*1080, the first position is the center of the screen, with coordinates (960, 540). After normalization, the coordinates of the first position are (0.5, 0.5).

[0064] S1200: Calculate the second position of the gaze point in the image frame based on the first position and the size of the image frame.

[0065] An image frame is an image frame that will be displayed on a head-mounted display device. The position of the gaze point on the screen is the first position, and its corresponding position on the image frame is the second position. For example, if the first position is the center of the screen, then the second position is the center of the image frame.

[0066] Because the resolution of the image frame may differ from the screen resolution, the image frame needs to be scaled when displayed on the screen. In this case, a second position is calculated based on the size of the image frame and the first position. The calculation of the second position is performed by the interactive device.

[0067] For example, if the coordinates of the first position are (0.5, 0.5) and the size of the image frame is 2048*1536, then the coordinates of the second position are (1024, 768).

[0068] S1300: Render the image frame according to the second position.

[0069] The image rendering process is performed by the interactive device, which renders the image frame based on the second position. The area around the second position is the area of ​​primary focus for the user, and during rendering, this area can be rendered in high definition to improve clarity. For areas that are not of primary focus for the user, the clarity can be appropriately reduced.

[0070] S1400: The rendered image frame is sent to the screen of the head-mounted display device for display.

[0071] After the interactive device finishes rendering the image frame, it sends the rendered image frame to the head-mounted display device. The head-mounted display device then displays the rendered image frame on its screen.

[0072] For example, the interactive device is a mobile phone, and the head-mounted display device is AR glasses. The mobile phone and AR glasses are connected via Wi-Fi. After the mobile phone renders the image frame, it sends it to the AR glasses, and the AR glasses display the image frame.

[0073] In this embodiment, the interactive device renders the image and sends the rendered image frame to the head-mounted display device for display, eliminating the need for the head-mounted display device to render the image, thus reducing latency.

[0074] In one embodiment, prior to step S1300, the method further includes: obtaining the position of the gaze point in the previous N frames of images, where N is a positive integer; and calculating a third position based on the position of the gaze point in the previous N frames of images and the second position. Step S1300 includes: rendering the image frame based on the third position.

[0075] The previous N frames are the image frames displayed by the head-mounted display device before the current image frame is displayed. The head-mounted display device displays the current image frame after displaying the previous N frames. N can be 3 or other values. The gaze point has a position on each image frame.

[0076] During the display of multiple frames of images on a head-mounted display device, the position of the gaze point within the image frames may change. A third position is calculated based on the gaze point's position in the previous N frames and its second position. This third position represents the gaze point's position within the current image frame at the time of display. After calculating the third position, the image frame is rendered according to this third position.

[0077] In one example, the head-mounted display device sequentially displays image frames including image 1, image 2, image 3, image 4, and image 5. Currently, the head-mounted display device is showing image 4, with the preceding three frames being images 1, 2, and 3, respectively. Image 5 is displayed after image 4. The user's gaze point on the screen when viewing image 4 is the first position. A second position is calculated based on the first position and the size of image 5. A third position is calculated based on the second position and the positions of the gaze point in image 1, image 2, and image 3. Image 5 is then rendered based on the third position.

[0078] In one embodiment, calculating the third position based on the position of the gaze point in the previous N frames and the second position includes: obtaining a fitting function based on the position of the gaze point in the previous N frames and the second position, wherein the fitting function represents the relationship between the position of the gaze point in the image frame and time; obtaining the delay between the interactive device and the head-mounted display device; and calculating the third position based on the delay and the fitting function.

[0079] The head-mounted display device sequentially displays each frame of an image, with each frame display corresponding to a specific time interval. A fitting function is obtained by fitting the display time of each frame to the position of the fixation point within that frame. This fitting function represents the relationship between the position of the fixation point within the image frame and the change over time.

[0080] The transmission of an image frame from the interactive device to the head-mounted display device takes a certain amount of time, resulting in a delay between the two devices. Based on this delay and a fitted function, a third position is calculated. This third position represents the time when the user views the image frame after it has been transmitted to the head-mounted display device.

[0081] This embodiment obtains the relationship between the position of the gaze point in the image frame and the time by fitting the position of the gaze point in multiple image frames, and calculates the third position based on the delay between the interactive device and the head-mounted display device, which can accurately predict the position of the gaze point when the head-mounted display device displays the image frame.

[0082] In one implementation, step 1300 includes S1310-S1320.

[0083] S1310: Determine the first region and the second region in the image frame based on the second position.

[0084] The first region is the area surrounding the second position in the image frame; this is the area that users focus on most. The second region is the rest of the image frame excluding the first region; users pay relatively less attention to the second region. For example... Figure 2As shown, the first region can be a rectangular area centered at the second position, and the size of the rectangle can be adjusted. The second region is any other area in the image frame besides this rectangular area.

[0085] For example, if the resolution of an image frame is 1920*1080, and the second position is the center point of that image frame with coordinates (960, 540), then the first region is a rectangular area centered at the second position. This rectangular area has a width of 500 and a height of 200. Therefore, the coordinates of the four vertices of the first region are (710, 440), (1210, 440), (1210, 640), and (710, 640).

[0086] S1320: and determine whether to send the image of the second region according to preset rules.

[0087] The first region is the area of ​​primary user focus, and all images of the first region in each frame need to be sent to the head-mounted display device. Users pay relatively less attention to the second region, so they can choose not to send images of the second region. The system determines whether to send images of the second region based on preset rules. If the rules are met, then the second region's image is not sent; otherwise, the second region's image is sent.

[0088] Step S1400 includes S1410-S1420.

[0089] S1410: If it is determined that the image of the second region is to be sent, the encoded image frame of the image of the second region is encoded and then sent to the head-mounted display device for display.

[0090] If it is determined that the image of the second region needs to be transmitted, the image frame is encoded, including encoding the image of the first region and encoding the image of the second region. After the image frame is encoded, the encoded image frame is sent to the head-mounted display device, and both the encoded images of the first and second regions are sent to the head-mounted display device. The images of the first and second regions are displayed on the head-mounted display device.

[0091] S1420: If it is determined that the image of the second region will not be sent, the encoded image frame of the image of the first region will be sent to the head-mounted display device for display after encoding the image frame of the first region.

[0092] If it is determined that the image of the second region will not be sent, only the image of the first region is encoded when encoding the image frame. The encoded image of the first region is then sent to the head-mounted display device. The head-mounted display device displays the image of the first region, and simultaneously displays the image of the corresponding second region from the previous frame.

[0093] This embodiment divides the image frame into a first region and a second region, and determines whether the image of the second region needs to be sent according to a preset rule, which can reduce the number of times the image of the second region is sent and reduce latency.

[0094] In one embodiment, step S1320 includes S1321-S1324.

[0095] S1321: Compare the color of each pixel in the second region with the color of the corresponding pixel in the previous frame to obtain the number of pixels with the same color.

[0096] Based on the position of the pixel in the second region, determine the corresponding pixel from the previous frame. Compare the color of the pixel in the second region with the color of the corresponding pixel in the previous frame, and count the number of pixels with the same color.

[0097] S1322: Determine whether the ratio between the number of pixels with the same color and the total number of pixels in the second region exceeds a first threshold.

[0098] The degree of difference between the second region and the previous frame can be represented by the ratio between the number of pixels of the same color and the total number of pixels in the second region. A larger ratio indicates a smaller difference between the second region and the previous frame, while a smaller ratio indicates a larger difference. The first threshold value ranges from 0 to 1 and can be adjusted according to actual needs.

[0099] S1323: If the limit is exceeded, determine not to send the image of the second region.

[0100] If the ratio between the number of pixels of the same color and the total number of pixels in the second region exceeds the first threshold, it indicates that the image in the second region is less different from the previous frame, and the image in the second region does not need to be sent.

[0101] S1324: If it does not exceed the specified value, confirm that the image of the second region is to be sent.

[0102] If the ratio between the number of pixels of the same color and the total number of pixels in the second region does not exceed the first threshold, it indicates that the image of the second region differs significantly from the previous frame, and the image of the second region needs to be sent.

[0103] This embodiment determines whether to send the image of the second region based on the difference between the second region and the previous frame. When the difference is small, the image of the second region is not sent, reducing the number of times the image of the second region is sent. The second image does not need to be encoded, thus reducing encoding time and image transmission time.

[0104] In one embodiment, prior to step 1320, the image of the second region is further blurred.

[0105] Blurring the image in the second region can reduce its resolution. For example, the resolution of the second region can be set to 50% of the original resolution. After blurring, the image quality of the second region decreases, which can reduce the image encoding time.

[0106] The second region can be further subdivided. For example... Figure 3 As shown, the image frame is divided into Layer 1, Layer 2, and Layer 3. Layer 1 is a rectangular area centered on the viewpoint, and is the first region. Layers 2 and 3 constitute the second region; Layer 2 is the outer region of Layer 1, and Layer 3 is the outer region of Layer 2. The user's attention is highest on Layer 1, lowest on Layer 3, and between Layer 1 and Layer 3 in terms of attention on Layer 2.

[0107] For layer 2, set its resolution to 50% of the original resolution. Compare the colors of pixels in layer 2 with the corresponding pixels in the previous frame, and determine if the ratio between the number of pixels with the same color and the total number of pixels in layer 2 exceeds a second threshold, which can be set to 50%. If it exceeds the threshold, it means the difference between layer 2 and the previous frame is less than 50%, and layer 2 does not need to be sent to the head-mounted display device; if it does not exceed the threshold, it means the difference between layer 2 and the previous frame exceeds 50%, and layer 2 needs to be sent to the head-mounted display device.

[0108] For layer 3, set its resolution to 25% of the original resolution. Compare the colors of pixels in layer 3 with the corresponding pixels in the previous frame, and determine if the ratio between the number of pixels with the same color and the total number of pixels in layer 3 exceeds a third threshold. Since the user's attention in layer 3 is lower than in layer 2, the third threshold can be set lower than the second threshold, for example, to 25%. If it exceeds the threshold, it means the difference between layer 2 and the previous frame is no more than 75%, so layer 3 does not need to be sent to the head-mounted display device; if it does not exceed the threshold, it means the difference between layer 3 and the previous frame is more than 75%, so layer 3 needs to be sent to the head-mounted display device.

[0109] In this embodiment, the image display method is performed by a head-mounted display device, and the method includes steps S2100-S2400.

[0110] S2100: Obtain the first position of the gaze point on the screen of the head-mounted display device.

[0111] The head-mounted display device is equipped with a variety of sensors. When the user uses the head-mounted display device, the head-mounted display device can calculate the first position of the gaze point on the head-mounted display device based on the data detected by the sensors and the preset algorithm.

[0112] S2200: Send the first location to the interactive device.

[0113] Head-mounted displays and interactive devices can communicate with each other. For example, they can connect wirelessly via Wi-Fi or via a wired connection using a USB cable.

[0114] After calculating the first position, the head-mounted display device sends the first position to the interactive device.

[0115] S2300: Receive an image frame rendered by the interactive device, the image frame being rendered by the interactive device according to the first position.

[0116] After receiving the first position from the head-mounted display device, the interactive device renders the image frame based on the first position. The interactive device then sends the rendered image frame to the head-mounted display device.

[0117] S2400: Displays the rendered image frame.

[0118] In one example, such as Figure 4 As shown, the head-mounted display device is AR glasses, and the interaction device is a mobile phone. When the user uses the AR glasses, the AR glasses determine the first position of the gaze point on the screen of the AR glasses based on the detection data of the sensors and the preset algorithm. The AR glasses send the first position to the mobile phone, which renders the image frame based on the first position, encodes the image frame, and sends it back to the AR glasses. The AR glasses then display the rendered image frame.

[0119] In one embodiment, step S2400 includes S2410-S2440.

[0120] S2410: Determine whether the rendered image frame is complete.

[0121] When rendering an image frame, the interactive device may render only a portion of the image frame. In this case, the image frame sent by the interactive device to the head-mounted display device only includes the aforementioned portion.

[0122] After receiving a rendered image frame, the head-mounted display device determines whether the rendered image frame is complete. For example, the head-mounted display device can determine this based on the size of the rendered image frame.

[0123] S2420: If it is determined that the rendered image frame is incomplete, determine the third region corresponding to the rendered image frame in the previous image frame.

[0124] The third region is the region in the previous frame that corresponds to the rendered image. For example, if the image frame before rendering includes a first region and a second region, and the interactive device only renders the first region, then the rendered image frame sent by the interactive device to the head-mounted display device only contains the first region. Therefore, the third region is the region in the previous frame that corresponds to the first region. For instance, if the first region is located at the center of the image frame before rendering, then the third region is also located at the center of the previous frame.

[0125] S2430: Display the rendered image frame and the fourth region in the previous frame image, wherein the fourth region is the other region in the previous frame image besides the third region.

[0126] S2440: And if it is determined that the rendered image frame is complete, display the rendered image frame.

[0127] If the rendered image frame is incomplete, displaying only the rendered image frame will result in some areas of the head-mounted display screen not showing the image. In this case, the head-mounted display selects a fourth region from the previous frame for display, and simultaneously displays both the rendered image frame and the fourth region from the previous frame.

[0128] If the rendered image frame is complete, the head-mounted display device directly displays the rendered image frame.

[0129] like Figure 5 As shown in the figure, an embodiment of the present invention introduces an interactive device, which includes an interactive device and a head-mounted display device.

[0130] The interactive device is configured to receive a first position of the gaze point on the screen of the head-mounted display device; calculate a second position of the gaze point in the image frame based on the first position and the size of the image frame; render the image frame based on the second position; and send the rendered image frame to the screen of the head-mounted display device for display.

[0131] This invention allows the interactive device to render the image and then sends the rendered image frame to the head-mounted display device for display, eliminating the need for the head-mounted display device to render the image, thus reducing latency.

[0132] In one embodiment, the interactive device is further configured to: obtain the position of the gaze point in the previous N frames of images, where N is a positive integer, before rendering the image frame according to the second position; calculate a third position based on the position of the gaze point in the previous N frames of images and the second position; and render the image frame according to the third position.

[0133] In one embodiment, the interactive device is further configured to obtain a fitting function based on the position of the gaze point in the previous N frames of images and the second position, wherein the fitting function represents the relationship between the position of the gaze point in the image frame and the change over time; obtain the delay between the interactive device and the head-mounted display device; and calculate the third position based on the delay and the fitting function.

[0134] In one embodiment, the interactive device is further configured to: determine a first region and a second region in the image frame based on the second position; and determine whether to send an image of the second region according to a preset rule; if it is determined that an image of the second region should be sent, the encoded image frame of the image of the second region is encoded and then sent to the head-mounted display device for display; and if it is determined that an image of the second region should not be sent, the encoded image frame of the image of the first region is encoded and then sent to the head-mounted display device for display.

[0135] In one embodiment, the interactive device is further configured to compare the color of each pixel in the second region with the color of the corresponding pixel in the previous frame to obtain the number of pixels with the same color; determine whether the ratio between the number of pixels with the same color and the total number of pixels in the second region exceeds a first threshold; if it exceeds the threshold, determine not to send the image of the second region; if it does not exceed the threshold, determine to send the image of the second region.

[0136] In one embodiment, the interactive device is further configured to blur the image of the second region before determining whether to send the image of the second region according to a preset rule.

[0137] In one embodiment, the head-mounted display device is configured to acquire a first position of the gaze point on the screen of the head-mounted display device; send the first position to an interactive device; receive an image frame rendered by the interactive device based on the first position; and display the rendered image frame.

[0138] In one embodiment, the head-mounted display device is further configured to: determine whether the rendered image frame is complete; if the rendered image frame is incomplete, determine a third region corresponding to the image frame in the previous frame; display the image frame and a fourth region in the previous frame, wherein the fourth region is any region in the previous frame other than the third region; and if the rendered image frame is complete, display the rendered image frame.

[0139] This embodiment introduces a computer-readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the image display method as described in any embodiment of the present invention.

[0140] This disclosure may also include a computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions, i.e., executable instructions, on which a processor is loaded to implement various aspects of this disclosure.

[0141] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0142] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0143] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0144] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, computing devices, and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0145] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0146] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of computing devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based computing device that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are equivalent.

[0148] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An image display method, characterized in that, The method includes: The gaze point is received at its first position on the screen of the head-mounted display device; Based on the first position and the size of the image frame, the second position of the gaze point in the image frame is calculated; Rendering the image frame according to the second position includes: determining a first region and a second region in the image frame according to the second position; and determining whether to send the image of the second region according to a preset rule, including: comparing the color of each pixel in the second region with the color of the corresponding pixel in the previous frame to obtain the number of pixels with the same color; determining whether the ratio between the number of pixels with the same color and the total number of pixels in the second region exceeds a first threshold; if it exceeds, determining not to send the image of the second region; if it does not exceed, determining to send the image of the determined second region. Sending the rendered image frame to the screen of the head-mounted display device for display includes: if it is determined that the image of the second region will be sent, the encoded image frame of the image of the second region will be encoded and then sent to the head-mounted display device for display; and if it is determined that the image of the second region will not be sent, the encoded image frame of the image of the first region will be encoded and then sent to the head-mounted display device for display.

2. The method according to claim 1, characterized in that, Before rendering the image frame according to the second position, the method further includes: Obtain the positions of the gaze points in the first N frames of images, where N is a positive integer; and The third position is calculated based on the position of the gaze point in the previous N frames and the second position; The step of rendering the image frame according to the second position includes: The image frame is rendered based on the third position.

3. The method according to claim 2, characterized in that, The step of calculating the third position based on the position of the gaze point in the previous N frames and the second position includes: Based on the position of the gaze point in the first N frames of the image and the second position, a fitting function is obtained, wherein the fitting function represents the relationship between the position of the gaze point in the image frame and the change over time; The delay between the interactive device and the head-mounted display device is obtained; and The third position is calculated based on the delay and the fitting function.

4. The method according to claim 1, characterized in that, Before determining whether to send the image of the second region according to a preset rule, the method further includes: The image of the second region is blurred.

5. The method according to any one of claims 1-4, characterized in that, Acquire the first position of the gaze point on the screen of the head-mounted display device; Send the first location to the interactive device; Receive an image frame rendered by the interactive device, the image frame being rendered by the interactive device according to the first position; Displays the rendered image frame.

6. The method according to claim 5, characterized in that, The rendered image frame includes: Determine whether the rendered image frame is complete; If it is determined that the rendered image frame is incomplete, a third region corresponding to the rendered image frame in the previous frame is determined, and the third region is the region in the previous frame that corresponds to the rendered image. Display the rendered image frame and the fourth region in the previous frame image, wherein the fourth region is any region in the previous frame image other than the third region; and If the rendered image frame is determined to be complete, the rendered image frame is displayed.

7. An interactive device, characterized in that, The device includes an interactive device and a head-mounted display device; The interactive device is used to receive a gaze point at a first position on the screen of the head-mounted display device; Based on the first position and the size of the image frame, the second position of the gaze point in the image frame is calculated; Rendering the image frame according to the second position includes: determining a first region and a second region in the image frame according to the second position; and determining whether to send the image of the second region according to a preset rule, including: comparing the color of each pixel in the second region with the color of the corresponding pixel in the previous frame to obtain the number of pixels with the same color; determining whether the ratio between the number of pixels with the same color and the total number of pixels in the second region exceeds a first threshold; if it exceeds, determining not to send the image of the second region; if it does not exceed, determining to send the image of the determined second region; and sending the rendered image frame to the screen of the head-mounted display device for display, including: if it is determined to send the image of the second region, encoding the image frame of the image of the second region and then sending the encoded image frame to the head-mounted display device for display; and if it is determined not to send the image of the second region, encoding the image frame of the image of the first region and then sending the encoded image frame to the head-mounted display device for display.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the image display method as described in any one of claims 1-6.

Images