Image display method and apparatus, and head-mounted device and medium

Abstract

The present invention relates to the technical field of image processing. Disclosed are an image display method and apparatus, and a head-mounted device and a medium. The method comprises: acquiring original image data, and first pose data of a head-mounted device, wherein the first pose data is pose data of the head-mounted device at the moment when the original image data is acquired; on the basis of the original image data, generating images to be displayed; selecting a first target image from among the images to be displayed, wherein the first target image is an image having the same screen size as the head-mounted device within a visual range, and the size of the first target image is smaller than the sizes of the images to be displayed; displaying the first target image and acquiring second pose data, wherein the second pose data is pose data of the head-mounted device at the moment when the first target image is displayed; on the basis of the first pose data, the second pose data and the first target image, selecting a second target image from among the images to be displayed; and displaying the second target image. The method provides a new intermediate-frame generation technique that can replace an ATW function.

Description

Image display method, device, head-mounted device, and medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 16, 2023, with application number 202311035122.X and invention name “Image display method, device, head-mounted device and medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention relates to the field of image processing technology, and more specifically, to an image display method, an image display device, a head-mounted device, and a computer-readable storage medium. Background Art

[0003] With the development of science and technology and economy, head-mounted display devices (such as XR head-mounted display devices) have been increasingly widely used.

[0004] At present, the main function of the Graphics Processing Unit (GPU) on a head-mounted display device is to implement image rendering and asynchronous timewarp (ATW). ATW is a technology for generating intermediate frames. For example, for a game scene, when the game cannot maintain a sufficient frame rate, it can generate intermediate frames, thereby effectively reducing the jitter of the game screen. When the GPU implements ATW, it is generally necessary to synchronize the image output by the GPU with the screen refresh. In order to achieve this synchronization, it is necessary to ensure that the thread implementing ATW in the GPU is not blocked in the GPU. Therefore, the ATW thread is usually given the highest thread priority. In addition, the ATW thread is usually deployed on a high-performance large core in the GPU for execution. This increases the power consumption and resource occupation of the GPU.

[0005] In summary, it is urgent to propose an image display method that can replace the ATW function.

[0006] Summary of the Invention

[0007] The main purpose of the present invention is to propose a new technical solution for image display.

[0008] According to a first aspect of the present invention, there is provided an image display method, the method comprising:

[0009] Acquire original image data and first pose data of a head-mounted device, where the first pose data is the pose data of the head-mounted device at the moment of acquiring the original image data;

[0010] generating an image to be displayed according to the original image data;

[0011] Selecting a first target image from the images to be displayed, where the first target image is an image within a visible range and has the same size as the screen of the head mounted device, and the size of the first target image is smaller than the size of the image to be displayed;

[0012] Displaying a first target image and acquiring second posture data, where the second posture data is posture data of the head mounted device at the moment of displaying the first target image;

[0013] Selecting a second target image from the image to be displayed according to the first posture data, the second posture data, and the first target image;

[0014] The second target image is displayed.

[0015] Optionally, selecting a second target image from the image to be displayed according to the first pose data, the second pose data, and the first target image includes:

[0016] Determining, based on the first posture data and the second posture data, a posture change amount of the head mounted device at the display time compared to the acquisition time;

[0017] Determining an image selection parameter according to the pose change, the image selection parameter including at least one of a scaling factor and an offset relative to the first target image;

[0018] A second target image is selected from the images to be displayed according to the image selection parameter and the first target image.

[0019] Optionally, the second target image is an RGB image, and displaying the second target image includes:

[0020] extracting a red pixel image, a green pixel image, and a blue pixel image from the second target image;

[0021] performing dispersion correction processing on the red pixel image, the green pixel image, and the blue pixel image respectively according to the dispersion parameter of the lens of the head mounted device;

[0022] fusing the red pixel image, the green pixel image, and the blue pixel image after the dispersion correction process to obtain a second target image after dispersion correction;

[0023] The second target image after the dispersion correction is displayed.

[0024] Optionally, selecting a first target image from the images to be displayed includes:

[0025] Obtaining an eye movement heat map of a wearer wearing the head mounted device;

[0026] A first target image is determined according to the human eye movement heat map and the image to be displayed.

[0027] Optionally, determining the image selection parameters according to the posture change includes:

[0028] Determining image selection parameters according to the posture change amount and preset mapping data;

[0029] The preset mapping data is data reflecting the corresponding relationship between the posture change amount and the image selection parameters.

[0030] Optionally, selecting a first target image from the images to be displayed includes:

[0031] Obtaining the screen size of the head mounted device;

[0032] A first target image is selected from the images to be displayed according to the size.

[0033] Optionally, the method is executed by a display processor.

[0034] According to a second aspect of the present invention, there is provided an image display device, the device comprising:

[0035] A first acquisition module is configured to acquire original image data and first pose data of a head-mounted device, wherein the first pose data is the pose data of the head-mounted device at the moment of acquiring the original image data;

[0036] A generating module, configured to generate an image to be displayed based on the original image data;

[0037] A first selection module is configured to select a first target image from the images to be displayed, wherein the first target image is an image within a visible range having the same size as the screen of the head mounted device and a size of the first target image is smaller than a size of the image to be displayed;

[0038] A first display module, configured to display a first target image;

[0039] A second acquisition module is configured to acquire second posture data, where the second posture data is posture data of the head mounted device at the moment when the first target image is displayed;

[0040] A second selection module, configured to select a second target image from the image to be displayed based on the first posture data, the second posture data, and the first target image;

[0041] The second display module is used to display a second target image.

[0042] According to a third aspect of the present invention, there is provided a head-mounted device, the head-mounted device comprising the image display device according to the second aspect; or,

[0043] The head-mounted device includes a memory and a processor, the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the image display method as described in any one of the first aspects.

[0044] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image display method according to any one of the first aspects is implemented.

[0045] In an embodiment of the present invention, an image display method is provided. The method includes: acquiring original image data and first pose data of a head-mounted device, the first pose data being the pose data of the head-mounted device at the moment the original image data is acquired; generating an image to be displayed based on the original image data; selecting a first target image from the image to be displayed, the first target image being an image within the visible range and having the same size as the head-mounted device screen and smaller in size than the image to be displayed; displaying the first target image and acquiring second pose data, the second pose data being the pose data of the head-mounted device at the moment the first target image is displayed; selecting a second target image from the image to be displayed based on the first pose data, the second pose data, and the first target image; and displaying the second target image. This method enables the display of an intermediate frame image after the first target image is displayed. This implements intermediate frame technology, replacing the traditional ATW technology. In other words, the embodiment of the present invention provides a new intermediate frame generation technology that can replace the ATW function.

[0046] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0048] FIG1 is a block diagram 1 of a hardware configuration of a head mounted device for implementing an image display method according to an embodiment of the present invention;

[0049] FIG2 is a first flow chart of a method for implementing image display according to an embodiment of the present invention;

[0050] FIG3 is a first flow chart of a method for implementing image display according to an embodiment of the present invention;

[0051] FIG4 is a schematic diagram of a dispersion principle provided according to an embodiment of the present invention;

[0052] FIG5 is a schematic structural diagram of an image display device according to an embodiment of the present invention;

[0053] FIG6 is a second block diagram of a hardware configuration of a head mounted device for implementing an image display method according to an embodiment of the present invention.

[0054] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0056] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications will also change accordingly.

[0057] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the meaning of "and / or" appearing throughout the text includes three parallel schemes. Taking "A and / or B" as an example, it includes scheme A, or scheme B, or a scheme in which A and B are satisfied at the same time. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0058] FIG1 is a block diagram 1 of a hardware configuration of a head mounted device for implementing an image display method according to an embodiment of the present invention.

[0059] Among them, the head-mounted device is exemplarily an AR, MR, VR or XR head-mounted device. In addition, the head-mounted device can be a split head-mounted device or an integrated head-mounted device. For a split head-mounted device, it is equipped with a head-mounted display device (such as AR glasses or a helmet, etc.) and a corresponding control device, and the control device is specifically a data processing unit of the head-mounted display device. The control device and the integrated head-mounted device are devices with an operating system. The operating system can be an Android operating system, an iOS operating system, or of course other possible operating systems, which are not specifically limited in the embodiments of the present invention.

[0060] It should be noted that when the head-mounted device is a split-type head-mounted device, technicians can deploy all or part of the steps in the image display method provided in the embodiment of the present invention on the head-mounted display device or control device according to actual conditions.

[0061] The head-mounted device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and the like. The processor 1100 may be a central processing unit (CPU), a microprocessor (MCU), a display processor (DPU), or the like. The memory 1200 may include, for example, ROM (read-only memory), RAM (random access memory), or a non-volatile memory such as a hard disk. The interface device 1300 may include, for example, a USB port or a headphone jack. The communication device 1400 may be capable of wired or wireless communication. The display device 1500 may be, for example, an LCD display or a touchscreen display. The input device 1600 may include, for example, a touchscreen or a keyboard. The user may input and output voice information through the speaker 1700 and microphone 1800.

[0062] Although multiple devices are shown for the head-mounted device 1000 in FIG1 , the present invention may only relate to some of the devices. For example, the head-mounted device 1000 may only relate to the memory 1200 , the processor 1100 , and the display device 1500 .

[0063] In the embodiment of the present invention, the memory 1200 of the head mounted device 1000 is used to store instructions, which are used to control the processor 1100 to execute the image display method provided by the embodiment of the present invention.

[0064] In the above description, a person skilled in the art can design instructions according to the solution disclosed in the present invention. How instructions control the operation of a processor is well known in the art and will not be described in detail here.

[0065] An embodiment of the present invention provides an image display method, which can generate intermediate frame images to replace the traditional ATW technology.

[0066] As shown in FIG2 , the method includes the following steps S2100 to S2600 :

[0067] S2100: Obtain original image data and first position data of the head-mounted device.

[0068] Among them, the first pose data is the pose data of the head-mounted device at the moment of acquiring the original image.

[0069] In an embodiment of the present invention, for an integrated head-mounted device, the raw image data is data captured by multiple image acquisition devices (e.g., cameras) on the head-mounted device that capture images of the external environment from different angles. Furthermore, the head-mounted device is provided with a posture sensor that acquires posture data of the head-mounted device. The posture data acquired by the posture sensor at the time the raw image is acquired is recorded as the first posture data.

[0070] For split-body head-mounted devices, the raw image data is captured by multiple image acquisition devices (e.g., cameras) on the head-mounted display device, which capture images of the external environment from different angles. Furthermore, the head-mounted display device is equipped with a posture sensor, which acquires the posture data of the head-mounted display device. The posture data acquired by the posture sensor at the time the raw image is acquired is recorded as the first posture data.

[0071] Based on the above content, it can be understood that the first position data is specifically the position data at the moment of image rendering, and the first position data can reflect the head position information of the wearer at the moment of image rendering.

[0072] S2200: Generate an image to be displayed according to original image data.

[0073] In this embodiment of the present invention, the raw image data is data captured by multiple image acquisition devices (e.g., cameras) that capture images of the external environment from different angles of a head-mounted device or head-mounted display. Based on this, the raw image data is fused to produce an image reflecting the environment surrounding the head-mounted device or head-mounted display, which is recorded as the image to be displayed. It will be understood that the image to be displayed includes the image displayed at the most recent display moment.

[0074] It should be noted that, in the embodiment of the present invention, the size of the image to be displayed needs to be larger than the size of the screen of the head-mounted device. In one example, the size of the image to be displayed is 1.2 times the size of the screen of the head-mounted device.

[0075] S2300: Acquire a first target image from images to be displayed.

[0076] The first target image is an image within the visible range that has the same size as the screen of the head mounted device, and the size of the first target image is smaller than the size of the image to be displayed.

[0077] In an embodiment of the present invention, the first target image is an image that needs to be displayed on the screen of the head mounted device at the latest display moment.

[0078] In one embodiment of the present invention, the above S2300 can be specifically implemented by the following S2310 and S2311:

[0079] S2310: Obtain an eye movement heat map of a wearer wearing a head mounted device.

[0080] In an embodiment of the present invention, the head-mounted device is further provided with an image acquisition device (such as a camera) for acquiring an image of the wearer's eyes. The image acquisition device is used to acquire an image of the wearer's eyes. On this basis, in one embodiment of the present invention, the specific implementation of the above-mentioned S2310 may be: inputting the image of the wearer's eyes into an eye tracking algorithm to obtain a human eye movement heat map. Among them, the human eye movement heat map is used to reflect the wearer's gaze and can show the wearer's attention distribution. Generally, the red color in the human eye movement heat map represents the area with the most concentrated gaze, and the yellow and green colors represent areas with less gaze.

[0081] S2311. Determine a first target image according to the human eye movement heat map and the image to be displayed.

[0082] In one embodiment of the present invention, an image to be displayed that contains a human eye movement heat map and has the same size as the screen of the head-mounted device may be determined as the first target image.

[0083] Furthermore, an image centered on the human eye movement heat map, containing the human eye movement heat map and having the same size as the screen of the head-mounted device can be determined as the first target image.

[0084] In one embodiment of the present invention, since the size of the first target image is the same as the screen size of the head-mounted device, when selecting the target image from the images to be displayed, it is necessary to first determine the screen size of the head-mounted device. Based on this, in one embodiment of the present invention, the above S2300 can be specifically implemented through the following S2320 and S2321:

[0085] S2320. Obtain the screen size of the head mounted device.

[0086] In one embodiment of the present invention, the size of the screen of the head-mounted device is pre-stored in the head-mounted device. Based on this, the above S2320 can be implemented.

[0087] S2321. Select a first target image from the images to be displayed according to the size.

[0088] In the embodiment of the present invention, a first target image having a size equal to the size of a screen of a head mounted device is selected from the images to be displayed.

[0089] S2400: Display a first target image and obtain second pose data.

[0090] The second posture data is the posture data of the head-mounted device at the moment of displaying the first target image.

[0091] It should be noted that the method for obtaining the second posture data in S2400 is the same as the method for obtaining the first posture data in S2100, and will not be repeated here. The second posture data can reflect the head posture information of the wearer at the time of image display.

[0092] In the embodiment of the present invention, when the image display time arrives, the first target image is displayed, based on which the display of the previous real image frame of the intermediate frame image can be realized.

[0093] S2500: Select a second target image from the images to be displayed according to the first pose data, the second pose data, and the first target image.

[0094] The second target image is an intermediate frame image.

[0095] In this embodiment of the present invention, the change in posture between the image display and image rendering times can be determined based on the first and second posture data. Based on this change in posture, it can be inferred which orientation of the partial image of the to-be-displayed image, relative to the first target image, the wearer saw after viewing the first target image. In this embodiment of the present invention, the inferred partial image is recorded as the second target image.

[0096] Since the first target image and the second target image are both images to be displayed, and the second target image is a partial image of a certain orientation of the image to be displayed relative to the first target image inferred based on the change in the above-mentioned posture, the second target image and the first target image are continuous and can serve as intermediate frame images.

[0097] In one embodiment of the present invention, the above S2500 can be specifically implemented through the following S2510-S2512:

[0098] S2510: Determine, based on the first posture data and the second posture data, a posture change amount of the head mounted device at the display time compared to the acquisition time.

[0099] In the embodiment of the present invention, the difference between the first pose data and the second pose data is used as the pose change.

[0100] S2511. Determine image selection parameters based on the posture change.

[0101] The image selection parameter includes at least one of a scaling factor and an offset relative to the first target image.

[0102] In an embodiment of the invention, at least one of an offset and a scaling factor of the image viewed by the wearer relative to the first target image is determined based on the change in posture. The scaling factor is a positive number greater than 0. If the scaling factor is greater than 0 and less than 1, the image to be displayed is scaled down. If the scaling factor is greater than 1, the image to be displayed is scaled up. If the scaling factor is equal to 1, the image to be displayed is not scaled. The offset can be understood as the amount of window sliding relative to the first target image and is a vector with a direction and magnitude.

[0103] In one example, based on the principle that objects appear larger when they are closer and smaller when they are farther away, if the change in posture indicates that the wearer is close to an object image in the first target image, the scaling factor is determined to be less than 1. Furthermore, the specific value of the scaling factor can be determined based on the degree of proximity.

[0104] In the case where the posture change indicates that the wearer deviates from a certain object image in the first target image, the offset is determined to be the deviation from the aforementioned certain object image.

[0105] It should be noted that when the image selection parameters include a scaling factor and an offset relative to the first target image, the image to be displayed is first scaled according to the scaling factor. Then, the second target image is selected from the scaled image to be displayed based on the offset relative to the first target image. In this scenario, even if the image to be displayed is scaled, the size of the window corresponding to the offset does not scale with the scaling of the image to be displayed; that is, the size of the window corresponding to the offset does not change.

[0106] In one embodiment of the present invention, the above S2511 can be specifically implemented by the following S2511-1:

[0107] S2511-1. Determine image selection parameters based on the posture change and preset mapping data.

[0108] The preset mapping data is data reflecting the corresponding relationship between the posture change amount and the image selection parameters.

[0109] In one embodiment of the present invention, the process of generating the preset mapping data may be:

[0110] The designers of the head-mounted device conducted multiple experiments, each of which involved the following steps: viewing an image through the head-mounted device at time T1, recording the image at time T1 as a first image, and recording the pose data of the head-mounted device at time T1; viewing an image through the head-mounted device at time T2, recording the image at time T2 as a second image, and recording the pose data of the head-mounted device at time T2; calculating the difference between the pose data of the head-mounted device at time T1 and the pose data of the head-mounted device at time T2, as well as the offset and scaling factor of the second image relative to the first image; and using the aforementioned difference and the corresponding offset and scaling factor as a set of mapping data;

[0111] Fitting multiple sets of mapping data obtained from multiple experiments to obtain a functional relationship reflecting the corresponding relationship between the pose change and the image selection parameters;

[0112] The above functional relationship is used as the preset mapping data in the above S2511-1.

[0113] S2512: Select a second target image from the images to be displayed according to the image selection parameter and the first target image.

[0114] In an example, taking the image selection parameters including a scaling factor and an offset relative to the first target image, and the scaling factor being 0.8 and the offset being 300 pixels to the left and 400 pixels upward as an example, the selection process of the second target image is specifically as follows: reducing the image to be displayed to 0.8 times the size of the original image to be displayed; in the reduced image to be displayed, taking the center of the first target image as the center and a window with the same size as the first target image in the original image to be displayed before scaling as the center, slide 500 pixels to the upper left, and the image in the window is used as the second target image.

[0115] S2600: Display a second target image.

[0116] Based on the above S2600 , the display of the intermediate frame image can be completed.

[0117] After executing the above S2600 , the above S2100 - S2600 are repeatedly executed, so that the intermediate frame technology can be implemented, replacing the traditional ATW technology.

[0118] In an embodiment of the present invention, an image display method is provided. The method includes: acquiring original image data and first pose data of a head-mounted device, the first pose data being the pose data of the head-mounted device at the moment the original image data is acquired; generating an image to be displayed based on the original image data; selecting a first target image from the image to be displayed, the first target image being an image within the visible range and having the same size as the head-mounted device screen and smaller in size than the image to be displayed; displaying the first target image and acquiring second pose data, the second pose data being the pose data of the head-mounted device at the moment the first target image is displayed; selecting a second target image from the image to be displayed based on the first pose data, the second pose data, and the first target image; and displaying the second target image. This method enables the display of an intermediate frame image after the first target image is displayed. This implements intermediate frame technology, replacing the traditional ATW technology. In other words, the embodiment of the present invention provides a new intermediate frame generation technology that can replace the ATW function.

[0119] In one embodiment of the present invention, the second target image is an RGB image. Based on this, the specific implementation of S2600 may be as follows S2610-S2613:

[0120] S2610 , extracting a red pixel image, a green pixel image, and a blue pixel image from the second target image.

[0121] In embodiments of the present invention, due to the dispersion of the head-mounted device's lenses, the second target image may exhibit dispersion issues, as shown in Figure 3, before dispersion correction is performed. Specifically, within each pixel of the second target image, the R, G, and B pixels do not overlap. This significantly impacts the wearer's viewing experience. Therefore, to avoid this issue, dispersion correction can be performed after the second target image is obtained, and the dispersion-corrected second target image can then be displayed.

[0122] When performing dispersion correction, S2610 is first executed. Specifically, an image consisting of red pixels in each pixel of the second target image is recorded as a red pixel image. An image consisting of green pixels in each pixel of the second target image is recorded as a green pixel image. And, an image consisting of blue pixels in each pixel of the second target image is recorded as a blue pixel image.

[0123] S2611: Perform dispersion correction processing on the red pixel image, the green pixel image, and the blue pixel image according to the dispersion parameters of the lens of the head-mounted device.

[0124] In this embodiment of the present invention, when the lens of the head-mounted device is fixed, the dispersion parameters of the lens of the head-mounted device are fixed. Based on this, the dispersion parameters of the lens of the head-mounted device are first obtained before executing S2611. Furthermore, according to the optical model of the lens, dispersion correction processing is performed on the red pixel image, the green pixel image, and the blue pixel image using the dispersion parameters of the lens of the head-mounted device.

[0125] S2612: Fusing the red pixel image, the green pixel image, and the blue pixel image after the dispersion correction to obtain a second target image after the dispersion correction.

[0126] In an embodiment of the present invention, after obtaining the red pixel image, green pixel image, and blue pixel image after dispersion correction, the three are fused. In the fused image, R pixels, G pixels, and B pixels at the same pixel position overlap.

[0127] S2613: Display the second target image after dispersion correction.

[0128] In the embodiment of the present invention, displaying the second target image after dispersion correction can avoid the problem of affecting the wearer's viewing experience due to the existence of dispersion in the lens of the head-mounted device.

[0129] In one embodiment of the present invention, the image display method provided by the embodiment of the present invention is executed by the display processor unit (DPU) of the head-mounted device. Compared to executing the image display method provided by the embodiment of the present invention by the CPU (or GPU), this does not increase the resource usage of the CPU (or GPU) and does not increase the power consumption of the CPU (or GPU).

[0130] Based on the above content, the image display method provided by the embodiment of the present invention includes the following as shown in FIG4:

[0131] S4001, obtaining original image data and first position data of the head-mounted device, where the first position data is the position data of the head-mounted device at the moment of obtaining the original image data;

[0132] S4002, generating an image to be displayed according to the original image data;

[0133] S4003, selecting a first target image from the images to be displayed;

[0134] S4004: Display the first target image and obtain second posture data, where the second posture data is the posture data of the head mounted device at the moment of displaying the first target image;

[0135] S4005: Determine, based on the first posture data and the second posture data, a change in the posture of the head mounted device at the display time compared to the acquisition time;

[0136] S4006. Determine an image selection parameter according to the posture change, where the image selection parameter includes at least one of a scaling factor and an offset relative to the first target image;

[0137] S4007, selecting a second target image from the images to be displayed according to the image selection parameter and the first target image;

[0138] S4008, extracting a red pixel image, a green pixel image, and a blue pixel image from the second target image;

[0139] S4009: Perform dispersion correction processing on the red pixel image, the green pixel image, and the blue pixel image according to the dispersion parameters of the lens of the head mounted device;

[0140] S4010, fusing the red pixel image, the green pixel image, and the blue pixel image after dispersion correction to obtain a second target image after dispersion correction;

[0141] S4011. Display the second target image after dispersion correction.

[0142] The embodiment of the present invention further provides an image display device 500, as shown in FIG5 , wherein the device 500 includes:

[0143] A first acquisition module 510 is configured to acquire original image data and first pose data of a head-mounted device, where the first pose data is the pose data of the head-mounted device at the moment of acquiring the original image data;

[0144] A generating module 520, configured to generate an image to be displayed based on the original image data;

[0145] A first selection module 530 is configured to select a first target image from the images to be displayed, where the first target image is an image within a visible range and has the same size as the screen of the head mounted device, and the size of the first target image is smaller than the size of the image to be displayed;

[0146] A first display module 540 is configured to display a first target image;

[0147] A second acquisition module 550 is configured to acquire second posture data, where the second posture data is posture data of the head mounted device at the moment when the first target image is displayed;

[0148] A second selection module 560 is configured to select a second target image from the image to be displayed based on the first posture data, the second posture data, and the first target image;

[0149] The second display module 570 is configured to display a second target image.

[0150] An embodiment of the present invention also provides an image display device, comprising: a first acquisition module for acquiring original image data and first pose data of a head-mounted device, the first pose data being the pose data of the head-mounted device at the moment the original image data is acquired; a generation module for generating an image to be displayed based on the original image data; a first selection module for selecting a first target image from the images to be displayed, the first target image being an image within the visible range and having the same size as the head-mounted device screen and smaller than the image to be displayed; a first display module for displaying the first target image; a second acquisition module for acquiring second pose data, the second pose data being the pose data of the head-mounted device at the moment the first target image is displayed; a second selection module for selecting a second target image from the images to be displayed based on the first pose data, the second pose data, and the first target image; and a second display module for displaying the second target image. This device can display an intermediate frame image after the first target image is displayed. This implements intermediate frame technology, replacing the traditional ATW technology. In other words, the embodiment of the present invention provides a new intermediate frame generation technology that can replace the ATW function.

[0151] In one embodiment of the present invention, the second selection module 560 is specifically configured to determine, based on the first posture data and the second posture data, a posture change amount of the head mounted device between the display time and the acquisition time;

[0152] Determining an image selection parameter according to the pose change, the image selection parameter including at least one of a scaling factor and an offset relative to the first target image;

[0153] A second target image is selected from the images to be displayed according to the image selection parameter and the first target image.

[0154] In one embodiment of the present invention, the second display module 570 is specifically configured to extract a red pixel image, a green pixel image, and a blue pixel image from the second target image;

[0155] performing dispersion correction processing on the red pixel image, the green pixel image, and the blue pixel image respectively according to the dispersion parameter of the lens of the head mounted device;

[0156] fusing the red pixel image, the green pixel image, and the blue pixel image after the dispersion correction process to obtain a second target image after dispersion correction;

[0157] The second target image after the dispersion correction is displayed.

[0158] In one embodiment of the present invention, the first selection module 530 is specifically configured to obtain an eye movement heat map of a wearer wearing the head mounted device;

[0159] A first target image is determined according to the human eye movement heat map and the image to be displayed.

[0160] In one embodiment of the present invention, the second selection module 560 is specifically configured to obtain the size of the screen of the head mounted device;

[0161] A first target image is selected from the images to be displayed according to the size.

[0162] In one embodiment of the present invention, the apparatus is disposed in a display processor.

[0163] An embodiment of the present invention further provides a head mounted device 600, wherein the head mounted device 600 includes the image display device 500 provided by any of the above device embodiments; or

[0164] As shown in FIG6 , the head mounted device 600 includes a memory 610 and a processor 620 , wherein the memory 610 is used to store computer instructions, and the processor 620 is used to call the computer instructions from the memory 610 to execute the image display method as described in any one of the above method embodiments.

[0165] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the image display method according to any one of the above method embodiments is implemented.

[0166] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0167] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0168] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0169] The computer program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), is personalized by utilizing the state information of the computer readable program instructions, and the electronic circuit can execute the computer readable program instructions, thereby realizing various aspects of the present invention.

[0170] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0171] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0172] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0173] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0174] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0175] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0176] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as mentioned above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, TV, or network device, etc.) to execute the methods of various embodiments of the present invention.

[0177] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made by using the contents of the present invention description and drawings under the inventive concept of the present invention, or direct / indirect application in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. An image display method, characterized in that: The method comprises: Acquire original image data and first pose data of a head mounted device, wherein the first pose data is the pose data of the head mounted device at the moment of acquiring the original image data; Generate an image to be displayed according to the original image data; Selecting a first target image from the images to be displayed, where the first target image is an image within a visible range with the same size as the screen size of the head mounted device, and the size of the first target image is smaller than the size of the image to be displayed; Display a first target image and obtain second posture data, where the second posture data is the posture data of the head mounted device at the moment of displaying the first target image; Selecting a second target image from the images to be displayed according to the first posture data, the second posture data and the first target image; The second target image is displayed.

2. The method according to claim 1, characterized in that The selecting a second target image from the image to be displayed according to the first posture data, the second posture data and the first target image comprises: Determine, according to the first posture data and the second posture data, a posture change amount of the head mounted device at the display time compared to the acquisition time; Determining an image selection parameter according to the posture change, the image selection parameter comprising at least one of a scaling factor and an offset relative to the first target image; A second target image is selected from the images to be displayed according to the image selection parameter and the first target image.

3. The method according to claim 1, characterized in that The second target image is an RGB image, and displaying the second target image includes: Extracting a red pixel image, a green pixel image, and a blue pixel image from the second target image; and performing dispersion correction processing on the red pixel image, the green pixel image, and the blue pixel image, respectively, according to the dispersion parameter of the lens of the head-mounted device; Fusing the red pixel image, the green pixel image and the blue pixel image after the dispersion correction process to obtain a second target image after dispersion correction; The second target image after the dispersion correction is displayed.

4. The method according to claim 1, characterized in that: The selecting a first target image from the images to be displayed comprises: Obtaining an eye movement heat map of a wearer wearing the head mounted device; A first target image is determined according to the human eye movement heat map and the image to be displayed.

5. The method according to claim 2, characterized in that: The step of determining the image selection parameters according to the posture change comprises: Determining image selection parameters according to the posture change amount and preset mapping data; The preset mapping data is data reflecting the corresponding relationship between the posture change amount and the image selection parameters.

6. The method according to claim 1, characterized in that The selecting a first target image from the images to be displayed comprises: Obtaining the size of the screen of the head mounted device; A first target image is selected from the images to be displayed according to the size.

7. The method according to any one of claims 1 to 6, characterized in that: The method is executed by a display processor.

8. An image display device, characterized in that: The device comprises: A first acquisition module, used to acquire original image data and first pose data of a head mounted device, wherein the first pose data is the pose data of the head mounted device at the moment of acquiring the original image data; A generating module, used for generating an image to be displayed according to the original image data; A first selection module, configured to select a first target image from the images to be displayed, wherein the first target image is an image within a visible range with the same size as the screen size of the head mounted device, and the size of the first target image is smaller than the size of the image to be displayed; A first display module, used for displaying a first target image; A second acquisition module is used to acquire second posture data, where the second posture data is the posture data of the head mounted device at the moment when the first target image is displayed; A second selection module, configured to select a second target image from the image to be displayed according to the first posture data, the second posture data and the first target image; The second display module is used to display a second target image.

9. A head mounted device, characterized in that: The head mounted device comprises the image display device according to claim 8; or, The head mounted device comprises a memory and a processor, the memory being used to store computer instructions, and the processor being used to call the computer instructions from the memory to execute the image display method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the image display method according to any one of claims 1 to 7 is implemented.

Images