Gaze point rendering method and device, head-mounted display equipment and storage medium

Abstract

The embodiment of the invention provides a fixation point rendering method and device, head-mounted display equipment and a storage medium, and belongs to the field of image rendering. The method comprises the following steps: taking pixel data of a fixation point in a preset character array as texture data to be transmitted to a buffer area proxy object for storage; and rendering and generating a target image corresponding to the fixation point according to the texture data in the buffer area proxy object. According to the technical scheme of the embodiment of the invention, the rendering efficiency of the fixation point is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a foveated rendering method, device, head-mounted display device and storage medium. Background Art

[0002] Currently, the user's gaze point can be rendered and displayed in the display page of the head-mounted display device, so that the user can know the display area he is focusing on. However, the rendering of the gaze point involves data transmission between the system layer and the engine layer. In a computer system, data transmission between different layers usually takes a certain amount of time, and the data transmission between the system layer and the engine layer will affect the rendering efficiency of the gaze point. Summary of the invention

[0003] The embodiments of the present invention provide a foveation rendering method, an apparatus, a head-mounted display device and a storage medium, aiming to improve the rendering efficiency of the foveation point.

[0004] In a first aspect, an embodiment of the present invention provides a foveated rendering method, which is applied to a head mounted display device, and the method includes:

[0005] The pixel data of the gaze point in the preset character array is transferred as texture data to the buffer proxy object for storage;

[0006] A target image corresponding to the gaze point is generated by rendering according to the texture data in the buffer proxy object.

[0007] In a second aspect, an embodiment of the present invention further provides a foveation rendering device, which is applied to a head mounted display device, and the foveation rendering device includes:

[0008] A data processing module is configured to transmit pixel data of a fixation point in a preset character array as texture data to the buffer proxy object for storage;

[0009] The image rendering module is configured to render and generate a target image corresponding to the gaze point according to the texture data in the buffer proxy object.

[0010] In a third aspect, an embodiment of the present invention further provides a head-mounted display device, comprising a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the foveated point rendering method as described in the first aspect is implemented.

[0011] In a fourth aspect, an embodiment of the present invention further provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the foveated rendering method as described in the first aspect.

[0012] The embodiment of the present invention provides a foveation rendering method, device, head-mounted display device and storage medium. The method uses a preset character array to cache pixel data of a foveation point, which can quickly pass the pixel data cached by the preset character array as texture data to a buffer proxy object for storage, thereby facilitating reading texture data from the buffer proxy object to render and generate a target image corresponding to the foveation point. The texture data is read at a fast speed and does not involve data transmission between the system layer and the engine layer. In this way, the time required for data transmission between the system layer and the engine layer in the process of rendering the foveation point can be omitted, greatly improving the rendering efficiency of the foveation point. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0014] Figure 1 is a schematic diagram of a scene for implementing the foveated rendering method provided by an embodiment of the present invention;

[0015] Figure 2 is a flow chart of a foveated rendering method provided by an embodiment of the present invention;

[0016] Figure 3 yes Figure 2 A schematic diagram of the sub-step flow of the foveated rendering method in FIG.

[0017] Figure 4 is a flowchart of another foveated rendering method provided by an embodiment of the present invention;

[0018] Figure 5 is a flowchart of another foveated rendering method provided by an embodiment of the present invention;

[0019] Figure 6 is a schematic diagram of a three-dimensional coordinate system in an embodiment of the present invention;

[0020] Figure 7 is a schematic block diagram of the structure of a foveated rendering device provided by an embodiment of the present invention;

[0021] Figure 8The present invention is a schematic structural block diagram of a head mounted display device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0024] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0025] Currently, the user's gaze point can be rendered and displayed in the display page of the head-mounted display device, so that the user can know the display area he is focusing on. However, the rendering of the gaze point involves data transmission between the system layer and the engine layer. In a computer system, data transmission between different layers usually takes a certain amount of time, and the data transmission between the system layer and the engine layer will affect the rendering efficiency of the gaze point.

[0026] To solve the above problems, an embodiment of the present invention provides a foveated rendering method, device, head-mounted display device and storage medium. The method uses a preset character array to cache pixel data of the foveated point, which can quickly pass the pixel data cached by the preset character array as texture data to the buffer proxy object for storage, thereby facilitating reading texture data from the buffer proxy object to render and generate a target image corresponding to the foveated point. The texture data is read at a fast speed and does not involve data transmission between the system layer and the engine layer. In this way, the time required for data transmission between the system layer and the engine layer in the process of rendering the foveated point can be omitted, greatly improving the rendering efficiency of the foveated point.

[0027] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0028] See also Figure 1 , Figure 1It is a schematic diagram of a scene for implementing the foveated rendering method provided in an embodiment of the present invention.

[0029] like Figure 1 As shown, the head mounted display device 100 is worn on the user's head 11, and the head mounted display device 100 includes a display screen 101 and a gaze point detection device ( Figure 1 (not shown). The display screen 101 is used to display images, videos or pages, etc., and the gaze point detection device is used to detect the current position of the user's gaze point in the display screen 101 of the head mounted display device 100.

[0030] In some embodiments, the display screen 101 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed or a quantum dot light-emitting diode (QLED), etc.

[0031] In different implementations of the present invention, the gaze point detection device may have different compositions depending on the different gaze point detection schemes adopted. For example, the gaze point detection device may include a laser source for generating detection light, a reflective micromirror for adjusting the incident angle / incident position of the detection light emitted by the laser source relative to the human eye, and a photodetector for detecting the light signal after the detection light is reflected by the human eye. For example, the photodetector may be a photodiode. In some implementations, the gaze point detection device may also be provided with a scanner. Among them, the reflective component in the scanner may be used to realize the role of the reflective micromirror. A micro-electro-mechanical system (MEMS) driver may also be provided in the scanner to adjust the angle of the reflective micromirror. The gaze point detection device may determine the gaze point of the human eye by a photodiode response method.

[0032] For example, the laser source can emit a detection laser to the reflective micromirror, and the reflective micromirror can reflect the detection laser to the human eye. The detection laser is reflected by the human eye through the refraction of the cornea, and is received by the photodiode after refraction of the cornea again. The photodiode can receive the detection laser, determine the intensity of the detection laser, and generate a corresponding electrical signal response according to the received intensity of the detection laser. The position of the human eye can be determined according to the electrical signal response, and then the position of the gaze point in the display screen 101 at the current moment can be determined according to the correspondence between the pre-calibrated human eye position and the coordinates of the gaze point in the display screen 101.

[0033] In some embodiments, the gaze point detection device includes a detection light source and a shooting unit. In this embodiment, the gaze point detection device can be used to determine the gaze point of the human eye through a model-based video eye diagram solution. For example, the head-mounted display device 100 can use a gaze point detection device to determine the position of the gaze point of the human eye in the display screen 101 using an eye tracking method. For example, when the light emitted by the detection light source is irradiated on the human eye, the human eye can be photographed by the shooting unit to the current human eye response image. Then, the current human eye position is determined by analyzing the human eye response image, and then the position of the gaze point in the display screen 101 at the current moment is determined based on the current human eye position and the correspondence between the pre-calibrated human eye position and the coordinates of the gaze point in the display screen 101.

[0034] In some implementations, the shooting unit may include a lens for adjusting the field of view angle and converging light, and an image sensor for photosensitivity. For example, the image sensor may be an image sensor based on a complementary metal oxide semiconductor (CMOS) or an image sensor based on a charge coupled device (CCD). It should be understood that in the above examples, several specific implementations of the functions of the shooting unit are listed, and in other implementations of the present invention, the shooting unit may also realize its functions through other components, which will not be elaborated here.

[0035] When executing the gaze point rendering method provided by the embodiment of the present invention, the head mounted display device 100 transmits the pixel data of the gaze point in the preset character array as texture data to the buffer proxy object for storage; and renders and generates the target image corresponding to the gaze point according to the texture data in the buffer proxy object. The entire rendering process does not involve data transmission between the system layer and the engine layer, so that the time required for data transmission between the system layer and the engine layer in the process of rendering the gaze point can be omitted, and by transmitting the pixel data cached in the preset character array as texture data to the buffer proxy object for storage, the texture image can be quickly read from the buffer proxy object, thereby greatly improving the rendering efficiency of the gaze point.

[0036] In some embodiments, after rendering and generating a target image corresponding to the gaze point, the head-mounted display device 100 obtains the current position of the gaze point of the user wearing the head-mounted display device 100 in the display screen 101 of the head-mounted display device 100 through a gaze point detection device, and displays the target image at the current position of the display screen 101.

[0037] In some embodiments, the head mounted display device 100 includes a sensor ( Figure 1 (not shown). For example, the sensor for collecting the posture data of the head mounted display device 100 may include an inertial measurement unit (IMU), and the inertial measurement unit may include an accelerometer, a gyroscope and / or a magnetometer. For example, the inertial measurement unit includes a three-axis accelerometer, a three-axis gyroscope and a three-axis magnetometer. Among them, the head mounted display device 100 may include augmented reality (AR) glasses, AR helmets, mixed reality (MR) glasses and MR helmets, etc.

[0038] The following will be combined Figure 1 The scene in the embodiment of the present invention provides a detailed introduction to the foveated rendering method provided by the embodiment of the present invention. It should be noted that Figure 1 The scenario is only used to explain the foveated rendering method provided in the embodiment of the present invention, but does not constitute a limitation on the application scenario of the foveated rendering method provided in the embodiment of the present invention.

[0039] See also Figure 2 , Figure 2 It is a flowchart of a foveated rendering method provided by an embodiment of the present invention.

[0040] like Figure 1 As shown, the foveated rendering method includes steps S101 to S102.

[0041] Step S101: transmitting pixel data of a gaze point in a preset character array as texture data to a buffer proxy object for storage.

[0042] In this embodiment, the gaze point refers to the point where the user's eyes fall on the display screen when the user looks at the display screen of the head-mounted display device. The pixel data of the gaze point in the preset character array can be set or updated based on actual business needs, and the embodiment of the present invention does not specifically limit this.

[0043] In some embodiments, before step S101, it also includes: determining whether a buffer proxy object has been created; if the buffer proxy object has been created, executing step S101, that is, transferring the pixel data of the gaze point in the preset character array as texture data to the buffer proxy object for storage; if the buffer proxy object has not been created, creating a buffer proxy object, and then executing step S101, that is, transferring the pixel data of the gaze point in the preset character array as texture data to the buffer proxy object for storage. Wherein, the buffer proxy object is a GraphicBuffer object, and the GraphicBuffer object is used to store the texture data of the gaze point. It can be understood that the texture data of the gaze point stored in the initially created buffer proxy object is empty.

[0044] In some embodiments, during the process of starting up the head-mounted display device, the current state identifier of the gaze point rendering button in the head-mounted display device is obtained; when the current state identifier of the gaze point rendering button is a first state identifier, it is determined that the gaze point rendering function of the head-mounted display device has been turned on, and the pixel data of the gaze point in the preset character array is transferred as texture data to the buffer proxy object for storage; when the current state identifier of the gaze point rendering button is a second state identifier, it is determined that the gaze point rendering function of the head-mounted display device is not turned on, and no processing is performed at this time. Among them, the specific implementation method of the gaze point rendering button can be set based on actual conditions, and the embodiment of the present invention does not make specific limitations on this. The first state identifier and the second state identifier are different. By determining that the gaze point rendering function has been turned on during the process of starting up the head-mounted display device, the pixel data of the gaze point in the preset character array is transferred as texture data to the buffer proxy object for storage in advance, so that after the head-mounted display device is turned on, the target image corresponding to the gaze point can be displayed in time, reducing the waiting time of the user.

[0045] In some embodiments, the head-mounted display device stores a plurality of preset character arrays, and the pixel data of the gaze point in different preset character arrays are different. Transferring the pixel data of the gaze point in the preset character array as texture data to the buffer proxy object for storage may include: obtaining the identity information of the user wearing the head-mounted display device, and determining the preset character array matching the identity information as the target character array; transferring the pixel data of the gaze point in the target character array as texture data to the buffer proxy object for storage. This embodiment adaptively transfers the pixel data of the gaze point in the corresponding preset character array as texture data to the buffer proxy object for storage according to the user using the head-mounted display device, so that the image of the gaze point subsequently rendered and generated according to the texture data can be more in line with the user's preferences, realizing personalized rendering of the gaze point and improving the user experience.

[0046] In some embodiments, determining a preset character array matching the identity information as the target character array may include: obtaining a pre-established correspondence table between identity information and array identifiers, obtaining an array identifier corresponding to the identity information of a user currently wearing the head mounted display device from the relationship table, and determining the preset character array corresponding to the obtained array identifier as the target character array. The identity information may be iris information.

[0047] For example, the correspondence table between identity identification information and array identification can be established in the following manner: when using a head-mounted display device, the user can select a favorite pattern of a gaze point (which can be a two-dimensional pattern or a three-dimensional pattern), and then the head-mounted display device collects the user's identity identification information (for example, the user's iris information), and binds the array identification of the preset character array corresponding to the pattern of the gaze point selected by the user to the collected identity identification information to establish a correspondence between the identity identification information and the array identification, and stores the correspondence between the identity identification information and the array identification in a data table to obtain a correspondence table between the identity identification information and the array identification.

[0048] Step S102: Render and generate a target image corresponding to the gaze point according to the texture data in the buffer proxy object.

[0049] In this embodiment, after the pixel data of the gaze point in the preset character array is transmitted as texture data to the buffer proxy object for storage, the graphics processing unit (GPU) in the head mounted display device can read the texture data from the buffer proxy object for image rendering to obtain the target image corresponding to the gaze point. The target image corresponding to the gaze point can be a two-dimensional image or a three-dimensional image, which is not specifically limited in the embodiment of the present invention.

[0050] In some embodiments, Figure 3 As shown, step S102 includes: sub-steps S1021 to S1023.

[0051] Sub-step S1031: Create a three-dimensional framework model of the gaze point and obtain multiple texture coordinates of the gaze point.

[0052] In this embodiment, the three-dimensional frame model of the gaze point is used to describe the mathematical model of the three-dimensional object corresponding to the gaze point, and it is composed of a series of line segments or edges, which are connected to each other to form a frame structure. Among them, the texture coordinates of the gaze point are the coordinates of the pixel points in the pixel data of the gaze point in the texture coordinate system of the gaze point, and the multiple texture coordinates of the gaze point can be pre-set, and there is a matching relationship between the three-dimensional frame data corresponding to the three-dimensional frame model of the gaze point and the pixel data of the gaze point in the preset character array.

[0053] In some embodiments, creating a three-dimensional framework model of a gaze point may include: creating a three-dimensional framework model of a gaze point according to preset three-dimensional framework data. The preset three-dimensional framework data includes a plurality of edges and a connection relationship and a position relationship between the edges, and there is a matching relationship between the preset three-dimensional framework data and the pixel data of the gaze point in the preset character array, that is, when the pixel data of the gaze point in the preset character array is updated, the preset three-dimensional framework data of the gaze point needs to be updated, and similarly, when the preset three-dimensional framework data of the gaze point is updated, the pixel data of the gaze point in the preset character array also needs to be updated.

[0054] In some embodiments, the head-mounted display device stores a plurality of different preset three-dimensional framework data, and creating a three-dimensional framework model of the gaze point may include: obtaining the identity information of the user wearing the head-mounted display device, and determining the preset three-dimensional framework data matching the identity information as the target three-dimensional framework data; and creating a three-dimensional framework model of the gaze point according to the target three-dimensional framework data. This embodiment adaptively selects the corresponding three-dimensional framework data to create a three-dimensional framework model of the gaze point according to the user using the head-mounted display device, so that the three-dimensional image of the gaze point generated by subsequently rendering the three-dimensional framework model according to the texture data can be more in line with the user's preferences, realizing personalized rendering of the gaze point and improving the user experience.

[0055] In some embodiments, determining the preset three-dimensional framework data matching the identity information as the target three-dimensional framework data may include: obtaining a pre-established correspondence table between the identity information and the preset three-dimensional framework data, obtaining the preset three-dimensional framework data corresponding to the identity information of the user currently wearing the head mounted display device from the relationship table, and determining the obtained preset three-dimensional framework data as the target three-dimensional framework data. The identity information may be iris information, fingerprint information, etc.

[0056] For example, the correspondence table between identity identification information and preset three-dimensional framework data can be established in the following manner: when using a head-mounted display device, the user can select a favorite pattern of a gaze point (which can be a two-dimensional pattern or a three-dimensional pattern), and then the head-mounted display device collects the user's identity identification information (for example, the user's iris information), and binds the preset three-dimensional framework data corresponding to the pattern of the gaze point selected by the user with the collected identity identification information to establish a correspondence between the identity identification information and the preset three-dimensional framework data, and stores the correspondence between the identity identification information and the preset three-dimensional framework data in a data table to obtain a correspondence table between the identity identification information and the preset three-dimensional framework data.

[0057] Sub-step S1022: assigning multiple texture coordinates to multiple vertices in the three-dimensional framework model, with one vertex corresponding to one texture coordinate.

[0058] For example, the multiple texture coordinates of the gaze point include a first texture coordinate of the upper left vertex of the gaze point, a second texture coordinate of the lower left vertex, a third texture coordinate of the lower right vertex, and a fourth texture coordinate of the upper right vertex. Then, the upper left vertex, lower left vertex, lower right vertex, and upper right vertex of the three-dimensional frame model can be determined in the three-dimensional frame model, and the first texture coordinate can be assigned to the upper left vertex of the three-dimensional frame model, the second texture coordinate can be assigned to the lower left vertex of the three-dimensional frame model, the third texture coordinate can be assigned to the lower right vertex of the three-dimensional frame model, and the fourth texture coordinate can be assigned to the upper right vertex of the three-dimensional frame model.

[0059] Sub-step S1023, rendering the three-dimensional framework model according to the texture coordinates of each vertex and the texture data in the buffer proxy object to generate a target image corresponding to the gaze point.

[0060] In this embodiment, after each of the multiple vertices in the three-dimensional framework model is configured with texture coordinates, the correspondence between the vertex coordinates and the texture coordinates can be obtained, so that the GPU in the head-mounted display device can read the texture data from the buffer proxy object based on the texture coordinates of each vertex and the correspondence between the vertex coordinates and the texture coordinates to render the three-dimensional framework model, thereby obtaining a target image corresponding to the gaze point, and the target image is a three-dimensional image. In this embodiment, a three-dimensional image of the gaze point is obtained by rendering, so that after the three-dimensional image of the gaze point is displayed on the display screen, a better visual experience can be brought to the user, and the user experience is improved.

[0061] In some embodiments, Figure 4 As shown, after step S102, the following steps are also included:

[0062] Step S103: obtaining the current position of the gaze point of the user wearing the head mounted display device in the display screen of the head mounted display device, and displaying the target image at the current position.

[0063] In this embodiment, the gaze point detection device in the head mounted display device can be used to obtain the current position of the gaze point of the user wearing the head mounted display device in the display screen of the head mounted display device, and then the target image of the gaze point is displayed at the current position of the gaze point in the display screen, so that the user can read the target image of the gaze point in the head mounted display device, which is convenient for the user to know the display area that he is paying attention to. In other embodiments, the current position of the gaze point in the display screen of the head mounted display device can also be determined by eye tracking, which is not specifically limited in the embodiment of the present invention.

[0064] In some embodiments, Figure 5 As shown, after step S103, the following steps are also included:

[0065] Step S104: Acquire posture data of the head mounted display device, and determine the current head movement of the user wearing the head mounted display device according to the posture data.

[0066] In this embodiment, the posture data of the head mounted display device can be collected by the inertial measurement unit of the head mounted display device itself, and the current head movement of the user wearing the head mounted display device can include any one of turning the head to the left, turning the head to the right, turning the head and returning, shaking the head, raising the head, and nodding, and turning the head and returning can include any one of turning the head to the left and returning, turning the head to the right and returning, raising the head and returning, and nodding and returning. Among them, the posture data of the head mounted display device can include the acceleration, angular velocity, and orientation information of the head mounted display device.

[0067] For example, Figure 6As shown, a three-axis coordinate system is constructed with the user's head 11 as the coordinate origin, and the three-axis coordinate system takes the positive direction of the Y axis directly in front of the user's head 11, the positive direction of the Z axis directly above the user's head 11, and the positive direction of the X axis directly to the right of the user's head 11. The acceleration of the head-mounted display device 100 may include the acceleration of the head-mounted display device 100 in the X-axis direction, the acceleration in the Y-axis direction, and / or the acceleration in the Z-axis direction, and the angular velocity of the head-mounted display device 100 may include the angular velocity of the head-mounted display device 100 around the X-axis, Y-axis, and / or Z-axis.

[0068] In some embodiments, the posture data of the head-mounted display device includes the first angular velocity of the head-mounted display device, and determining the current head movement of the user wearing the head-mounted display device according to the posture data may include: when the absolute value and / or modulus of the first angular velocity is greater than or equal to a preset first threshold, performing Euler integration on the first angular velocity to obtain a first rotation angle of the user's head; and determining the current head movement of the user according to the first rotation angle of the user's head. Wherein, when the first angular velocity is less than the first threshold, the first angular velocity is discarded, and the first threshold can be set based on actual conditions, and the embodiment of the present invention does not specifically limit this. This embodiment can reduce user misoperation by identifying the user's current head movement only when the first angular velocity of the head-mounted display device is greater than or equal to a set value, and discarding the first angular velocity when the first angular velocity is less than the first threshold.

[0069] In some embodiments, according to the first rotation angle of the user's head, determining the user's current head movement may include: when the first rotation angle of the user's head is the rotation angle of the user's head around the Z axis, if the first rotation angle is greater than zero, then determining the user's current head movement is to turn right, if the first rotation angle is less than zero, then determining the user's current head movement is to turn left. Or when the first rotation angle of the user's head is the rotation angle of the user's head around the X axis, if the first rotation angle is greater than zero, then determining the user's current head movement is to raise the head, if the first rotation angle is less than zero, then determining the user's current head movement is to lower the head.

[0070] In some embodiments, the posture data of the head mounted display device includes a first angular velocity and a second angular velocity of the head mounted display device, the first angular velocity is in opposite directions to the second angular velocity, and determining the current head movement of the user wearing the head mounted display device according to the posture data may include: when the absolute value and / or modulus of the first angular velocity and the second angular velocity are both greater than or equal to a preset first threshold, performing Euler integration on the first angular velocity to obtain a first rotation angle of the user's head; when the absolute value of the first rotation angle is greater than a preset second threshold or is within a preset numerical range, performing Euler integration on the second angular velocity to obtain a second rotation angle of the user's head; when the absolute value of the difference between the first rotation angle and the second rotation angle is less than a preset third threshold, according to the first rotation angle and the second rotation angle, determining that the current head movement of the user wearing the head mounted display device is turning the head and returning. Among them, the second threshold, the third threshold and the preset numerical range can be set based on actual conditions, and the embodiment of the present invention does not specifically limit this. The head movement determination method provided in this embodiment can reduce user misoperation.

[0071] In some embodiments, determining, based on the first rotation angle and the second rotation angle, that the current head movement of the user wearing the head-mounted display device is turning the head and returning may include: if the first rotation angle and the second rotation angle are both rotation angles of the user's head around the Z-axis, then when the first rotation angle is less than zero and the second rotation angle is greater than zero, determining that the current head movement of the user wearing the head-mounted display device is turning the head to the left and returning; when the first rotation angle is greater than zero and the second rotation angle is less than zero, determining that the current head movement of the user wearing the head-mounted display device is turning the head to the right and returning; if the first rotation angle and the second rotation angle are both rotation angles of the user's head around the X-axis, then when the first rotation angle is less than zero and the second rotation angle is greater than zero, determining that the current head movement of the user wearing the head-mounted display device is nodding and returning; when the first rotation angle is greater than zero and the second rotation angle is less than zero, determining that the current head movement of the user wearing the head-mounted display device is raising the head and returning.

[0072] Step S105: when it is determined that the current head movement is the same as any preset head movement in the preset head movement set, determine a target character array from a plurality of preset character arrays according to the current head movement.

[0073] In this embodiment, the preset head action set may include turning the head to the left, turning the head to the right, shaking the head, raising the head, lowering the head, and nodding the head, etc. When it is determined that the current head action is the same as any preset head action in the preset head action set, it can be determined that the user wants to switch the display style of the gaze point.

[0074] In some embodiments, according to the current head movement, determining the target character array from multiple preset character arrays may include: obtaining an array identifier that matches the current head movement as the target array identifier from a pre-stored mapping relationship table, the mapping relationship table being used to describe the mapping relationship between the head movement and the array identifier; and selecting a preset character array corresponding to the target array identifier from multiple preset character arrays as the target character array. Among them, the mapping relationship table used to describe the mapping relationship between the head movement and the array identifier is pre-established, and the mapping relationship between the head movement and the array identifier can be set based on actual conditions, which is not specifically limited in the embodiments of the present invention.

[0075] For example, the head mounted display device stores a preset character array A, a preset character array B, a preset character array C, and a preset character array D. Table 1 is a mapping relationship table for describing the mapping relationship between head movements and array identifiers.

[0076] Table 1

[0077] Head movements Array Identification Turn left Preset array identifier of character array A Turn right Preset array identifier of character array B nod Preset array identifier of character array C Shake your head Array identifier of the preset character array D

[0078] For example, when the user's current head movement is turning left, the array identifier of the preset character array A can be used as the target array identifier, and the preset character array A can be determined as the target character array. For another example, when the user's current head movement is turning right, the array identifier of the preset character array B can be used as the target array identifier, and the preset character array B can be determined as the target character array. For another example, when the user's current head movement is nodding, the array identifier of the preset character array C can be used as the target array identifier, and the preset character array C can be determined as the target character array. For another example, when the user's current head movement is shaking his head, the array identifier of the preset character array D can be used as the target array identifier, and the preset character array D can be determined as the target character array.

[0079] Step S106: transmitting the pixel data of the gaze point in the target character array as texture data to the buffer proxy object for storage, so as to update the texture data in the buffer proxy object.

[0080] In this embodiment, after updating the texture data in the buffer proxy object, the execution of steps S102 to S103 can be returned to change the target image corresponding to the gaze point displayed on the display screen to achieve switching of the display style of the gaze point, thereby improving the user experience.

[0081] For example, when the buffer proxy object is created, the head mounted display device transfers the pixel data pixel_data_C of the gaze point in the preset character array C as texture data to the buffer proxy object for storage. At this time, the texture data in the buffer proxy object can be recorded as Texture_data_C. Then, the head mounted display device renders and generates the target image image_C corresponding to the gaze point according to the texture data Texture_data_C in the buffer proxy object, and displays the target image image_C at the current position of the user's gaze point in the display screen. Then, if the user wants to switch the display style of the gaze point, he can change his head movement. If the head mounted display device detects that the user's current head movement is to turn right, the pixel data pixel_data_D in the preset character array B can be transferred as texture data to the buffer proxy object for storage. At this time, the texture data in the buffer proxy object can be recorded as Texture_data_D. Then, the head mounted display device renders and generates the target image image_D corresponding to the gaze point according to the texture data Texture_data_D in the buffer proxy object, and displays the target image image_D at the current position of the user's gaze point in the display screen, thereby realizing the switching of the display style of the gaze point.

[0082] In some embodiments, after step S104, it also includes: when it is determined that the current head movement is different from any preset head movement in the preset head movement set, it is determined that the user does not need to switch the display style of the gaze point. At this time, the process can return to step S103, that is, obtaining the current position of the gaze point of the user wearing the head-mounted display device in the display screen of the head-mounted display device, and displaying the target image at the current position, so that the position of the target image corresponding to the gaze point can dynamically change with the change of the position of the user's gaze point, thereby improving the user experience.

[0083] See also Figure 7 , Figure 7 It is a schematic block diagram of the structure of a foveated rendering device provided by an embodiment of the present invention.

[0084] like Figure 6 As shown, the foveated rendering device 200 includes:

[0085] The data processing module 210 is configured to transmit the pixel data of the gaze point in the preset character array as texture data to the buffer proxy object for storage;

[0086] The image rendering module 220 is configured to render and generate a target image corresponding to the gaze point according to the texture data in the buffer proxy object.

[0087] In some embodiments, the image rendering module 220 includes:

[0088] A model creation submodule, configured to create a three-dimensional framework model of the gaze point;

[0089] An acquisition submodule, configured to acquire a plurality of texture coordinates of the gaze point;

[0090] A coordinate allocation submodule is configured to allocate a plurality of the texture coordinates to a plurality of vertices in the three-dimensional framework model, wherein one vertex corresponds to one texture coordinate;

[0091] The model rendering submodule is configured to render the three-dimensional framework model according to the texture coordinates of each vertex and the texture data in the buffer proxy object, and generate a target image corresponding to the gaze point.

[0092] In some embodiments, the head mounted display device stores a plurality of different preset three-dimensional framework data, and the model creation submodule is further configured to:

[0093] Acquire identity information of a user wearing the head mounted display device, and determine the preset three-dimensional framework data matching the identity information as target three-dimensional framework data;

[0094] A three-dimensional framework model of the gaze point is created according to the target three-dimensional framework data.

[0095] In some embodiments, the head mounted display device stores a plurality of preset character arrays, and the pixel data of the gaze points in different preset character arrays are different. The data processing module 210 is further configured to:

[0096] Acquire identity information of a user wearing the head mounted display device, and determine the preset character array matching the identity information as a target character array;

[0097] The pixel data of the gaze point in the target character array is transferred as texture data to the buffer proxy object for storage.

[0098] In some embodiments, the foveated rendering apparatus 200 further includes:

[0099] The position acquisition module is configured to acquire the current position of the gaze point of the user wearing the head-mounted display device in the display screen of the head-mounted display device, and display the target image at the current position.

[0100] In some embodiments, the head mounted display device stores a plurality of preset character arrays, and the pixel data of the gaze points in different preset character arrays are different. The gaze point rendering apparatus 200 further includes:

[0101] a head action recognition module, configured to obtain posture data of the head mounted display device, and determine a current head action of a user wearing the head mounted display device according to the posture data;

[0102] an array determination module, configured to determine a target character array from a plurality of preset character arrays according to the current head action when it is determined that the current head action is the same as any preset head action in the preset head action set;

[0103] A texture data updating module is configured to transmit the pixel data of the fixation point in the target character array as texture data to the buffer proxy object for storage, so as to update the texture data in the buffer proxy object;

[0104] By means of the image rendering module 220, a target image corresponding to the gaze point is rendered and generated according to the texture data in the buffer proxy object; and

[0105] The position acquisition module is used to acquire the current position of the gaze point of the user wearing the head mounted display device in the display screen of the head mounted display device, and the target image is displayed at the current position.

[0106] In some embodiments, the array determination module is further configured to:

[0107] Acquire an array identifier that matches the current head action as a target array identifier from a pre-stored mapping relationship table, wherein the mapping relationship table is used to describe a mapping relationship between the head action and the array identifier;

[0108] The preset character array corresponding to the target array identifier is selected from the plurality of preset character arrays as the target character array.

[0109] It should be noted that, those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the foveated rendering device described above can refer to the corresponding process in the aforementioned foveated rendering method embodiment, and will not be repeated here.

[0110] See also Figure 8 , Figure 8 It is a schematic block diagram of the structure of a head mounted display device provided by an embodiment of the present invention.

[0111] like Figure 8 As shown, the head mounted display device 100 includes a processor 110 and a memory 120, and the processor 110 and the memory 120 are connected via a bus 130, such as an I2C (Inter-integrated Circuit) bus.

[0112] Specifically, the processor 110 is used to provide computing and control capabilities to support the operation of the entire head-mounted display device. The processor 110 can be a central processing unit (CPU), and the processor 110 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0113] Specifically, the memory 120 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0114] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a partial structure related to the embodiment of the present invention, and does not constitute a limitation on the head-mounted display device to which the embodiment of the present invention is applied. The specific head-mounted display device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0115] The processor 110 is used to run a computer program stored in the memory 120, and implement any one of the foveated rendering methods provided by the embodiments of the present invention when executing the computer program.

[0116] In one embodiment, the processor 110 is used to run a computer program stored in the memory, and implement the following steps when executing the computer program:

[0117] The pixel data of the gaze point in the preset character array is transferred as texture data to the buffer proxy object for storage;

[0118] A target image corresponding to the gaze point is generated by rendering according to the texture data in the buffer proxy object.

[0119] In one embodiment, when rendering and generating a target image corresponding to the gaze point according to the texture data in the buffer proxy object, the processor 110 is configured to implement:

[0120] Creating a three-dimensional framework model of the gaze point and obtaining a plurality of texture coordinates of the gaze point;

[0121] Allocating a plurality of the texture coordinates to a plurality of vertices in the three-dimensional framework model, one vertex corresponding to one texture coordinate;

[0122] The three-dimensional framework model is rendered according to the texture coordinates of each vertex and the texture data in the buffer proxy object to generate a target map corresponding to the gaze point.

[0123] In one embodiment, the head mounted display device stores a plurality of different preset three-dimensional framework data, and the processor 110 is used to implement the following when creating the three-dimensional framework model of the gaze point:

[0124] Acquire identity information of a user wearing the head mounted display device, and determine the preset three-dimensional framework data matching the identity information as target three-dimensional framework data;

[0125] A three-dimensional framework model of the gaze point is created according to the target three-dimensional framework data.

[0126] In one embodiment, the head mounted display device stores a plurality of preset character arrays, and the pixel data of the gaze point in different preset character arrays are different. When the processor 110 transmits the pixel data of the gaze point in the preset character array as texture data to the buffer proxy object for storage, it is used to implement:

[0127] Acquire identity information of a user wearing the head mounted display device, and determine the preset character array matching the identity information as a target character array;

[0128] The pixel data of the gaze point in the target character array is transferred as texture data to the buffer proxy object for storage.

[0129] In one embodiment, after rendering and generating a target image corresponding to the gaze point according to the texture data in the buffer proxy object, the processor 110 is further configured to implement:

[0130] The current position of a gaze point of a user wearing the head mounted display device in a display screen of the head mounted display device is acquired, and the target image is displayed at the current position.

[0131] In one embodiment, the head mounted display device stores a plurality of preset character arrays, and the pixel data of the gaze points in different preset character arrays are different. After the processor 110 realizes displaying the target image at the current position, it is further configured to realize:

[0132] Acquire posture data of the head mounted display device, and determine a current head movement of a user wearing the head mounted display device according to the posture data;

[0133] When it is determined that the current head movement is the same as any preset head movement in the preset head movement set, determining a target character array from the plurality of preset character arrays according to the current head movement;

[0134] The pixel data of the gaze point in the target character array is transmitted as texture data to the buffer proxy object for storage, so as to update the texture data in the buffer proxy object;

[0135] Return to the step of rendering and generating a target image corresponding to the gaze point according to the texture data in the buffer proxy object.

[0136] In one embodiment, when determining the target character array from the plurality of preset character arrays according to the current head movement, the processor 110 is configured to implement:

[0137] Acquire an array identifier that matches the current head action as a target array identifier from a pre-stored mapping relationship table, wherein the mapping relationship table is used to describe a mapping relationship between the head action and the array identifier;

[0138] The preset character array corresponding to the target array identifier is selected from the plurality of preset character arrays as the target character array.

[0139] It should be noted that technicians in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of the head-mounted display device described above can refer to the corresponding process in the aforementioned foveation rendering method embodiment, and will not be repeated here.

[0140] An embodiment of the present invention further provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement any of the foveated rendering methods provided in the description of the embodiment of the present invention.

[0141] The storage medium may be an internal storage unit of the head mounted display device described in the foregoing embodiment, such as a hard disk or memory of the head mounted display device. The storage medium may also be an external storage device of the head mounted display device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc., provided on the head mounted display device.

[0142] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0143] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0144] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A foveated rendering method, characterized in that: Applied to a head mounted display device, the method comprises: The pixel data of the gaze point in the preset character array is transferred as texture data to the buffer proxy object for storage; A target image corresponding to the gaze point is generated by rendering according to the texture data in the buffer proxy object.

2. The foveated rendering method according to claim 1, characterized in that: The rendering and generating a target image corresponding to the gaze point according to the texture data in the buffer proxy object includes: Creating a three-dimensional framework model of the gaze point and obtaining a plurality of texture coordinates of the gaze point; Allocating a plurality of the texture coordinates to a plurality of vertices in the three-dimensional framework model, one vertex corresponding to one texture coordinate; The three-dimensional framework model is rendered according to the texture coordinates of each vertex and the texture data in the buffer proxy object to generate a target image corresponding to the gaze point.

3. The foveated rendering method according to claim 2, characterized in that: The head mounted display device stores a plurality of different preset three-dimensional framework data, and the three-dimensional framework model of the gaze point is created, including: Acquire identity information of a user wearing the head mounted display device, and determine the preset three-dimensional framework data matching the identity information as target three-dimensional framework data; A three-dimensional framework model of the gaze point is created according to the target three-dimensional framework data.

4. The foveated rendering method according to claim 1, characterized in that: The head mounted display device stores a plurality of preset character arrays, and the pixel data of the gaze points in different preset character arrays are different. The pixel data of the gaze points in the preset character arrays are transmitted as texture data to the buffer proxy object for storage, including: Acquire identity information of a user wearing the head mounted display device, and determine the preset character array matching the identity information as a target character array; The pixel data of the gaze point in the target character array is transferred as texture data to the buffer proxy object for storage.

5. The foveated rendering method according to any one of claims 1 to 4, characterized in that: After rendering and generating the target image corresponding to the gaze point according to the texture data in the buffer proxy object, the method further includes: The current position of a gaze point of a user wearing the head mounted display device in a display screen of the head mounted display device is acquired, and the target image is displayed at the current position.

6. The foveated rendering method according to claim 5, characterized in that: The head mounted display device stores a plurality of preset character arrays, and the pixel data of the gaze points in different preset character arrays are different. After displaying the target image at the current position, the head mounted display device further includes: Acquire posture data of the head mounted display device, and determine a current head movement of a user wearing the head mounted display device according to the posture data; When it is determined that the current head movement is the same as any preset head movement in the preset head movement set, determining a target character array from the plurality of preset character arrays according to the current head movement; The pixel data of the gaze point in the target character array is transmitted as texture data to the buffer proxy object for storage, so as to update the texture data in the buffer proxy object; Return to the step of rendering and generating a target image corresponding to the gaze point according to the texture data in the buffer proxy object.

7. The foveated rendering method according to claim 6, characterized in that: The step of determining a target character array from a plurality of preset character arrays according to the current head movement comprises: Acquire an array identifier that matches the current head action as a target array identifier from a pre-stored mapping relationship table, wherein the mapping relationship table is used to describe a mapping relationship between the head action and the array identifier; The preset character array corresponding to the target array identifier is selected from the plurality of preset character arrays as the target character array.

8. A foveated rendering device, characterized in that: Applied to a head mounted display device, the gaze point rendering device comprises: A data processing module is configured to transmit pixel data of a fixation point in a preset character array as texture data to the buffer proxy object for storage; The image rendering module is configured to render and generate a target image corresponding to the gaze point according to the texture data in the buffer proxy object.

9. A head mounted display device, characterized in that: The head-mounted display device includes a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the foveated point rendering method described in any one of claims 1 to 7 is implemented.

10. A storage medium for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the foveated rendering method described in any one of claims 1 to 7.

Images