Virtual reality image rendering method, storage medium and virtual reality device

Abstract

The present application discloses a method for rendering a virtual reality image, a storage medium, and a virtual reality device, the method comprising obtaining first posture information, determining a texture image based on the first posture information; obtaining second posture information when determining that the texture image is obtained, and performing position transformation on a number of target image points in the texture image based on the second posture information to obtain a target texture image; and rendering the target texture image to obtain a display image. The present application obtains the second posture information of the user when the texture image is obtained, and transforms the vertices of the texture image based on the second posture information to reproject the texture image, so that the reprojected texture image matches the second posture information, and when reprojecting the texture image, only the vertices of the texture image need to be transformed, thereby reducing the amount of calculation for reprojecting the texture image, improving the speed of reprojecting the texture image, and thus reducing the delay of the display screen.

Description

Technical Field

[0001] The present application relates to the field of virtual reality technology, and in particular to a virtual reality image rendering method, a storage medium, and a virtual reality device. Background Art

[0002] Virtual reality technology is a computer simulation mechanism that can create and experience a virtual world. It combines computer technology and display technology to construct a virtual environment that immerses users in the virtual environment, giving them a strong sense of immersion.

[0003] In order to correctly present the user's perception of virtual objects, the device display will display images at a certain frame rate (for example, 60Hz, 120Hz, etc.) during the image display process to ensure that the user's own movement is consistent with the image seen on the device. However, under the current hardware architecture, the image rendering capability of the graphics processor GPU (Graphics Processing Unit, referred to as GPU) is limited. When the GPU's image rendering capability is insufficient, it is easy to cause image display delay problems. Summary of the invention

[0004] The technical problem to be solved by the present application is to provide a virtual reality image rendering method, a storage medium and a virtual reality device in view of the deficiencies in the prior art.

[0005] In order to solve the above technical problems, the first aspect of the embodiments of the present application provides a method for rendering a virtual reality image, the method comprising:

[0006] Acquire first posture information, and determine a texture image based on the first posture information;

[0007] When it is determined that the texture image is obtained, second posture information is acquired, and based on the second posture information, a plurality of target image points in the texture image are positionally transformed to obtain a target texture image;

[0008] The target texture image is rendered to obtain a display image.

[0009] The method for rendering a virtual reality image is applied to AR glasses.

[0010] The method for rendering a virtual reality image, wherein the obtaining of first posture information and determining a texture image based on the first posture information specifically comprises:

[0011] Acquire first posture information, and determine a model matrix, an observation matrix, and a projection matrix based on the first posture information;

[0012] The current image frame is rendered based on the model matrix, the observation matrix, and the projection matrix to obtain a texture image.

[0013] In the method for rendering virtual reality images, the plurality of target image points include all image vertices in the texture image.

[0014] The method for rendering a virtual reality image, wherein the step of performing position transformation on a plurality of target image points in the texture image based on the second posture information to obtain a target texture image specifically comprises:

[0015] Obtaining a projection matrix and a disparity matrix corresponding to the texture image;

[0016] Determine a mapping matrix corresponding to the texture image based on the projection matrix, the disparity matrix and the second posture information;

[0017] Transforming the position of each target image point among the plurality of target image points based on the mapping matrix to obtain a transformed image point corresponding to each target image point;

[0018] A target texture image is determined based on the transformed image points corresponding to each target image point and the texture image.

[0019] The method for rendering a virtual reality image, wherein the step of determining a target texture image based on the transformed image points corresponding to each target image point and the texture image specifically comprises:

[0020] Replacing the position coordinates of each target image point in the texture image with the position coordinates of the corresponding changed image point;

[0021] Use the replaced texture image as the target texture image.

[0022] The method for rendering a virtual reality image, wherein the mapping matrix is ​​a homogeneous mapping matrix; the step of performing position transformation on each of a plurality of target image points based on the mapping matrix to obtain a transformed image point corresponding to each of the target image points specifically comprises:

[0023] For each target image point among the plurality of target image points, determining a position coordinate of the target image point, and converting the position coordinate into a homogeneous coordinate, wherein the homogeneous coordinate includes the position coordinate;

[0024] transforming the homogeneous coordinates based on the mapping matrix to obtain transformed position coordinates;

[0025] The target coordinates corresponding to the position coordinates of the target image point are selected from the transformed position coordinates to obtain the transformed image point corresponding to the target image point.

[0026] A second aspect of an embodiment of the present application provides a virtual reality image rendering device, the virtual reality image rendering device comprising:

[0027] An acquisition module, used for acquiring first posture information and determining a texture image based on the first posture information;

[0028] A transformation module, configured to obtain second posture information when determining to obtain a texture image, and based on the second posture information, perform position transformation on a plurality of target image points in the texture image to obtain a target texture image;

[0029] The rendering module is used to render the target texture image to obtain a display image.

[0030] A third aspect of an embodiment of the present application provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in any of the above-described methods for rendering virtual reality images.

[0031] A fourth aspect of the embodiments of the present application provides a virtual reality device, comprising: a processor, a memory, and a communication bus; the memory stores a computer-readable program that can be executed by the processor;

[0032] The communication bus realizes the connection and communication between the processor and the memory;

[0033] When the processor executes the computer-readable program, the processor implements the steps in any of the above-mentioned methods for rendering virtual reality images.

[0034] Beneficial effects: Compared with the prior art, the present application provides a virtual reality image rendering method, storage medium and virtual reality device, the virtual reality image rendering method comprising obtaining first posture information, and determining a texture image based on the first posture information; obtaining second posture information when the texture image is determined to be obtained, and performing position transformation on a number of target image points in the texture image based on the second posture information to obtain a target texture image; and rendering the target texture image to obtain a display image. The present application obtains the second posture information of the user when the texture image is obtained, and transforms the vertices of the texture image based on the second posture information to reproject the texture image, so that the reprojected texture image matches the second posture information, and when reprojecting the texture image, only the vertices of the texture image need to be transformed, thereby reducing the amount of calculation for reprojecting the texture image, improving the speed of reprojecting the texture image, and thus reducing the delay of the display screen. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 A schematic diagram of a texture image in the virtual reality image rendering method provided in this application.

[0037] Figure 2 A schematic diagram of a texture image desired to be displayed in the virtual reality image rendering method provided in the present application.

[0038] Figure 3 A flowchart of a method for rendering a virtual reality image provided in this application.

[0039] Figure 4 This is an example diagram of the acquisition time of the motion sensor in the virtual reality image rendering method provided in the present application.

[0040] Figure 5 A schematic diagram of the coordinate system change process in the virtual reality image rendering method provided in this application.

[0041] Figure 6 This is a structural principle diagram of the virtual reality image rendering device provided in this application.

[0042] Figure 7 This is a schematic diagram of the structure of the virtual reality device provided in this application. DETAILED DESCRIPTION

[0043] The present application provides a method for rendering a virtual reality image, a storage medium, and a virtual reality device. To make the purpose, technical solution, and effect of the present application clearer and more explicit, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0044] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0045] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of the ordinary technicians in the field to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have a meaning consistent with the meaning in the context of the prior art, and unless specifically defined as here, will not be interpreted with an idealized or overly formal meaning. In addition, it should be understood that the sequence number and size of each step in the present embodiment do not mean the order of execution, and the execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0046] After research, the inventors found that virtual reality technology is a computer simulation mechanism that can create and experience a virtual world. It combines computer technology and display technology to construct a virtual environment that allows users to immerse themselves in the virtual environment and has a strong sense of immersion. In order to correctly present the user's perception of virtual objects, the device display will display images at a certain frame rate (for example, 60Hz, 120Hz, etc.) during the image display process to ensure that the user's own movement is consistent with the picture seen on the device. However, under the current hardware architecture, the image rendering capability of the graphics processor GPU (Graphics Processing Unit, referred to as GPU) is limited. When the GPU's image rendering capability is insufficient, it is easy to cause image display delays. For example, the texture image rendered is Figure 1 As shown in the figure, after the rendering is completed, the user's head posture changes, and it is expected that the image rendered on the AR glasses will no longer be Figure 1 Instead of the texture image shown Figure 2 The vertex coordinate values ​​shown become (-0.9, -1), (0.9, -0.8), (-0.8, 1), (0.8, 0.7) texture images, while the display device of the AR glasses shows Figure 1 The texture image shown, resulting in image display delay.

[0047] In order to solve the delay problem, the commonly used Timewarp method samples the latest data from the motion sensor to distort the last rendered image sent to the device display to regenerate a new frame of rendered image to reduce delay and effectively alleviate the problem of image jitter. However, due to the existence of optical distortion of AR devices, the traditional Timewarp method usually performs homography transformation on texture coordinates, and vertex coordinates are used for distortion correction. This will increase the amount of calculation required for the new rendered image and increase the rendering overhead. However, the optical solutions used by the AR (Augmented Reality) glasses currently in common use mainly include Birdbath, optical waveguides, free-form surfaces and prisms, and the field of view (FOV) of these optical solutions on AR glasses is generally small, and the smaller field of view FOV will make the radial distortion correspondingly smaller so that it can be ignored.

[0048] Based on this, in an embodiment of the present application, first posture information is obtained, and a texture image is determined based on the first posture information; when it is determined that a texture image is obtained, second posture information is obtained, and based on the second posture information, a plurality of target image points in the texture image are transformed in position to obtain a target texture image; the target texture image is rendered to obtain a display image. The present application obtains the second posture information of the user when the texture image is obtained, and transforms the vertices of the texture image based on the second posture information to reproject the texture image, so that the reprojected texture image matches the second posture information, and when reprojecting the texture image, only the vertices of the texture image need to be transformed, which reduces the amount of calculation for reprojecting the texture image, improves the speed of reprojecting the texture image, and thus reduces the delay of the display screen.

[0049] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.

[0050] This embodiment provides a method for rendering a virtual reality image, the application scenario of the method for rendering a virtual reality image may be AR (Augmented Reality) glasses, and the AR glasses may have a smaller field of view, for example, the AR glasses are Magic Leap One, whose field of view is 50°, or the AR glasses are HoloLens2, whose field of view is 52°, or the AR glasses are Nreal Light, whose field of view is 52°, etc. Among them, the GPU configured in the AR glasses will render a texture image according to the 3DoF / 6DoF data (i.e., the first posture information) sampled by the motion sensor sensor, and after obtaining the texture image, the 3DoF / 6DoF data (i.e., the second posture information) of the motion sensor is collected again, and the rendered texture image is reprojected based on the second posture information, without repeated rendering, so that the display image data can be obtained in this way, which can effectively reduce the delay.

[0051] like Figure 3 As shown, the virtual reality image rendering method provided in this embodiment includes:

[0052] S10: Acquire first posture information, and determine a texture image based on the first posture information.

[0053] Specifically, only when the user wears AR glasses on his head can he watch the virtual scene played in the AR glasses. The AR glasses will change the playing scene according to the movement of the user's head, so that the motion sensor configured by the AR glasses will collect the posture information of the user's head in real time, so as to determine the scene picture to be displayed based on the posture information. Therefore, the first posture information is used for the posture information of the user of the AR glasses, and the first posture information can be collected by the motion sensor configured by the AR glasses. The motion sensor can be a 9-axis motion sensor, and the sampling frequency of the motion sensor in the AR device can reach 800Hz, which is higher than the refresh frame rate of the display component of the AR glasses, and within the refresh cycle of a frame, the motion sensor can perform multiple motion data acquisition, such as Figure 4 As shown, the sensing data collected by the motion sensor can be obtained at three different times within the refresh cycle of a frame. For example, time t1 is before time t0, and the delay time of t1 is reduced compared to the sampling data at time t0. Time t2 is before time t1, and the delay time of t2 is reduced compared to the sampling data at time t1. Therefore, in the process of rendering the current image frame, the first posture information and the second posture information can be collected by the motion sensor.

[0054] In addition, when the display system of AR glasses displays images, the CPU in the display system can receive image data generated by an image acquisition device (e.g., a camera, etc.), and the continuously acquired multiple frames of image data can be encoded to generate a video stream, which can be transmitted to the CPU in real time, or pre-stored in a storage device and then transmitted to the CPU. The CPU performs frame splicing and decoding on the received video data and transmits it to the GPU, which renders the received image data. The rendering is the process of converting a three-dimensional light energy transfer process into a two-dimensional image. Rendering usually uses a three-dimensional geometric model, through geometric transformation, projection transformation, perspective transformation and window clipping, and various ray tracing, radiosity and other algorithms in pursuit of three-dimensional realism. The image data after image rendering can be output to the display buffer of the display screen and displayed on the display screen.

[0055] Based on this, the acquisition time of the first posture information is the moment when the GPU receives the image data that needs to be rendered. In other words, when the GPU receives the image data that needs to be rendered, the sensor data collected by the motion sensor is obtained, and the obtained sensor data is used as the first posture information, wherein the acquisition time of the first posture information is before the rendered image is generated. In one implementation of this embodiment, the first posture information includes a rotation matrix R and a translation matrix T, wherein the rotation matrix R is used to reflect the rotation data of the user's head relative to the world coordinate system, and the translation matrix is ​​used to reflect the translation data of the user's head relative to the world coordinate system.

[0056] In an implementation of this embodiment, the acquiring the first posture information and determining the texture image based on the first posture information specifically includes:

[0057] Acquire first posture information, and determine a model matrix, an observation matrix, and a projection matrix based on the first posture information;

[0058] The current image frame is rendered based on the model matrix, the observation matrix, and the projection matrix to obtain a texture image.

[0059] Specifically, the texture image can be a two-dimensional image obtained by rendering the current image frame, wherein OpenGL can be used to render the current image when the current image frame is rendered. The model matrix is ​​used to convert the model coordinate system to the world coordinate system, and when the model coordinate system is converted to the world coordinate system, the coordinate origin of the model coordinate system is located at the coordinate origin of the world coordinate system. The observation transformation is used to transform from the world coordinate system to the observation coordinate system. In other words, an observation point (for example, camera position and camera direction) is set in the world coordinate system, and then the coordinates in the world coordinate system are transformed to the space where the observation point is located (for example, camera space). The projection matrix is ​​used to display a 3D graphics effect on a 2D screen, that is, to flatten the view seen by the eye onto the screen to obtain a texture image.

[0060] S20: When it is determined that a texture image is obtained, second posture information is acquired, and based on the second posture information, positions of a plurality of target image points in the texture image are transformed to obtain a target texture image.

[0061] Specifically, the second posture information is acquired through a motion sensor configured by the AR glasses, and is used to reflect the posture information of the user's posture, wherein the acquisition time of the second posture information is the time of rendering the texture image. It can be understood that the second posture information is acquired after the rendering of the current image frame is completed, and the acquisition time of the second posture information is later than the acquisition time of the first posture information. This is because the acquisition time of the first posture information is acquired before the rendering of the current image frame is completed, and the second posture information is acquired after the rendering of the current image frame is completed. In other words, the first posture information is used for the first rendering of the current image frame, and the second posture information is used for the second rendering of the texture image determined based on the first posture information, wherein the texture image is the texture image determined by the first rendering image of the current image frame. Among them, the second posture information includes a rotation matrix R1 and a translation matrix T1.

[0062] In one implementation of this embodiment, the plurality of target image points include all image vertices in the texture image. It can be understood that each of the plurality of target image points is an image vertex of the texture image, and each image vertex in the texture image is included in the plurality of target image points, that is, the plurality of target image points correspond one-to-one to all image vertices in the texture image. In other words, when transforming the texture image, this embodiment only needs to transform the image vertices of the texture image, which can reduce the amount of calculation required for reprojecting the texture image, thereby increasing the speed of reprojecting the texture image and solving the display delay problem.

[0063] In an implementation of this embodiment, the step of performing position transformation on a plurality of target image points in the texture image based on the second posture information to obtain a target texture image specifically includes:

[0064] Obtaining a projection matrix and a disparity matrix corresponding to the texture image;

[0065] Determine a mapping matrix corresponding to the texture image based on the projection matrix, the disparity matrix and the second posture information;

[0066] Transforming the position of each target image point among the plurality of target image points based on the mapping matrix to obtain a transformed image point corresponding to each target image point;

[0067] A target texture image is determined based on the transformed image points corresponding to each target image point and the texture image.

[0068] Specifically, the mapping matrix is ​​used to map the target image points in the texture image determined based on the first posture information to the transformed image points corresponding to the second posture information, wherein the mapping matrix is ​​determined based on the projection matrix and the parallax matrix. In an implementation of this embodiment, the mapping matrix may be:

[0069] Re=P·Te·dR·Te^·Pv

[0070] Wherein, Re represents the mapping matrix, P represents the projection matrix, Te represents the disparity matrix, dR represents the second posture information, Te^ represents the disparity matrix of the IPD disparity, and Pv represents the inverse matrix of the projection matrix P.

[0071] The process of determining the mapping matrix can be as follows: Figure 5 As shown, for each target image point h in the texture image, the target image point is an observation point v in the world coordinate system obtained by MVP transformation, so we can have:

[0072] h=P·V·M·v

[0073] Among them, M represents the model matrix, V represents the view matrix, P is the projection matrix, and the projection matrix P includes perspective division, that is, the last row of the projection matrix P is 0, 0, -1, 0, so as to skip the clipping space.

[0074] The time interval between the first posture information acquisition time and the second posture information acquisition time is the rendering time for the current image frame, which is generally short (for example, less than 0.5s). During this rendering time, the user's head movement is generally relatively small. Based on, the movement of the model itself can be ignored, so the model matrix can be the unit matrix I. Correspondingly, the correspondence between the target image point h and the observation point v can be expressed as:

[0075] h=P·V·v.

[0076] The observation matrix V is used to reflect the movement of the user's head, which is the change of the virtual camera posture. Therefore, the calculation formula for collecting the first posture information of the wearer's head movement can be:

[0077] V=Te·R·T

[0078] Among them, Te is the disparity matrix, R is the rotation matrix in the first posture information, and T is the translation matrix in the first posture information.

[0079] The disparity matrix is ​​used to adjust the disparity between binocular eyes, and the disparity matrix includes a left-eye disparity matrix and a right-eye disparity matrix, and the mapping matrix corresponding to the left eye is determined according to the left-eye disparity matrix, and the mapping matrix corresponding to the right eye is determined according to the right-eye disparity matrix. In other words, there will be mapping matrices for both the left eye and the right eye in binocular eyes, and when reprojecting the texture image, the texture image is transformed using the change matrices corresponding to the left eye and the right eye, respectively, to obtain the display images corresponding to the left eye and the right eye, respectively, and the display image corresponding to the left eye is displayed to the left eye, and the display image corresponding to the right eye is displayed to the right eye, so as to improve the display effect. In addition, the determination process of the change matrices corresponding to the left eye and the right eye, and the process of transforming the texture image based on the mapping matrix are the same, and the determination process of the transformation mapping corresponding to the left eye is taken as an example for explanation.

[0080] In this embodiment, the disparity matrix corresponding to the left eye and the disparity matrix corresponding to the right eye can be expressed as:

[0081] Te1=((1,0,0,0),(0,1,0,0),(0,0,1,0),(+IPD / 2,0,0,1))

[0082] Te2=((1,0,0,0),(0,1,0,0),(0,0,1,0),(-IPD / 2,0,0,1))

[0083] Among them, Te1 is used to represent the disparity matrix of the left target, Te2 is used to represent the disparity matrix of the right target, and the disparity matrix Te1 of the left target is referred to as the disparity matrix Te.

[0084] Furthermore, V = Te·R·T can be obtained, so the corresponding relationship between the target image point h and the observation point v can be expressed as:

[0085] h=P·Te·R·T·v

[0086] The depth information of the image point h is retained when acquiring the texture image, so that the spatial position v in the world coordinate system can be determined according to the texture regeneration process, wherein P, Te, R, T are known.

[0087] Based on this, the image points in the texture image can be converted into points in NDC coordinates, the image points in NDC coordinates are converted to the world coordinate system, and then the image points in the world coordinate system are converted to the NDC coordinate system based on the posture information to become texture image points, wherein the second posture information is used when converting the image points in the world coordinate system to the NDC coordinate system based on the posture information. However, in the process of converting from NDC coordinates to texture coordinates, the inverse matrix of the P matrix cannot be directly determined due to the loss of depth. Therefore, in this embodiment, the texture image is used as a virtual model and placed on the plane of z=-1, and the inverse Pv of the projection matrix P is calculated based on the projection matrix P. Of course, it should be noted that Pv processes points in the NDC coordinate system. After Pv is returned to the observation coordinate system, h=P·Te·R·T·v can be converted to:

[0088] Pv·h=Pv·P·Te·R·T·v

[0089] Then, the motion data obtained by the second sampling is added to the formula to calculate the new transformed image point coordinate h1, where the coordinate of the transformed image point coordinate h1 can be expressed as:

[0090] h1=P·Te·dR·Te^·Pv·P·Te·R·T·v

[0091] Among them, Te^ is the IPD parallax removed, and dR is the second posture information. In practical applications, there is no 6DoF translation on Archery, and dR can only include the rotation matrix. Of course, in the case of translation, only rotation can be considered. This is because the motion data is sampled at a small time interval, and the impact of rotation on the display result is much greater than translation. Then you need to multiply the parallax translation and projection matrices to switch the vertices to NDC coordinates again.

[0092] Finally, from h = P·Te·R·T·v, we can know that the coordinates of the transformed image point h1 can be expressed as:

[0093] h1=P·Te·dR·Te^·Pv·h

[0094] Therefore, the mapping matrix is ​​expressed as:

[0095] Re=P·Te·dR·Te^·Pv.

[0096] In an implementation of this embodiment, the mapping matrix is ​​a homogeneous mapping matrix; and performing position transformation on each of the plurality of target image points based on the mapping matrix to obtain a transformed image point corresponding to each of the target image points specifically includes:

[0097] For each target image point among the plurality of target image points, determining a position coordinate of the target image point, and converting the position coordinate into a homogeneous coordinate, wherein the homogeneous coordinate includes the position coordinate;

[0098] transforming the homogeneous coordinates based on the mapping matrix to obtain transformed position coordinates;

[0099] The target coordinates corresponding to the position coordinates of the target image point are selected from the transformed position coordinates to obtain the transformed image point corresponding to the target image point.

[0100] Specifically, the position coordinates of the target image point refer to the position coordinates of the target image point in the texture image coordinate system, which are recorded as (sx, sy). After obtaining the position coordinates of the target image point, the position coordinates are converted into four-dimensional homogeneous coordinates. For example, 0, 1 are added after the position coordinates to obtain homogeneous coordinates (sx, sy, 0, 1). After obtaining the homogeneous coordinates (sx, sy, 0, 1), the secondary coordinates (sx, sy, 0, 1) are aligned through the mapping matrix Re for homogeneous transformation to obtain the transformed position coordinates, where the transformed position coordinates can be expressed as:

[0101] (tx,ty,0,1)=Re(sx,sy,0,1)

[0102] Among them, the mapping matrix Re is a homogeneous transformation, which is a 4x4 matrix, (sx,sy,0,1) is the position coordinate of the target image point, and (tx,ty,0,1) is the transformed position coordinate corresponding to (sx,sy,0,1).

[0103] After obtaining the transformed position coordinates, select the value corresponding to the coordinate position of the position coordinate of the target image point to obtain the target coordinates, where the target coordinates are (tx, ty). In this embodiment, the target image point is a vertex of the texture image, and the texture image includes 4 vertices, namely (-1, -1), (1, -1), (-1, 1) and (1, 1), and the corresponding homogeneous coordinates are (-1, -1, 0, 1), (1, -1, 0, 1), (-1, 1, 0, 1), (1, 1, 0, 1).

[0104] In an implementation of this embodiment, determining the target texture image based on the transformed image points corresponding to each target image point and the texture image specifically includes:

[0105] Replacing the position coordinates of each target image point in the texture image with the position coordinates of the corresponding changed image point;

[0106] Use the replaced texture image as the target texture image.

[0107] Specifically, after obtaining the changed image point, the position coordinates of each target image point in the texture image are modified to the position coordinates of the corresponding changed image point. For example, if the position coordinates of the target image point are (1, 1) and the position coordinates of the changed image point corresponding to the target image point are (0.9, 0.9), then the position coordinates of the target image point are modified to (0.9, 0.9). After the position coordinates of each target image point are modified, the modified texture image is used as the target texture image for display.

[0108] S30: Rendering the target texture image to obtain a display image.

[0109] Specifically, after the target texture image is acquired, the target texture image is rendered to obtain a rendered image, and the obtained rendered image is output to a display buffer of a display screen and displayed through the display screen.

[0110] In summary, the present embodiment provides a method for rendering a virtual reality image, the method comprising obtaining first posture information, and determining a texture image based on the first posture information; obtaining second posture information when it is determined that the texture image is obtained, and based on the second posture information, performing position transformation on a number of target image points in the texture image to obtain a target texture image; and rendering the target texture image to obtain a display image. The present application obtains the second posture information of the user when the texture image is obtained, and transforms the vertices of the texture image based on the second posture information to reproject the texture image, so that the reprojected texture image matches the second posture information, and when reprojecting the texture image, only the vertices of the texture image need to be transformed, thereby reducing the amount of computation for reprojecting the texture image, improving the speed of reprojecting the texture image, and thus reducing the latency of the display screen.

[0111] Based on the above virtual reality image rendering method, this embodiment provides a virtual reality image rendering device, such as Figure 6 As shown, the rendering device of the virtual reality image includes:

[0112] An acquisition module 100, configured to acquire first posture information and determine a texture image based on the first posture information;

[0113] The transformation module 200 is used to obtain second posture information when it is determined that a texture image is obtained, and based on the second posture information, perform position transformation on a plurality of target image points in the texture image to obtain a target texture image;

[0114] The rendering module 300 is used to render the target texture image to obtain a display image.

[0115] In one embodiment, the rendering device of the virtual reality image is installed in AV glasses.

[0116] In one embodiment, the acquisition module is specifically used for:

[0117] Acquire first posture information, and determine a model matrix, an observation matrix, and a projection matrix based on the first posture information; and render the current image frame based on the model matrix, the observation matrix, and the projection matrix to obtain a texture image.

[0118] In one embodiment, the plurality of target image points include all image vertices in the texture image.

[0119] In one embodiment, the transformation module is specifically used to:

[0120] Obtain a projection matrix and a disparity matrix corresponding to the texture image; determine a mapping matrix corresponding to the texture image based on the projection matrix, the disparity matrix and the second posture information; perform a position transformation on each target image point among a plurality of target image points based on the mapping matrix to obtain a transformed image point corresponding to each target image point; and determine a target texture image based on the transformed image point corresponding to each target image point and the texture image.

[0121] In one embodiment, the transformation module is specifically used to:

[0122] The position coordinates of each target image point in the texture image are replaced with the position coordinates of the corresponding changed image points; and the replaced texture image is used as the target texture image.

[0123] In one embodiment, the mapping matrix is ​​a homogeneous mapping matrix; the transformation module is specifically used for:

[0124] For each target image point among a number of target image points, determine the position coordinates of the target image point and convert the position coordinates into homogeneous coordinates; transform the homogeneous coordinates based on the mapping matrix to obtain transformed position coordinates; and select the target coordinates corresponding to the position coordinates of the target image point in the transformed position coordinates to obtain the transformed image point corresponding to the target image point, wherein the homogeneous coordinates include the position coordinates.

[0125] In addition, it is worth mentioning that the working process of each functional module in the virtual reality image rendering device provided in this embodiment is the same as the working process of each step in the above-mentioned virtual reality image rendering method, which will not be repeated here. For details, please refer to the description of the above-mentioned virtual reality image rendering method.

[0126] Based on the above virtual reality image rendering method, this embodiment provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in the virtual reality image rendering method as described in the above embodiment.

[0127] Based on the above virtual reality image rendering method, the present application also provides a virtual reality device, such as Figure 7 As shown, it includes at least one processor (processor) 20; display screen 21; and memory (memory) 22, and may also include a communications interface (Communications Interface) 23 and a bus 24. Among them, the processor 20, the display screen 21, the memory 22 and the communication interface 23 can communicate with each other through the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setting mode. The communication interface 23 can transmit information. The processor 20 can call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0128] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0129] The memory 22 is a computer-readable storage medium that can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions or modules stored in the memory 22, that is, implementing the methods in the above embodiments.

[0130] The memory 22 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the virtual reality device, etc. In addition, the memory 22 may include a high-speed random access memory and may also include a non-volatile memory. For example, a variety of media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, may also be a transient storage medium.

[0131] In addition, the specific process of loading and executing the multiple instruction processors in the above-mentioned storage medium and the virtual reality device has been described in detail in the above-mentioned method, and will not be described one by one here.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for rendering a virtual reality image, characterized in that: The method comprises: Acquire first posture information, and determine a texture image based on the first posture information; When it is determined that the texture image is obtained, second posture information is acquired, and based on the second posture information, a plurality of target image points in the texture image are positionally transformed to obtain a target texture image; Rendering the target texture image to obtain a display image; The first posture information and the second posture information are obtained by collecting motion data multiple times by a motion sensor; The performing position transformation on a plurality of target image points in the texture image based on the second posture information to obtain a target texture image specifically includes: Obtaining a projection matrix and a disparity matrix corresponding to the texture image; Determine a mapping matrix corresponding to the texture image based on the projection matrix, the disparity matrix and the second posture information; Transforming the position of each target image point among the plurality of target image points based on the mapping matrix to obtain a transformed image point corresponding to each target image point; Determining a target texture image based on the transformed image points corresponding to each target image point and the texture image; The mapping matrix is ​​a homogeneous mapping matrix; the step of performing position transformation on each of the plurality of target image points based on the mapping matrix to obtain a transformed image point corresponding to each target image point specifically includes: For each target image point among the plurality of target image points, determining a position coordinate of the target image point, and converting the position coordinate into a homogeneous coordinate, wherein the homogeneous coordinate includes the position coordinate; transforming the homogeneous coordinates based on the mapping matrix to obtain transformed position coordinates; The target coordinates corresponding to the position coordinates of the target image point are selected from the transformed position coordinates to obtain the transformed image point corresponding to the target image point.

2. The method for rendering a virtual reality image according to claim 1, characterized in that: The virtual reality image rendering method is applied to AR glasses.

3. The method for rendering a virtual reality image according to claim 1, characterized in that: The acquiring of the first posture information and determining the texture image based on the first posture information specifically includes: Acquire first posture information, and determine a model matrix, an observation matrix, and a projection matrix based on the first posture information; The current image frame is rendered based on the model matrix, the observation matrix, and the projection matrix to obtain a texture image.

4. The method for rendering a virtual reality image according to claim 1, characterized in that: The plurality of target image points include all image vertices in the texture image.

5. The method for rendering a virtual reality image according to claim 1 or 4, characterized in that: The determining of the target texture image based on the transformed image points corresponding to the target image points and the texture image specifically includes: Replacing the position coordinates of each target image point in the texture image with the position coordinates of the corresponding changed image point; Use the replaced texture image as the target texture image.

6. A virtual reality image rendering device, characterized in that: The rendering device of the virtual reality image comprises: An acquisition module, used for acquiring first posture information and determining a texture image based on the first posture information; A transformation module, configured to obtain second posture information when determining to obtain a texture image, and based on the second posture information, perform position transformation on a plurality of target image points in the texture image to obtain a target texture image; A rendering module, used for rendering the target texture image to obtain a display image; The first posture information and the second posture information are obtained by collecting motion data multiple times by a motion sensor; The performing position transformation on a plurality of target image points in the texture image based on the second posture information to obtain a target texture image specifically includes: Obtaining a projection matrix and a disparity matrix corresponding to the texture image; Determine a mapping matrix corresponding to the texture image based on the projection matrix, the disparity matrix and the second posture information; Transforming the position of each target image point among the plurality of target image points based on the mapping matrix to obtain a transformed image point corresponding to each target image point; Determining a target texture image based on the transformed image points corresponding to each target image point and the texture image; The mapping matrix is ​​a homogeneous mapping matrix; the step of performing position transformation on each of the plurality of target image points based on the mapping matrix to obtain a transformed image point corresponding to each target image point specifically includes: For each target image point among the plurality of target image points, determining a position coordinate of the target image point, and converting the position coordinate into a homogeneous coordinate, wherein the homogeneous coordinate includes the position coordinate; transforming the homogeneous coordinates based on the mapping matrix to obtain transformed position coordinates; The target coordinates corresponding to the position coordinates of the target image point are selected from the transformed position coordinates to obtain the transformed image point corresponding to the target image point.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the method for rendering a virtual reality image as described in any one of claims 1 to 5.

8. A virtual reality device, characterized in that: include: Processor, memory and communication bus; The memory stores a computer-readable program executable by the processor; The communication bus realizes the connection and communication between the processor and the memory; When the processor executes the computer-readable program, the steps in the method for rendering a virtual reality image according to any one of claims 1 to 5 are implemented.

Images