Binocular rendering method and device, head-mounted display equipment and readable storage medium

Abstract

The invention discloses a binocular rendering method and device, head-mounted display equipment and a readable storage medium, and relates to the technical field of head-mounted display equipment, the binocular rendering method is applied to a graphics processor, and the binocular rendering method comprises the following steps: obtaining geometric data of a to-be-rendered three-dimensional model, and obtaining a left-eye view matrix and a right-eye view matrix of the head-mounted display equipment; storing the left-eye view matrix and the right-eye view matrix into a preset view matrix array to obtain a target view matrix array; performing image rendering on the to-be-rendered three-dimensional model according to the geometric data and the target view matrix array to obtain a left eye texture image and a right eye texture image; and writing the left-eye texture image and the right-eye texture image into a texture image array of the head-mounted display equipment so as to respectively display the left-eye texture image and the right-eye texture image on a left-eye screen and a right-eye screen of the head-mounted display equipment. According to the invention, the rendering duration of binocular rendering can be shortened, so that the rendering efficiency of binocular rendering is improved.

Description

Technical Field

[0001] This application relates to the field of head-mounted display technology, and in particular to a binocular rendering method, apparatus, head-mounted display device, and readable storage medium. Background Technology

[0002] In the rendering process of head-mounted display devices such as AR (Augmented Reality) glasses and VR (Virtual Reality) glasses, the GPU (Graphics Processing Unit) plays a central role in creating realistic stereoscopic visual effects, responsible for binocular rendering of the content in the virtual scene. Current binocular rendering processes typically require two cameras to image the same scene, i.e., setting different view matrices to render two slightly different images so that the human eye can perceive depth through visual differences.

[0003] However, currently, the rendering time for stereo rendering is usually quite long due to the impact of data transfer efficiency between the CPU (Central Processing Unit) and the GPU. Summary of the Invention

[0004] The main objective of this application is to provide a binocular rendering method, apparatus, head-mounted display device, and readable storage medium, with the aim of shortening the rendering time of binocular rendering and improving the rendering efficiency of binocular rendering.

[0005] To achieve the above objectives, this application provides a binocular rendering method applied to a graphics processor, the method comprising:

[0006] Obtain the geometric data of the 3D model to be rendered, and obtain the left and right eye view matrices of the head-mounted display device;

[0007] The left-eye view matrix and the right-eye view matrix are stored in a preset view matrix array to obtain the target view matrix array;

[0008] Based on the geometric data and the target view matrix array, the three-dimensional model to be rendered is rendered to obtain the left eye texture image and the right eye texture image;

[0009] The left-eye texture image and the right-eye texture image are written into the texture image array of the head-mounted display device, so as to display the left-eye texture image and the right-eye texture image on the left-eye screen and the right-eye screen of the head-mounted display device, respectively.

[0010] In one embodiment, before the step of storing the left-eye view matrix and the right-eye view matrix into a preset view matrix array to obtain the target view matrix array, the method further includes:

[0011] Verify whether the matrix dimensions of the left eye view matrix are consistent with those of the right eye view matrix;

[0012] If so, then the process of storing the left-eye view matrix and the right-eye view matrix into a preset view matrix array is executed to obtain the target view matrix array;

[0013] If not, then the left eye view matrix and the right eye view matrix are subjected to matrix dimension unification processing, and after the matrix dimension unification processing is completed, the step of storing the left eye view matrix and the right eye view matrix into a preset view matrix array to obtain the target view matrix array is executed.

[0014] In one embodiment, before the step of storing the left-eye view matrix and the right-eye view matrix into a preset view matrix array to obtain the target view matrix array, the method further includes:

[0015] Obtain the first space size occupied by the left eye view matrix in the central processing unit of the head-mounted display device, and the second space size occupied by the right eye view matrix in the central processing unit;

[0016] The storage space size of the view matrix array is determined based on the sum of the first space size and the second space size.

[0017] Create the view matrix array based on the storage space size.

[0018] In one embodiment, the step of determining the storage space size of the view matrix array based on the sum of the first space size and the second space size includes:

[0019] The space size and value are used as the storage space size of the view matrix array;

[0020] Alternatively, the storage space size and value can be added to a preset space size to obtain the storage space size of the view matrix array;

[0021] Alternatively, the storage space size and value can be multiplied by a preset coefficient value to obtain the storage space size of the view matrix array.

[0022] In one embodiment, the step of obtaining the left-eye view matrix and the right-eye view matrix of the head-mounted display device includes:

[0023] Acquire the pose data of the left eye camera and the pose data of the right eye camera of the head-mounted display device;

[0024] The left-eye view matrix and the right-eye view matrix are constructed based on the pose data of the left-eye camera and the pose data of the right-eye camera, respectively.

[0025] In one embodiment, the step of rendering the 3D model to be rendered based on the geometric data and the target view matrix array to obtain a left-eye texture image and a right-eye texture image includes:

[0026] Based on the target view matrix array, the geometric data is subjected to coordinate transformation to obtain left eye geometric data and right eye geometric data;

[0027] Based on the visual cone data from the left eye camera and the visual cone data from the right eye camera of the head-mounted display device, a left-eye projection matrix and a right-eye projection matrix are constructed respectively.

[0028] Based on the left-eye projection matrix, coordinate transformation processing is performed on the left-eye geometric data to project the three-dimensional model to be rendered onto a two-dimensional plane, generating the left-eye texture image. Similarly, based on the right-eye projection matrix, coordinate transformation processing is performed on the right-eye geometric data to project the three-dimensional model to be rendered onto a two-dimensional plane, generating the right-eye texture image.

[0029] In one embodiment, after the step of rendering the 3D model to be rendered based on the geometric data and the target view matrix array to obtain the left eye texture image and the right eye texture image, the method further includes:

[0030] Based on the image display parameters of the head-mounted display device, the left eye texture image and the right eye texture image are subjected to image adjustment processing to obtain the left eye texture image and the right eye texture image after image adjustment processing.

[0031] The image-adjusted texture images of the left and right eyes are written into the texture image array of the head-mounted display device, so as to display the image-adjusted texture images of the left and right eyes respectively on the left and right eye screens of the head-mounted display device.

[0032] Furthermore, to achieve the above objectives, this application also provides a binocular rendering device applied to a graphics processor, the device comprising:

[0033] The data acquisition module is used to acquire the geometric data of the 3D model to be rendered, and to acquire the left-eye view matrix and right-eye view matrix of the head-mounted display device;

[0034] The data processing module is used to store the left-eye view matrix and the right-eye view matrix into a preset view matrix array to obtain a target view matrix array; and to perform image rendering on the three-dimensional model to be rendered based on the geometric data and the target view matrix array to obtain a left-eye texture image and a right-eye texture image.

[0035] The data output module is used to write the left-eye texture image and the right-eye texture image into the texture image array of the head-mounted display device, so as to display the left-eye texture image and the right-eye texture image on the left-eye screen and the right-eye screen of the head-mounted display device, respectively.

[0036] In addition, to achieve the above objectives, this application also provides a head-mounted display device, the head-mounted display device comprising: a graphics processor, a central processing unit, a memory, and a computer program stored in the memory and executable on the graphics processor, the computer program being configured to implement the steps of the binocular rendering method as described above.

[0037] In addition, to achieve the above objectives, this application also provides a readable storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a graphics processor, implements the steps of the stereo rendering method described above.

[0038] In addition, to achieve the above objectives, this application also provides a computer program product, including a computer program that, when executed by a graphics processor, implements the steps of the stereo rendering method described above.

[0039] This application provides a binocular rendering method applied to a graphics processor. The method first acquires the geometric data of the 3D model to be rendered using the graphics processor, and then acquires the left-eye view matrix and right-eye view matrix of the head-mounted display device. Next, the left-eye view matrix and right-eye view matrix are stored in a preset view matrix array to obtain a target view matrix array. Then, based on the geometric data and the target view matrix array, the 3D model to be rendered is image rendered, thus simultaneously obtaining the left-eye texture image and the right-eye texture image. Finally, the left-eye texture image and the right-eye texture image are written into the texture image array of the head-mounted display device, so that the left-eye texture image and the right-eye texture image are displayed on the left-eye screen and the right-eye screen of the head-mounted display device, respectively. Since the geometric data used in the process of rendering the left-eye texture image and the process of rendering the right-eye texture image are the same, the method is applicable to both cases. Therefore, in this application, the graphics processor obtains the target view matrix array by storing the left-eye view matrix and the right-eye view matrix together in a preset view matrix array. Thus, the graphics processor only needs to obtain the geometric data of the 3D model to be rendered once, and can use the geometric data and the target view matrix array to simultaneously render and generate the left-eye texture image and the right-eye texture image.

[0040] In summary, during the generation of left-eye and right-eye texture images in binocular rendering, the graphics processor (GPU) only needs to acquire the geometric data of the 3D model to be rendered once. That is, the central processing unit (CPU) only needs to transmit the geometric data to the GPU once to complete the binocular rendering. Compared to the conventional method that requires the CPU to transmit geometric data to the GPU multiple times to complete binocular rendering, the binocular rendering method provided in this application effectively reduces the impact of data transmission efficiency between the CPU and GPU on the overall rendering time, shortens the rendering duration, and improves the rendering efficiency of binocular rendering. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A schematic flowchart of the binocular rendering method provided in the first embodiment of this application;

[0044] Figure 2 This is a schematic diagram of a conventional rendering process provided in an embodiment of this application;

[0045] Figure 3 A schematic diagram of the rendering process when using the binocular rendering method of this application, provided for an embodiment of this application;

[0046] Figure 4 This is a schematic diagram of the module structure of the binocular rendering device provided in the embodiments of this application;

[0047] Figure 5 This is a schematic diagram of the hardware operating environment of the head-mounted display device involved in the embodiments of this application.

[0048] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0049] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0050] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0051] In the rendering process of head-mounted display devices such as AR glasses and VR glasses, the GPU plays a core role in creating realistic stereoscopic visual effects, responsible for binocular rendering of the content in the virtual scene. The current binocular rendering process usually requires the use of two cameras to image the same scene, that is, setting different view matrices to render two slightly different images so that the human eye can perceive depth through visual differences.

[0052] Currently, in stereo rendering, the GPU uses two separate threads to render and generate the left and right eye texture images. However, research has revealed that the geometric data used in rendering the left and right eye texture images is identical. Since the left and right eye texture images are rendered in separate threads within the GPU, the CPU needs to send geometric data to the GPU twice. Therefore, current stereo rendering methods are significantly affected by the data transfer efficiency between the CPU and GPU, resulting in typically longer rendering times.

[0053] Based on this, this application provides a binocular rendering method applied to a graphics processor. This application first acquires the geometric data of the 3D model to be rendered using the graphics processor, and then acquires the left-eye view matrix and right-eye view matrix of the head-mounted display device. Next, the left-eye view matrix and right-eye view matrix are stored in a preset view matrix array to obtain a target view matrix array. Then, based on the geometric data and the target view matrix array, the 3D model to be rendered is image rendered, thus simultaneously obtaining the left-eye texture image and the right-eye texture image. Afterwards, the left-eye texture image and the right-eye texture image are written into the texture image array of the head-mounted display device, so that the left-eye texture image and the right-eye texture image are displayed on the left-eye screen and the right-eye screen of the head-mounted display device, respectively. Since the geometric data used in the process of rendering the left-eye texture image and the process of rendering the right-eye texture image are the same,... Therefore, in this application, the graphics processor obtains the target view matrix array by storing the left-eye view matrix and the right-eye view matrix together in a preset view matrix array. Thus, the graphics processor only needs to obtain the geometric data of the 3D model to be rendered once, and can use the geometric data and the target view matrix array to simultaneously render and generate the left-eye texture image and the right-eye texture image.

[0054] In summary, during the generation of left-eye and right-eye texture images in binocular rendering, the graphics processor (GPU) only needs to acquire the geometric data of the 3D model to be rendered once. That is, the central processing unit (CPU) only needs to transmit the geometric data to the GPU once to complete the binocular rendering. Compared to the conventional method that requires the CPU to transmit geometric data to the GPU multiple times to complete binocular rendering, the binocular rendering method provided in this application effectively reduces the impact of data transmission efficiency between the CPU and GPU on the overall rendering time, shortens the rendering duration, and improves the rendering efficiency of binocular rendering.

[0055] The execution subject of the binocular rendering method of this application can be a graphics processor, a head-mounted display device including a graphics processor and a central processing unit, or a control system or control circuit capable of data processing, network communication and program operation functions. This embodiment does not specifically limit it in this regard.

[0056] The following description uses a head-mounted display device as an example to illustrate the various embodiments.

[0057] Based on this, this application proposes a binocular rendering method according to a first embodiment, applied to a graphics processor. Please refer to... Figure 1 The binocular rendering method may include steps S10 to S40:

[0058] Step S10: Obtain the geometric data of the 3D model to be rendered, and obtain the left eye view matrix and right eye view matrix of the head-mounted display device;

[0059] It should be noted that the 3D model to be rendered refers to the virtual scene to be rendered. Geometric (mesh) data includes, but is not limited to, the vertex coordinates, texture coordinates, and normal information of the 3D model to be rendered. Geometric data is used to define the shape and structure of the 3D model to be rendered in virtual 3D space. Head-mounted display devices can include, but are not limited to, Mixed Reality (MR) devices (e.g., MR glasses or MR helmets), Augmented Reality (AR) devices (e.g., AR glasses or AR helmets), Virtual Reality (VR) devices (e.g., VR glasses or VR helmets), Extended Reality (XR) devices, or some combination thereof. The left-eye view matrix represents the position and orientation of the left-eye camera of the head-mounted display device in 3D space, and the right-eye view matrix represents the position and orientation of the right-eye camera of the head-mounted display device in 3D space. The left-eye view matrix and the right-eye view matrix can transform the 3D model to be rendered from the world coordinate system to the camera coordinate system to simulate the scene from the camera's perspective.

[0060] In one feasible implementation, the graphics processor can obtain the geometric data of the 3D model to be rendered from the central processing unit of the head-mounted display device, and obtain the left-eye view matrix and right-eye view matrix of the head-mounted display device from the central processing unit.

[0061] In another feasible implementation, the graphics processor can obtain the geometric data of the 3D model to be rendered from the central processing unit of the head-mounted display device, while the left-eye view matrix and right-eye view matrix of the head-mounted display device can be determined using the pose data of the left-eye camera and the pose data of the right-eye camera of the head-mounted display device obtained from the central processing unit. Specifically, step S10 may include steps S11 to S12:

[0062] Step S11: Obtain the pose data of the left eye camera and the pose data of the right eye camera of the head-mounted display device;

[0063] It should be noted that the pose data of the left eye camera may include, but is not limited to, the position (translation vector) data and orientation (rotation matrix) data of the left eye camera in three-dimensional space; the pose data of the right eye camera may include, but is not limited to, the position (translation vector) data and orientation (rotation matrix) data of the right eye camera in three-dimensional space.

[0064] Step S12: Construct the left-eye view matrix and the right-eye view matrix based on the pose data of the left-eye camera and the pose data of the right-eye camera, respectively.

[0065] In one feasible implementation, when the pose data includes position data and orientation data, step S12 may include: constructing a translation matrix for the left eye camera and a translation matrix for the right eye camera based on the position data of the left eye camera and the position data of the right eye camera, respectively; constructing a rotation matrix for the left eye camera and a rotation matrix for the right eye camera based on the orientation data of the left eye camera and the orientation data of the right eye camera, respectively; calculating the product of the inverse matrix of the translation matrix of the left eye camera and the inverse matrix of the rotation matrix of the left eye camera to obtain the left eye view matrix; and calculating the product of the inverse matrix of the translation matrix of the right eye camera and the inverse matrix of the rotation matrix of the right eye camera to obtain the right eye view matrix.

[0066] This embodiment does not specifically limit the implementation of step S12. For example, in other feasible implementations, the pose data of the left-eye camera and the pose data of the right-eye camera can be input into the graphics library function to calculate the left-eye view matrix and the right-eye view matrix using the graphics library function.

[0067] The above are only two feasible implementations of step S10 provided in this embodiment. This embodiment does not specifically limit the specific implementation of step S10.

[0068] Step S20: Store the left-eye view matrix and the right-eye view matrix into a preset view matrix array to obtain the target view matrix array;

[0069] It should be noted that the default view matrix array is a view matrix array created within the graphics processor. This array stores the left-eye and right-eye view matrices required during the rendering process. During the creation of the view matrix array, the array can be flexibly created based on the left-eye and right-eye view matrices needed for rendering (for example, the storage space required for the view matrix array can be determined based on the space occupied by the left-eye and right-eye view matrices; then, the view matrix array is created based on this storage space size). The target view matrix array is the view matrix array that stores the left-eye and right-eye view matrices.

[0070] Step S30: Based on the geometric data and the target view matrix array, perform image rendering on the 3D model to be rendered to obtain the left eye texture image and the right eye texture image;

[0071] It should be noted that the left-eye texture image is the image generated during the binocular rendering process for the left-eye perspective. This left-eye texture image can include all visual details (e.g., color, texture, and shadows) of the virtual scene (i.e., the 3D model to be rendered) as observed from the left-eye position. The right-eye texture image is the image generated during the binocular rendering process for the right-eye perspective. This right-eye texture image can include all visual details (e.g., color, texture, and shadows) of the virtual scene (i.e., the 3D model to be rendered) as observed from the right-eye position.

[0072] Step S40: Write the left-eye texture image and the right-eye texture image into the texture image array of the head-mounted display device, so as to display the left-eye texture image and the right-eye texture image on the left-eye screen and the right-eye screen of the head-mounted display device, respectively.

[0073] It should be noted that the texture image array is used to store the left-eye texture image and the right-eye texture image that need to be displayed on the left-eye screen and the right-eye screen of the head-mounted display device. This texture image array can be created in the graphics processor or the central processing unit. This embodiment does not specifically limit it in this way.

[0074] Additionally, it should be noted that during the creation of the texture image array, the array can be flexibly created based on the rendered left-eye and right-eye texture images. For example, the storage space required for the texture image array can be determined based on the space occupied by the left-eye and right-eye texture images; then, the texture image array can be created based on that storage space size.

[0075] This embodiment provides a binocular rendering method applied to a graphics processor. The graphics processor first acquires the geometric data of the 3D model to be rendered, and then acquires the left-eye view matrix and right-eye view matrix of the head-mounted display device. Next, the left-eye view matrix and right-eye view matrix are stored in a preset view matrix array to obtain a target view matrix array. Then, based on the geometric data and the target view matrix array, the 3D model to be rendered is image rendered, simultaneously obtaining left-eye and right-eye texture images. Afterwards, the left-eye and right-eye texture images are written into the texture image array of the head-mounted display device, so that the left-eye texture image and right-eye texture image are displayed on the left-eye and right-eye screens of the head-mounted display device, respectively. Since the geometric data used in the rendering of the left-eye texture image and the rendering of the right-eye texture image are the same, in this embodiment, the graphics processor obtains the target view matrix array by storing the left-eye view matrix and right-eye view matrix together in a preset view matrix array. Therefore, the graphics processor only needs to acquire the geometric data of the 3D model to be rendered once, and can simultaneously render and generate the left-eye and right-eye texture images using the geometric data and the target view matrix array.

[0076] In summary, in this embodiment, during the generation of left-eye and right-eye texture images through binocular rendering, the graphics processor only needs to acquire the geometric data of the 3D model to be rendered once. That is, the central processing unit (CPU) only needs to transmit the geometric data to the GPU once to complete the binocular rendering. Compared to the conventional method that requires the CPU to transmit geometric data to the GPU multiple times to complete binocular rendering, the binocular rendering method provided in this embodiment effectively reduces the impact of data transmission efficiency between the CPU and GPU on the overall rendering time, shortens the rendering duration, and improves the rendering efficiency of binocular rendering.

[0077] In one feasible implementation, step S30 may include steps S31 to S33:

[0078] Step S31: Based on the target view matrix array, perform coordinate transformation on the geometric data to obtain the left eye geometric data and the right eye geometric data;

[0079] It should be noted that the left-eye geometric data is the geometric data after coordinate transformation, which is essentially the geometric data in the camera coordinate system of the left-eye camera; the right-eye geometric data is the geometric data after coordinate transformation, which is essentially the geometric data in the camera coordinate system of the right-eye camera.

[0080] Additionally, it should be noted that the process of performing coordinate transformation on the geometric data based on the target view matrix array to obtain the left-eye and right-eye geometric data actually involves two coordinate transformation processes. Specifically, firstly, the pose data of the 3D model to be rendered is used to transform the geometric data from the model coordinate system to the world coordinate system to obtain the initial geometric data; then, the left-eye view matrix and right-eye view matrix stored in the target view matrix array are used to transform the initial geometric data from the world coordinate system to the camera coordinate system of the left-eye camera and the camera coordinate system of the right-eye camera, respectively, to obtain the left-eye and right-eye geometric data.

[0081] Step S32: Construct the left-eye projection matrix and the right-eye projection matrix based on the visual cone data of the left-eye camera and the visual cone data of the right-eye camera of the head-mounted display device, respectively.

[0082] It should be noted that the frustum data of the left-eye camera is used to record the visible range of the left-eye camera, which may include, but is not limited to, the top and bottom visible areas (characterizing the vertical visible range of the left-eye camera), the left and right visible areas (characterizing the horizontal visible range of the left-eye camera), the near clipping plane, and the far clipping plane. The frustum data of the right-eye camera is used to record the visible range of the right-eye camera, which may include, but is not limited to, the top and bottom visible areas (characterizing the vertical visible range of the right-eye camera), the right and right visible areas (characterizing the horizontal visible range of the right-eye camera), the near clipping plane, and the far clipping plane. The projection matrix is ​​used to map points in three-dimensional space onto a two-dimensional plane.

[0083] In one feasible implementation, taking the construction of the left-eye projection matrix as an example, assuming the left-eye camera's frustum data includes the near clipping plane distance, far clipping plane distance, horizontal field of view, vertical field of view, and viewport aspect ratio, then in constructing the left-eye projection matrix, the horizontal scaling factor can be calculated first based on the horizontal field of view and viewport aspect ratio, and the vertical scaling factor can be calculated based on the vertical field of view; then, the first coefficient and the second coefficient can be calculated based on the near clipping plane distance and far clipping plane distance; finally, by utilizing the far clipping plane distance, horizontal scaling factor, vertical scaling factor, first coefficient, and second coefficient, the following left-eye projection matrix can be constructed:

[0084]

[0085] Where scaleX is the horizontal scaling factor, scaleY is the vertical scaling factor, A is the first coefficient, B is the second coefficient, and far is the distance to the far cut surface. When calculating the first and second coefficients based on the near and far cut surface distances, the ratio of the negative of the sum of the near and far cut surface distances to the difference between the near and far cut surface distances can be used as the first coefficient; the ratio of the negative of twice the product of the near and far cut surface distances to the difference between the near and far cut surface distances can be used as the second coefficient.

[0086] The above is merely an implementation example of a process for constructing a projection matrix provided in this embodiment. This embodiment does not specifically limit the specific implementation of step S32.

[0087] Step S33: Based on the left eye projection matrix, perform coordinate transformation processing on the left eye geometric data to project the 3D model to be rendered onto a 2D plane and generate a left eye texture image; and based on the right eye projection matrix, perform coordinate transformation processing on the right eye geometric data to project the 3D model to be rendered onto a 2D plane and generate a right eye texture image.

[0088] It should be noted that, based on the left-eye projection matrix, performing coordinate transformation on the left-eye geometric data essentially transforms the left-eye geometric data from the camera coordinate system of the left-eye camera to the clipping coordinate system of the left-eye camera; similarly, based on the right-eye projection matrix, performing coordinate transformation on the right-eye geometric data essentially transforms the right-eye geometric data from the camera coordinate system of the right-eye camera to the clipping coordinate system of the right-eye camera.

[0089] Based on the first embodiment described above, a second embodiment of the binocular rendering method of this application is proposed. In the second embodiment, before step S20, the binocular rendering method may further include steps S21 to S23:

[0090] Step S21: Verify whether the matrix dimensions of the left eye view matrix and the right eye view matrix are consistent.

[0091] It should be noted that matrix dimension is used to indicate the number of rows and columns of a matrix. Matrix dimension is usually represented by two numbers: the first number represents the number of rows and the second number represents the number of columns. That is, matrix dimension is usually expressed as "number of rows × number of columns".

[0092] Step S22, if yes, then store the left eye view matrix and the right eye view matrix into a preset view matrix array to obtain the target view matrix array;

[0093] Step S23: If not, perform matrix dimension unification processing on the left eye view matrix and the right eye view matrix, and after performing matrix dimension unification processing, execute the step of storing the left eye view matrix and the right eye view matrix into a preset view matrix array to obtain the target view matrix array.

[0094] It should be noted that when unifying the matrix dimensions of the left eye view matrix and the right eye view matrix, both can be unified to the matrix dimensions of the left eye view matrix or the right eye view matrix by default, or they can be unified to a set matrix dimension. This embodiment does not make specific limitations on this.

[0095] It is understandable that during the process of storing the left-eye view matrix and the right-eye view matrix into a preset view matrix array, if their matrix dimensions are inconsistent, it may lead to space allocation errors or data overwriting in the view matrix array, resulting in storage failure. Therefore, this embodiment ensures that the left-eye view matrix and the right-eye view matrix are stored in the preset view matrix array only when their matrix dimensions are consistent. This ensures that they can be successfully stored in the preset view matrix array, and that the left-eye view matrix and the right-eye view matrix are accurately placed in the predetermined positions, thereby improving the stability and accuracy of binocular rendering.

[0096] Based on the first and / or second embodiments described above, a third embodiment of the binocular rendering method of this application is proposed. In the third embodiment, before step S20, the binocular rendering method may further include steps S24 to S26:

[0097] Step S24: Obtain the first space size occupied by the left eye view matrix in the central processing unit of the head-mounted display device, and the second space size occupied by the right eye view matrix in the central processing unit;

[0098] It should be noted that the first space size refers to the space occupied by the left eye view matrix in the central processing unit of the head-mounted display device; the second space size refers to the space occupied by the right eye view matrix in the central processing unit of the head-mounted display device.

[0099] Step S25: Determine the storage space size of the view matrix array based on the sum of the sizes and values ​​of the first and second space sizes;

[0100] It should be noted that when determining the storage space size of the view matrix array based on the sum of the first and second space sizes, the sum of the space sizes can be used directly as the storage space size of the view matrix array; alternatively, the sum of the space sizes can be added to a preset space size; or the sum of the space sizes can be multiplied by a preset coefficient value. This embodiment does not impose specific limitations on these methods.

[0101] The preset coefficient value can be any value greater than 1. Both the preset space size and the preset coefficient value can be a default value, or they can be flexibly set by the user according to the actual situation. This embodiment does not impose any specific limitations on this.

[0102] Step S26: Create a view matrix array based on the storage space size.

[0103] In this embodiment, the required storage space size for the view matrix array is determined by the first space size occupied by the left-eye view matrix in the central processing unit (CPU) of the head-mounted display device, and the second space size occupied by the right-eye view matrix in the CPU. This storage space size is then used to create the view matrix array. This ensures that the created view matrix array has sufficient storage space to store the left-eye and right-eye view matrices. It avoids the problem of incorrect space allocation for the left-eye and right-eye view matrices in the view matrix array due to insufficient storage space, ensuring that the left-eye and right-eye view matrices can be successfully stored in the preset view matrix array. This also ensures that the left-eye and right-eye view matrices can be accurately placed in their predetermined positions, thereby improving the stability and accuracy of binocular rendering.

[0104] Based on the first, second, and / or third embodiments described above, a fourth embodiment of the binocular rendering method of this application is proposed. In the fourth embodiment, after step S30, the binocular rendering method may further include steps S50 to S60:

[0105] Step S50: Based on the image display parameters of the head-mounted display device, perform image adjustment processing on the left-eye texture image and the right-eye texture image to obtain the left-eye texture image and the right-eye texture image after image adjustment processing.

[0106] It should be noted that image display parameters may include, but are not limited to, image display resolution, color, etc.

[0107] Step S60: Write the image-adjusted left-eye texture image and right-eye texture image into the texture image array of the head-mounted display device, so as to display the image-adjusted left-eye texture image and right-eye texture image on the left-eye screen and right-eye screen of the head-mounted display device, respectively.

[0108] In this embodiment, after the left-eye texture image and the right-eye texture image are rendered, the left-eye texture image and the right-eye texture image can be adjusted according to the image display parameters of the head-mounted display device to obtain the left-eye texture image and the right-eye texture image that meet the image display requirements of the head-mounted display device. This improves the display effect of the left-eye texture image and the right-eye texture image on the head-mounted display device, and thus improves the rendering effect of the 3D model to be rendered on the head-mounted display device.

[0109] For example, to help understand the difference between the rendering process of the stereo rendering method of this application and the conventional rendering process, please refer to... Figure 2 and Figure 3 .in, Figure 2 The process shown is a standard rendering process. Figure 3The illustrated flow is the rendering flow of the stereo rendering method of this application. Figure 2 and Figure 3 As is known, conventional rendering processes require two separate threads to generate the left-eye and right-eye texture images. Consequently, the CPU needs to transmit geometric data to the GPU twice to complete the stereo rendering. However, the stereo rendering method of this application renders the generated left-eye and right-eye texture images in a single thread. Therefore, the CPU only needs to transmit geometric data to the GPU once to complete the stereo rendering.

[0110] in, Figure 2 and Figure 3 The rendering process involves several steps: 1) Clipping: This primarily involves evaluating objects in the virtual scene to determine whether they need to be rendered; 2) Shadow rendering: This adds shadow effects to objects in the virtual scene; 3) Opaque objects: These are objects drawn sequentially during rendering without considering transparency; 4) Background rendering: This renders the background of the virtual scene; 5) Image effects application 1: This applies a series of image effects to the image after opaque objects are rendered but before transparent objects are rendered. These effects may include, but are not limited to, color correction, tone mapping, depth of field, and motion blur, aiming to enhance the overall visual effect of the virtual scene; 6) Transparent objects: These are objects whose transparency needs to be considered during rendering; 7) Image effects application 2: This applies a series of image effects to the final image after all objects (including transparent objects) have been rendered. These effects may include, but are not limited to, color correction, filters, and special effects, aiming to further enhance the overall visual effect of the virtual scene.

[0111] It should be noted that this example is only for the purpose of assisting in understanding this application and does not constitute a limitation on the binocular rendering method of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0112] This application also provides a binocular rendering device applied to a graphics processor; please refer to... Figure 4 A binocular rendering device may include:

[0113] The data acquisition module 10 is used to acquire the geometric data of the 3D model to be rendered, and to acquire the left eye view matrix and right eye view matrix of the head-mounted display device;

[0114] The data processing module 20 is used to store the left-eye view matrix and the right-eye view matrix into a preset view matrix array to obtain a target view matrix array; based on the geometric data and the target view matrix array, it performs image rendering on the 3D model to be rendered to obtain the left-eye texture image and the right-eye texture image.

[0115] The data output module 30 is used to write the left-eye texture image and the right-eye texture image into the texture image array of the head-mounted display device, so as to display the left-eye texture image and the right-eye texture image on the left-eye screen and the right-eye screen of the head-mounted display device, respectively.

[0116] In one embodiment, the data processing module 20 is further configured to:

[0117] Verify whether the matrix dimensions of the left-eye view matrix are consistent with those of the right-eye view matrix;

[0118] If so, then the left-eye view matrix and the right-eye view matrix are stored in a preset view matrix array to obtain the target view matrix array;

[0119] If not, perform matrix dimension unification processing on the left-eye view matrix and the right-eye view matrix, and after completing the matrix dimension unification processing, execute the step of storing the left-eye view matrix and the right-eye view matrix into a preset view matrix array to obtain the target view matrix array.

[0120] In one embodiment, the data processing module 20 is further configured to:

[0121] Obtain the first space size occupied by the left eye view matrix in the central processing unit of the head-mounted display device, and the second space size occupied by the right eye view matrix in the central processing unit;

[0122] The storage space size of the view matrix array is determined based on the size and value of the first and second space sizes.

[0123] Create a view matrix array based on the storage space size.

[0124] In one embodiment, the data processing module 20 is further configured to:

[0125] Use the size of the space and the value as the storage space size of the view matrix array;

[0126] Alternatively, add the space size and value to the preset space size to obtain the storage space size of the view matrix array;

[0127] Alternatively, multiply the space size and value by the preset coefficient value to obtain the storage space size of the view matrix array.

[0128] In one embodiment, the data acquisition module 10 is further configured to:

[0129] Acquire pose data from the left eye camera and the right eye camera of the head-mounted display device;

[0130] Based on the pose data of the left eye camera and the pose data of the right eye camera, construct the left eye view matrix and the right eye view matrix respectively.

[0131] In one embodiment, the data processing module 20 is further configured to:

[0132] Based on the target view matrix array, the geometric data is processed by coordinate transformation to obtain the left eye geometric data and the right eye geometric data;

[0133] Based on the visual cone data from the left-eye camera and the visual cone data from the right-eye camera of the head-mounted display device, the left-eye projection matrix and the right-eye projection matrix are constructed respectively.

[0134] Based on the left-eye projection matrix, coordinate transformation is performed on the left-eye geometric data to project the 3D model to be rendered onto a 2D plane, generating a left-eye texture image. Similarly, based on the right-eye projection matrix, coordinate transformation is performed on the right-eye geometric data to project the 3D model to be rendered onto a 2D plane, generating a right-eye texture image.

[0135] In one embodiment, the data processing module 20 is further configured to perform image adjustment processing on the left eye texture image and the right eye texture image according to the image display parameters of the head-mounted display device, so as to obtain the left eye texture image and the right eye texture image after image adjustment processing;

[0136] The data output module 30 is also used to write the image-adjusted left-eye texture image and right-eye texture image into the texture image array of the head-mounted display device, so as to display the image-adjusted left-eye texture image and right-eye texture image on the left-eye screen and right-eye screen of the head-mounted display device, respectively.

[0137] The binocular rendering apparatus provided in this application, employing the binocular rendering method described in the above embodiments, can shorten the rendering time of binocular rendering, thereby improving the rendering efficiency of binocular rendering. Compared with the prior art, the beneficial effects of the binocular rendering apparatus provided in this application are the same as those of the binocular rendering method provided in the above embodiments, and other technical features in this binocular rendering apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0138] This application provides a head-mounted display device, which includes: a graphics processor, a central processing unit, and a memory communicatively connected to the graphics processor and the central processing unit; wherein, the memory stores instructions executable by the graphics processor, which are executed by the graphics processor to enable the graphics processor to perform the binocular rendering method in the above embodiment.

[0139] The following is for reference. Figure 5The diagram illustrates a structural schematic suitable for implementing the head-mounted display device of the embodiments of this application. The head-mounted display devices in the embodiments of this application include, but are not limited to, Mixed Reality (MR) devices (e.g., MR glasses or MR helmets), Augmented Reality (AR) devices (e.g., AR glasses or AR helmets), Virtual Reality (VR) devices (e.g., VR glasses or VR helmets), Extended Reality (XR) devices, or some combination thereof. Figure 5 The head-mounted display device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0140] like Figure 5 As shown, the head-mounted display device may include a processing unit 1001 (which may include a central processing unit and a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM 1002) or a program loaded from a storage device into a random access memory (RAM 1004). The RAM 1004 also stores various programs and data required for the operation of the head-mounted display device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface is also connected to the bus 1005.

[0141] Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the head-mounted display to communicate wirelessly or wiredly with other devices to exchange data. Although head-mounted display devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems may be implemented alternatively.

[0142] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of embodiments of this application.

[0143] The head-mounted display device provided in this application, employing the binocular rendering method in the above embodiments, can shorten the rendering time of binocular rendering, thereby improving the rendering efficiency of binocular rendering. Compared with the prior art, the beneficial effects of the head-mounted display device provided in this application are the same as the beneficial effects of the binocular rendering method provided in the above embodiments, and other technical features in this head-mounted display device are the same as the features disclosed in the methods of the above embodiments, and will not be repeated here.

[0144] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0145] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0146] This application also provides a computer-readable storage medium storing a computer program that can run on a graphics processor, the computer program being used to execute the steps of the stereo rendering method in the above embodiments.

[0147] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0148] The aforementioned computer-readable storage medium may be included in the head-mounted display device; or it may exist independently and not assembled into the head-mounted display device.

[0149] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a head-mounted display device, cause the head-mounted display device to: acquire geometric data of a 3D model to be rendered, and acquire the left-eye view matrix and right-eye view matrix of the head-mounted display device; store the left-eye view matrix and right-eye view matrix into a preset view matrix array to obtain a target view matrix array; perform image rendering on the 3D model to be rendered based on the geometric data and the target view matrix array to obtain a left-eye texture image and a right-eye texture image; and write the left-eye texture image and right-eye texture image into the texture image array of the head-mounted display device to display the left-eye texture image and right-eye texture image on the left-eye screen and right-eye screen of the head-mounted display device, respectively.

[0150] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0152] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0153] The computer-readable storage medium provided in this application embodiment stores computer-readable program instructions for executing the above-described stereo rendering method, which can shorten the rendering time of stereo rendering and improve the rendering efficiency of stereo rendering. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application embodiment are the same as the beneficial effects of the stereo rendering method provided in the above embodiments, and will not be repeated here.

[0154] This application also provides a computer program product, including a computer program that, when executed by a graphics processor, implements the steps of the stereo rendering method in the above embodiments.

[0155] The computer program product provided in this application can shorten the rendering time of stereo rendering, thereby improving the rendering efficiency of stereo rendering. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the stereo rendering method provided in the above embodiments, and will not be repeated here.

[0156] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. A binocular rendering method, characterized in that, Applied to a graphics processor, the method includes: Obtain the geometric data of the 3D model to be rendered, and obtain the left and right eye view matrices of the head-mounted display device; The left-eye view matrix and the right-eye view matrix are stored in a preset view matrix array to obtain the target view matrix array; Based on the geometric data and the target view matrix array, the three-dimensional model to be rendered is rendered to obtain the left eye texture image and the right eye texture image; The left-eye texture image and the right-eye texture image are written into the texture image array of the head-mounted display device, so as to display the left-eye texture image and the right-eye texture image on the left-eye screen and the right-eye screen of the head-mounted display device, respectively.

2. The binocular rendering method as described in claim 1, characterized in that, Before the step of storing the left-eye view matrix and the right-eye view matrix into a preset view matrix array to obtain the target view matrix array, the method further includes: Verify whether the matrix dimensions of the left eye view matrix are consistent with those of the right eye view matrix; If so, then the process of storing the left-eye view matrix and the right-eye view matrix into a preset view matrix array is executed to obtain the target view matrix array; If not, then the left eye view matrix and the right eye view matrix are subjected to matrix dimension unification processing, and after the matrix dimension unification processing is completed, the step of storing the left eye view matrix and the right eye view matrix into a preset view matrix array to obtain the target view matrix array is executed.

3. The binocular rendering method as described in claim 1, characterized in that, Before the step of storing the left-eye view matrix and the right-eye view matrix into a preset view matrix array to obtain the target view matrix array, the method further includes: Obtain the first space size occupied by the left eye view matrix in the central processing unit of the head-mounted display device, and the second space size occupied by the right eye view matrix in the central processing unit; The storage space size of the view matrix array is determined based on the sum of the first space size and the second space size. Create the view matrix array based on the storage space size.

4. The binocular rendering method as described in claim 3, characterized in that, The step of determining the storage space size of the view matrix array based on the sum of the first space size and the second space size includes: The space size and value are used as the storage space size of the view matrix array; Alternatively, the storage space size and value can be added to a preset space size to obtain the storage space size of the view matrix array; Alternatively, the storage space size and value can be multiplied by a preset coefficient value to obtain the storage space size of the view matrix array.

5. The binocular rendering method as described in any one of claims 1 to 4, characterized in that, The steps of obtaining the left-eye view matrix and right-eye view matrix of the head-mounted display device include: Acquire the pose data of the left eye camera and the pose data of the right eye camera of the head-mounted display device; The left-eye view matrix and the right-eye view matrix are constructed based on the pose data of the left-eye camera and the pose data of the right-eye camera, respectively.

6. The binocular rendering method as described in any one of claims 1 to 4, characterized in that, The step of rendering the 3D model to be rendered based on the geometric data and the target view matrix array to obtain the left eye texture image and the right eye texture image includes: Based on the target view matrix array, the geometric data is subjected to coordinate transformation to obtain left eye geometric data and right eye geometric data; Based on the visual cone data from the left eye camera and the visual cone data from the right eye camera of the head-mounted display device, a left-eye projection matrix and a right-eye projection matrix are constructed respectively. Based on the left-eye projection matrix, coordinate transformation processing is performed on the left-eye geometric data to project the three-dimensional model to be rendered onto a two-dimensional plane, generating the left-eye texture image. Similarly, based on the right-eye projection matrix, coordinate transformation processing is performed on the right-eye geometric data to project the three-dimensional model to be rendered onto a two-dimensional plane, generating the right-eye texture image.

7. The binocular rendering method as described in any one of claims 1 to 4, characterized in that, After the step of rendering the 3D model to be rendered based on the geometric data and the target view matrix array to obtain the left eye texture image and the right eye texture image, the method further includes: Based on the image display parameters of the head-mounted display device, the left eye texture image and the right eye texture image are subjected to image adjustment processing to obtain the left eye texture image and the right eye texture image after image adjustment processing. The image-adjusted texture images of the left and right eyes are written into the texture image array of the head-mounted display device, so as to display the image-adjusted texture images of the left and right eyes respectively on the left and right eye screens of the head-mounted display device.

8. A binocular rendering device, characterized in that, The device, applied to a graphics processor, includes: The data acquisition module is used to acquire the geometric data of the 3D model to be rendered, and to acquire the left-eye view matrix and right-eye view matrix of the head-mounted display device; The data processing module is used to store the left-eye view matrix and the right-eye view matrix into a preset view matrix array to obtain a target view matrix array; and to perform image rendering on the three-dimensional model to be rendered based on the geometric data and the target view matrix array to obtain a left-eye texture image and a right-eye texture image. The data output module is used to write the left-eye texture image and the right-eye texture image into the texture image array of the head-mounted display device, so as to display the left-eye texture image and the right-eye texture image on the left-eye screen and the right-eye screen of the head-mounted display device, respectively.

9. A head-mounted display device, characterized in that, The head-mounted display device includes: a graphics processor, a central processing unit, a memory, and a computer program stored in the memory and executable on the graphics processor, the computer program being configured to implement the steps of the binocular rendering method as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a graphics processor, it implements the steps of the stereo rendering method as described in any one of claims 1 to 7.

Images