Reprojection display method and device and electronic equipment
By creating a plane mesh in a head-mounted device and updating the vertex coordinates for reprojection, the problem of large memory usage in traditional PTW methods is solved, improving the XR experience and algorithm performance, and avoiding hole problems.
Patent Information
- Application Number
- CN202411000089.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-08
AI Technical Summary
The traditional positioning time distortion (PTW) method causes excessive memory usage in image reprojection and cannot effectively move pixels in the rendering pipeline, affecting the user experience.
By creating a plane grid rendered with the headset in the first position, obtaining the depth value and 3D spatial coordinates of each pixel, updating the vertex's 2D screen coordinates, and reprojecting it in the fragment shading rendering, cleverly Pixel processing is converted to vertex processing, reducing memory footprint and improving algorithm performance.
It effectively reduces memory usage, improves algorithm performance, improves user's extended reality (XR) experience, and avoids hole problems and processing delays in traditional solutions.
Smart Images

Figure CN118918295A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of image processing, and specifically relates to a reprojection display method, device and electronic equipment. Background Art
[0002] Extended reality (XR) refers to the combination of reality and virtuality through computers to create a virtual environment for human-computer interaction, which brings the experiencer an "immersive feeling" of seamless transition between the virtual world and the real world. However, in actual applications, the perspective of the user through the XR device is different from the perspective of the user's brain. One reason for the difference is that the user's head rotates.
[0003] In the related art, the traditional Positional Timewarp (PTW) method is used to reproject the image. However, PTW can only generate a new image by pixel movement through image processing, which increases memory usage. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide a reprojection display method, device and electronic device that can solve the problem of large memory usage.
[0005] In a first aspect, an embodiment of the present application provides a reprojection display method, the method comprising:
[0006] Creating a plane mesh corresponding to a first frame of image rendered by the head mounted device at a first pose, wherein the plane mesh includes a plurality of vertices, each of the vertices corresponds to a pixel of the first frame of image, and the stored data of the first frame of image includes a depth value of each pixel;
[0007] Based on the depth value of the pixel corresponding to each vertex in the first frame image, obtaining the three-dimensional space coordinate value corresponding to each vertex of the head mounted device in the first posture;
[0008] When the head mounted device changes to a second posture, based on a transformation matrix of the head mounted device in the second posture and a three-dimensional space coordinate value of each vertex, obtaining a projection coordinate of each vertex on the two-dimensional screen;
[0009] Update the two-dimensional screen coordinates of each vertex of the plane mesh according to the projection coordinates of each vertex;
[0010] Based on the first frame image, fragment shading rendering is performed on the updated plane mesh to obtain a reprojected second frame image.
[0011] In a second aspect, an embodiment of the present application provides a reprojection display device, the device comprising:
[0012] A creation module, configured to create a plane mesh corresponding to a first frame image rendered by the head mounted device in a first posture, wherein the plane mesh includes a plurality of vertices, each of the vertices corresponds to a pixel of the first frame image, and the storage data of the first frame image includes a depth value of each pixel;
[0013] A first acquisition module, configured to acquire a three-dimensional space coordinate value corresponding to each vertex of the head mounted device in a first posture based on a depth value of a pixel corresponding to each vertex in the first frame image;
[0014] A second acquisition module is used to obtain the projection coordinates of each vertex on the two-dimensional screen based on the transformation matrix of the head-mounted device in the second posture and the three-dimensional space coordinate value of each vertex when the head-mounted device changes to a second posture, wherein the transformation matrix is used to convert the three-dimensional view collected by the head-mounted device in the second posture into a projection view displayed on the two-dimensional screen of the head-mounted device;
[0015] An updating module, used for updating the two-dimensional screen coordinates of each vertex of the plane mesh according to the projection coordinates of each vertex;
[0016] A shading module is used to perform fragment shading rendering on the updated plane mesh based on the first frame image to obtain a reprojected second frame image.
[0017] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.
[0018] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.
[0019] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.
[0020] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.
[0021] In an embodiment of the present application, a plane mesh including multiple vertices corresponding to the first frame image rendered by the head-mounted device in the first pose can be created, and then the three-dimensional space coordinate value corresponding to each vertex of the head-mounted device in the first pose is obtained based on the depth value of the pixel corresponding to each vertex in the first frame image; when the head-mounted device changes to the second pose, the projection coordinates of each vertex on the two-dimensional screen are obtained based on the transformation matrix of the head-mounted device in the second pose and the three-dimensional space coordinate value of each vertex; then the two-dimensional screen coordinates of each vertex of the plane mesh are updated according to the projection coordinates of each vertex; finally, based on the first frame image, the updated plane mesh is fragment shaded and rendered to obtain the reprojected second frame image. The method provided in the embodiment of the present application can solve the problem that the traditional solution can only process image pixels, make full use of the rendering pipeline architecture, and cleverly convert pixel processing into vertex processing. And compared with the traditional solution, it can reduce memory usage, improve algorithm performance, and further enhance the user's XR experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 A schematic diagram showing a flow chart of a reprojection display method provided in an embodiment of the present application;
[0023] Figure 2 A schematic diagram showing a plane grid parallel to the near clipping plane and covering the entire screen provided by an embodiment of the present application;
[0024] Figure 3 A schematic diagram showing a plane grid mapped on a two-dimensional screen after reprojection provided by an embodiment of the present application;
[0025] Figure 4 A schematic diagram showing a spatial position reprojection result provided by an embodiment of the present application is shown;
[0026] Figure 5 A schematic diagram showing reprojection of plane mesh vertices mapped onto a two-dimensional screen provided by an embodiment of the present application;
[0027] Figure 6 A schematic diagram showing a flow chart of another reprojection display method provided in an embodiment of the present application;
[0028] Figure 7 A schematic structural diagram of a re-projection display device provided in an embodiment of the present application is shown;
[0029] Figure 8 A structural block diagram of an electronic device provided by an exemplary embodiment of the present application is shown;
[0030] Fig. 9 A schematic diagram of the hardware structure of an electronic device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION
[0031] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.
[0032] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0033] In the field of graphics rendering, a conventional rendering process is as follows:
[0034] The first step is to obtain the virtual camera position in the current 3D scene through interactive devices (usually mouse, keyboard, handle) or preset camera roaming data;
[0035] The second step is to render a frame of 3D scene image according to the current virtual camera’s position;
[0036] The third step is to send the rendered three-dimensional scene image to the display for display.
[0037] But there is a problem here. Rendering takes time, and the rendering engine often renders based on the camera posture a few milliseconds before sending it to the display device. During these few milliseconds, a new camera posture transformation will occur, so there is actually a certain delay in the rendered image seen by the user. This is difficult for users to detect in ordinary three-dimensional application scenarios. But on head-mounted devices, since the camera posture is obtained through the head-mounted display's sensor, it simulates the user's real head movement. The perspective that the user actually sees is different from the perspective that the user's brain perceives. Although this difference is only a few milliseconds, it can cause dizziness in the user and greatly reduce the user experience. Secondly, for head-mounted devices, a higher rendering frame rate is also an important factor in ensuring user experience, but when the frame rate is low, the user will also feel dizzy.
[0038] There are generally three reasons for this difference. The first is caused by the rotation of the user's head. The second is caused by the displacement of the user's head. The third is caused by the movement of the object itself in the three-dimensional scene. For the three reasons, there are three solutions in the relevant technology. On the one hand, it can reduce the delay mentioned above, and on the other hand, it can perform frame interpolation operations to achieve an increase in frame rate:
[0039] The first solution is Orientation Timewarp (OTW). Its principle is to reproject the previous frame of rendering based on the head rotation data when the previous frame of rendering starts and when the current OTW starts. The specific implementation method is to reproject the previous frame of rendering onto a grid that covers the entire screen and is parallel to the screen, and then perform a matrix transformation on the grid to finally achieve the redrawing of the previous frame of rendering.
[0040] The second solution is PTW. Its principle is to reproject the previous frame of rendering based on the head position data when the previous frame rendering started and when the current PTW starts. The positions of different pixels of the previous frame of rendering image will change differently on the screen after PTW, which is different from the uniform transformation of all pixels in OTW. PTW requires the use of depth data, and its implementation can generally include the OTW part instead of OTW.
[0041] The third solution is space warp (SW). When the objects in the 3D scene move, it is not enough to reproject the image using the change of head posture. A prediction method is often required. The conventional method is to use the optical flow method (image-based processing method) to generate the motion vector data of the pixels between two consecutive frames of rendered images, and then perform frame extrapolation on the rendered image to achieve reprojection.
[0042] PTW generally uses depth data to restore the 3D spatial coordinates corresponding to the pixels, and then reprojects them onto the 2D screen using the latest head pose information. The pixels of the original real rendered image are then moved to the new position to redraw the previous frame of the rendered image.
[0043] Traditional PTW methods generally have the following defects:
[0044] (1) Reprojection is done on a pixel basis. In XR scenarios, the rendering resolution of a single eye often reaches 2k, and images of both eyes need to be drawn. The amount of computation required for reprojection of each pixel is huge.
[0045] (2) It is very difficult to move pixels in the fragment shader of a conventional rendering pipeline. Often, pixels can only be moved through image processing to generate a new image. On the one hand, this image processing method can generally only be performed in the central processing unit (CPU) and cannot be implemented in the traditional rendering pipeline (graphics processing unit (GPU)), and the processing time is relatively high. On the other hand, an image buffer needs to be added for pixel movement, which increases memory usage;
[0046] (3) The pixels in the original real rendering image will move to the new position in the new image, but there may not be a pixel that can be moved to the position of the pixel. In this case, the pixel at the position will be missing, forming a hole, which will greatly affect the user's visual experience. The additional repair of the hole will add new processing links and increase new time consumption, which is not conducive to the overall performance of the system.
[0047] The following is a detailed description of the reprojection display method provided in the embodiment of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.
[0048] Figure 1 A reprojection display method provided by an exemplary embodiment of the present application is shown. The method 100 can be executed by a terminal device, which can be an extended reality device, such as an XR helmet, XR glasses, etc., or the terminal device can also be a personal terminal device such as a mobile phone or a computer connected to the extended reality device, which is not limited in the embodiments of the present application. Figure 1 As shown, the method mainly includes the following steps:
[0049] S101: Create a plane mesh corresponding to a first frame image rendered by a head mounted device at a first pose.
[0050] The plane grid includes a plurality of vertices, each of the vertices corresponds to a pixel of the first frame image, and the storage data of the first frame image includes a depth value of each pixel.
[0051] In the embodiments of the present application, the head-mounted device includes but is not limited to various XR devices, such as augmented reality technology (AR) devices, virtual reality technology (VR) devices, mixed reality technology (MR) devices, etc.
[0052] In the embodiment of the present application, by creating a plane mesh corresponding to the first frame image rendered by the head mounted device in the first pose, the rendering pipeline architecture can be fully utilized, the pixel processing can be cleverly converted into vertex processing, and the balance between effect and performance can be fully achieved. Compared with the traditional solution, the memory usage can be reduced and the algorithm performance can be improved.
[0053] In an embodiment of the present application, the depth value of each pixel of the first frame image can be cached, so that the depth value of each pixel can be obtained from the depth value cache of the first frame image. The depth value in the depth value cache represents the distance between the camera and the corresponding position of the object and has been normalized.
[0054] In an optional implementation, creating a plane mesh corresponding to a first frame image rendered by a head-mounted device in a first posture may include: creating a plane mesh consistent with a resolution of the first frame image, wherein the plane mesh is parallel to a near clipping plane of the head-mounted device, and a size of the plane mesh is the same as a two-dimensional screen of the head-mounted device.
[0055] In the above implementation of the embodiment of the present application, a plane grid consistent with the resolution of the first frame image (the original real rendering image resolution) can be created, wherein the plane grid is parallel to the near clipping plane of the head-mounted device, and the size of the plane grid can be the same as the two-dimensional screen of the head-mounted device by setting the distance, such as Figure 2 As shown. The purpose of this is to simulate pixels through geometric vertices in the vertex shader (unlike images, the use of geometric vertices is very flexible). The plane mesh includes vertex coordinates, UV coordinates and other information, and because the plane mesh is required to be parallel to the near clipping plane and cover the entire screen, the vertex coordinates of the mesh are directly established in the camera coordinate system, that is, projection transformation can be performed during the space transformation process, without model transformation and view transformation.
[0056] Since the created plane mesh is completely uniform and regular, and in order to further save bandwidth usage and performance loss of data transmission between CPU and GPU, in actual applications, the vertex index of the plane mesh can be created in the CPU, and then the vertices can be created according to the index number and mesh resolution in the vertex shader.
[0057] S102: Based on the depth value of the pixel corresponding to each vertex in the first frame image, obtain the three-dimensional space coordinate value corresponding to each vertex of the head-mounted device in the first posture. In the embodiment of the present application, the three-dimensional space coordinate value corresponding to each vertex of the head-mounted device in the first posture can be obtained based on the depth value of the pixel corresponding to each vertex in the first frame image. The corresponding three-dimensional space coordinate value obtained based on the depth value corresponding to the vertex is more accurate, which helps to improve the user experience.
[0058] In an optional implementation, based on the depth value of the pixel corresponding to each vertex in the first frame image, obtaining the three-dimensional space coordinate value corresponding to each vertex of the head mounted device in the first posture may include the following steps:
[0059] Step 1, obtaining a depth value corresponding to each vertex in the first frame image;
[0060] Step 2, based on the projection relationship between the plane grid and the two-dimensional screen, obtaining the two-dimensional screen coordinates of the vertex projected onto the two-dimensional screen;
[0061] Step 3: Based on the two-dimensional screen coordinates of the vertices projected onto the two-dimensional screen and the transformation matrix of the head mounted device in the first posture, obtain the three-dimensional space coordinates corresponding to each vertex of the head mounted device in the first posture.
[0062] In practical applications, the transformation of the three-dimensional coordinates of a three-dimensional model into the two-dimensional coordinates of the screen is called projection, and the transformation of the two-dimensional coordinates on the screen into three-dimensional coordinates can be called back-projection. In an embodiment of the present application, each vertex can be traversed, and according to its uv coordinate, it can be sampled in the depth buffer corresponding to the first frame image to obtain the corresponding depth value, and then the vertex can be transformed into the two-dimensional screen coordinate according to the projection relationship between the plane grid and the two-dimensional screen (projection matrix), and finally the three-dimensional space coordinates corresponding to each vertex of the head-mounted device in the first posture are obtained through the two-dimensional screen coordinates projected onto the two-dimensional screen and the transformation matrix of the head-mounted device in the first posture.
[0063] In an embodiment of the present application, optionally, the transformation matrix of the head-mounted device in the first posture can be used to convert the three-dimensional view captured by the head-mounted device in the first posture into a projected view displayed on the two-dimensional screen of the head-mounted device. The transformation matrix may include a model matrix, a view matrix, and a projection matrix. The model matrix can transform vertices from the local coordinate system to the world coordinate system; the view matrix can transform vertices from the world coordinate system to the view coordinate system; the projection matrix can transform vertices from the view coordinate system to the canonical cube (i.e., the two-dimensional screen). (The process of calculating the three-dimensional space coordinates corresponding to each vertex of the head-mounted device in the first posture is the opposite of the normal three-dimensional to two-dimensional projection process, and the inverse matrices of the three matrices are used for calculation).
[0064] S103: When the head mounted device changes to a second posture, based on a transformation matrix of the head mounted device in the second posture and a three-dimensional space coordinate value of each vertex, obtain a projection coordinate of each vertex on the two-dimensional screen.
[0065] It should be noted that the transformation matrix is used to convert the three-dimensional view captured by the head-mounted device in the second posture into a projection view displayed on the two-dimensional screen of the head-mounted device, and may include a model matrix, a view matrix and a projection matrix in the second posture.
[0066] The embodiment of the present application generates a reprojected image in a head movement scenario, so the three-dimensional space coordinate values corresponding to the plane mesh vertices remain unchanged, but because the head posture changes (the virtual camera moves), the position of the three-dimensional space coordinates projected onto the two-dimensional screen will change. Specifically, based on the transformation matrix (model matrix, view matrix, and projection matrix) of the head-mounted device in the second posture, combined with the three-dimensional space coordinates corresponding to each vertex of the head-mounted device in the first posture calculated in the previous step, the three-dimensional space coordinate value of each vertex can be projected onto the two-dimensional screen of the head-mounted device to obtain the projection coordinates of each vertex on the two-dimensional screen.
[0067] S104: updating the two-dimensional screen coordinates of each vertex of the plane mesh according to the projection coordinates of each vertex.
[0068] In an embodiment of the present application, the two-dimensional screen coordinates of each vertex of the plane mesh can be updated according to the projection coordinates of each vertex, that is, the projection coordinates of each vertex on the two-dimensional screen calculated when the head-mounted device changes to the second posture are used to update the two-dimensional screen coordinates of each vertex of the plane mesh obtained when the head-mounted device is in the first posture.
[0069] In the technical solution provided in the embodiment of the present application, each vertex of the plane mesh can be projected to the two-dimensional screen coordinates on the two-dimensional screen and updated to the projection coordinates (including three coordinate values of x, y, and z, where z represents the depth value). The x and y of the projection coordinates can be used to replace the x and y coordinates of the two-dimensional screen coordinates of the vertices of the plane mesh projected to the two-dimensional screen, and the depth value of the projection coordinates can be used to replace the z coordinates of the two-dimensional screen coordinates of the vertices of the plane mesh projected to the two-dimensional screen, thereby completing the movement of the projection position of the plane mesh vertex on the two-dimensional screen. The schematic diagram of the plane mesh mapped on the two-dimensional screen after reprojection is shown in FIG. Figure 3 shown.
[0070] In an optional implementation, the two-dimensional screen coordinates include: two-dimensional plane coordinates and depth values; the projection coordinates include: two-dimensional plane coordinates and depth values; the updating of the two-dimensional screen coordinates of each vertex of the plane mesh according to the projection coordinates of each vertex may include: when the two-dimensional plane coordinates of multiple vertices are the same, updating the multiple vertices in the plane mesh to the vertices with the smallest depth value among the multiple vertices. In practical applications, during the movement of the two-dimensional screen projection coordinates of the vertices of the plane mesh, different vertices may move to the same position, and it is necessary to decide which vertex to display. Since the plane mesh is parallel to the near clipping plane, its own z value remains unchanged in the screen coordinate system after a series of matrix transformations, and each vertex is the same. However, it represents information in a three-dimensional scene, so the z value of the coordinate value of the mesh vertex corresponding to the three-dimensional space coordinate projected to the two-dimensional plane at the current time can be replaced by the z value of the plane mesh coordinate projected to the two-dimensional plane after the projection transformation. In the depth test stage of plane mesh drawing, a depth test can be performed to display the vertex color with a smaller depth value. The embodiments of the present application make full use of the conversion process between two-dimensional and three-dimensional projections, the characteristics of the plane grid and the rendering pipeline flow, and solve the problem of vertices overlapping each other without adding additional calculations.
[0071] S105: Based on the first frame image, perform fragment shading rendering on the updated plane mesh to obtain a reprojected second frame image.
[0072] In the embodiment of the present application, the fragment shader of the plane mesh drawing can be entered, and the previous frame of real rendering image (first frame image) is sampled according to the UV value of each fragment, and then the updated plane mesh is fragment shaded and rendered to obtain the reprojected second frame image (the spatial position reprojection result is as follows: Figure 4 shown).
[0073] In an optional implementation, performing fragment shading rendering on the updated plane mesh based on the first frame image to obtain a reprojected second frame image may include the following steps:
[0074] Step 1, obtaining the updated two-dimensional screen coordinates of multiple fragments of the planar network;
[0075] Step 2, sampling the color value of the first frame image based on the two-dimensional screen coordinates of each fragment;
[0076] Step 3: Based on the sampling result, each of the fragments is shaded and rendered to obtain the reprojected second frame image.
[0077] In an embodiment of the present application, the two-dimensional screen coordinates of multiple fragments of the plane network can be obtained, and then based on the two-dimensional screen coordinates of each fragment, the color value of the first frame image is sampled, and then based on the sampling result, each fragment is shaded and rendered to obtain a reprojected second frame image. Optionally, this process can be implemented using a rendering pipeline.
[0078] In an optional implementation, obtaining the projection coordinate values of the plurality of fragments of the planar network includes:
[0079] Performing rasterization processing on the updated plane grid to obtain a plurality of fragments;
[0080] Based on the position of each fragment in the plane grid, the projection coordinates of each fragment are obtained.
[0081] In an embodiment of the present application, the updated plane grid can be rasterized to obtain multiple fragments. Then, the projection coordinates of each fragment can be obtained based on the position of each fragment in the plane grid. In addition, even if no new vertex moves to the previous position after the vertex moves, since the rasterization process will interpolate the discrete vertices to generate continuous pixels, the pixels will inevitably have color values when the texture is sampled, that is, the color of the pixel after reprojection is continuous and no holes are formed, which can effectively solve the hole problem of the reprojection result of the traditional solution.
[0082] In the related art, it is very difficult to move pixels in the fragment shader of a conventional rendering pipeline. The embodiment of the present application utilizes the process characteristics of the rendering pipeline itself to simulate this process in the vertex shader. Figure 5 A planar mesh can be created that is exactly the same resolution as the original real-world rendered image (e.g. Figure 2 As shown). The purpose of this is to simulate pixels through geometric vertices in the vertex shader (unlike images, the use of geometric vertices is very flexible, so the performance can be improved by reducing the resolution of this plane mesh. Although the reduction in resolution will cause a reduction in the reprojection effect, since the GPU can automatically interpolate and generate fragments (i.e. pixels) during the rasterization stage between the vertex shader and the fragment shader, the ability of automatic interpolation of rasterization can be used to improve the resolution, and rasterization is an inevitable stage of the rendering pipeline without adding additional operations. Therefore, the reduction in the resolution of the plane mesh will not cause a significant decrease in the final effect. According to experience, a good effect can still be achieved when the resolution (length and width) is reduced to one twentieth of the original. The embodiment of the present application can fully achieve a balance between effect and performance by flexibly setting the vertex mesh resolution. Compared with the traditional solution, it can reduce memory usage, improve algorithm performance, and effectively solve the hole problem of the reprojection result of the traditional solution.
[0083] In an embodiment of the present application, a plane mesh including multiple vertices corresponding to the first frame image rendered by the head-mounted device in the first pose can be created, and then the three-dimensional space coordinate value corresponding to each vertex of the head-mounted device in the first pose can be obtained based on the depth value of the pixel corresponding to each vertex in the first frame image; when the head-mounted device changes to the second pose, the projection coordinates of each vertex on the two-dimensional screen are obtained based on the transformation matrix of the head-mounted device in the second pose and the three-dimensional space coordinate value of each vertex; then the two-dimensional screen coordinates of each vertex of the plane mesh are updated according to the projection coordinates of each vertex; finally, based on the first frame image, the updated plane mesh is fragment shaded and rendered to obtain the reprojected second frame image. Compared with the traditional solution, the method provided in the embodiment of the present application can reduce memory usage, improve algorithm performance, and further enhance the user's XR experience.
[0084] Figure 6 Another reprojection display method provided by an exemplary embodiment of the present application is shown. The method 600 can be executed by a terminal device, which can be an extended reality device, such as an XR helmet, XR glasses, etc., or the terminal device can also be a personal terminal device such as a mobile phone or a computer connected to the extended reality device, which is not limited in the embodiments of the present application. Figure 6 As shown, the method mainly includes the following steps:
[0085] S601: Create plane mesh vertex index;
[0086] S602: Rendering the Nth frame image A;
[0087] S603: Create plane mesh vertices in the vertex shader;
[0088] S604: Sampling to obtain pixel depth values in image A corresponding to the current plane mesh vertex;
[0089] S605: Calculate and obtain the three-dimensional coordinates of the pixels in the image A corresponding to the vertices of the current plane mesh;
[0090] S606: Calculate and obtain the two-dimensional coordinates of the current plane mesh vertex on the screen corresponding to the N+1th frame;
[0091] S607: transforming the plane mesh vertices into coordinates of the screen space for replacement;
[0092] S608: Have all plane mesh vertices been processed? If so, proceed to S609; if not, proceed to the next plane mesh vertex and return to S603;
[0093] S609: Enter the fragment shader to perform sampling and complete the rendering of the N+1th frame image B.
[0094] In the vertex shader, the depth value of the current vertex can be obtained by sampling the depth buffer corresponding to the previous frame of the real rendering image through the uv coordinates of the current vertex. Then use the projection matrix to transform the plane mesh vertex to the two-dimensional screen coordinates first, and then combine the model matrix, view matrix and projection matrix of the corresponding time to calculate the three-dimensional coordinates corresponding to the vertex in the three-dimensional scene through matrix transformation. Then combine the three-dimensional coordinates with the model matrix, view matrix and projection matrix at the current time to calculate the coordinate value of the vertex projected on the two-dimensional screen at the current posture (this coordinate is three-dimensional, and z represents depth). Then replace the x and y coordinates of the plane mesh vertex transformed to the x and y coordinates of the coordinates on the two-dimensional screen with the z coordinate of the plane mesh vertex transformed to the two-dimensional screen, and replace the z coordinate of the plane mesh vertex transformed to the z coordinate of the coordinates on the two-dimensional screen, completing the movement of the projection position of the plane mesh vertex on the two-dimensional screen. Through the corresponding ratio of the screen coordinates and the plane mesh, calculate its latest position on the plane mesh. Then move the plane mesh vertex to this position (the z value remains unchanged, only the xy value is changed). Then, in the fragment shader of the plane mesh drawing, the real rendered image of the previous frame is sampled, and the final rendering result is the target image after reprojection (see the reprojection process). Figure 4 : Figure 4 The rendered image at the camera position of the previous frame is A, the rendered image after reprojection of the current camera position is B, and the projection positions of the two positions in the three-dimensional space in image A and the positions in image B after reprojection are shown in the figure).
[0095] The reprojection display method provided in the embodiment of the present application can be executed by a reprojection display device. In the embodiment of the present application, the reprojection display device provided in the embodiment of the present application is described by taking the reprojection display method executed by the reprojection display device as an example.
[0096] Figure 7 The structure diagram of a re-projection display device provided by an exemplary embodiment of the present application is shown. The re-projection display device can be implemented as follows: Figure 1 or Figure 6 All or part of the content in the illustrated embodiment, the reprojection display device includes: a creation module 701, a first acquisition module 702, a second acquisition module 703, an update module 704 and a coloring module 705.
[0097] In an embodiment of the present application, a creation module 701 is used to create a plane mesh corresponding to a first frame image rendered by an extended reality head-mounted device in a first pose, wherein the plane mesh includes multiple vertices, each of the vertex corresponds to a pixel of the first frame image, and the storage data of the first frame image includes a depth value of each pixel; a first acquisition module 702 is used to obtain a three-dimensional space coordinate value corresponding to each vertex of the head-mounted device in the first pose based on the depth value of the pixel corresponding to each vertex in the first frame image; a second acquisition module 703, when the head-mounted device changes to a second pose, obtains the projection coordinates of each vertex on the two-dimensional screen based on the transformation matrix of the head-mounted device in the second pose and the three-dimensional space coordinate value of each vertex; an update module 704 is used to update the two-dimensional screen coordinates of each vertex of the plane mesh according to the projection coordinates of each vertex; a shading module 705 is used to perform fragment shading rendering on the updated plane mesh based on the first frame image to obtain a reprojected second frame image.
[0098] Optionally, when the creation module 701 is used to create the plane mesh corresponding to the first frame image rendered by the extended reality head mounted device in the first pose, it is specifically used to:
[0099] A plane grid is created that is consistent with a resolution of the first frame of image, wherein the plane grid is parallel to a near clipping plane of the head mounted device, and a size of the plane grid is the same as a two-dimensional screen of the head mounted device.
[0100] Optionally, when the first acquisition module 702 is used to acquire the three-dimensional space coordinate value corresponding to each vertex of the head mounted device in the first posture based on the depth value of the pixel corresponding to each vertex in the first frame image, it is specifically used to:
[0101] Obtaining a depth value corresponding to each vertex in the first frame image;
[0102] Based on the projection relationship between the plane grid and the two-dimensional screen, obtaining the two-dimensional screen coordinates of the vertex projected onto the two-dimensional screen;
[0103] Based on the two-dimensional screen coordinates of the vertices projected onto the two-dimensional screen and the transformation matrix of the head mounted device in the first posture, the three-dimensional space coordinates corresponding to each vertex of the head mounted device in the first posture are obtained.
[0104] Optionally, when the updating module 704 is used to update the two-dimensional screen coordinates of each vertex of the plane mesh according to the projection coordinates of each vertex, it is specifically used to:
[0105] When the two-dimensional plane coordinates of a plurality of vertices are the same, the plurality of vertices in the plane mesh are updated to vertices having the smallest depth value among the plurality of vertices.
[0106] Optionally, when the shading module 705 is used to perform fragment shading rendering on the updated plane mesh based on the first frame image to obtain the reprojected second frame image, it is specifically used to:
[0107] Obtaining updated two-dimensional screen coordinates of a plurality of fragments of the planar network;
[0108] Based on the two-dimensional screen coordinates of each fragment, sampling the color value of the first frame of image;
[0109] Based on the sampling result, each of the fragments is shaded and rendered to obtain the reprojected second frame image.
[0110] Optionally, when the coloring module 705 is used to obtain the projection coordinate values of the plurality of fragments of the plane network, it is specifically used to:
[0111] Performing rasterization processing on the updated plane grid to obtain a plurality of fragments;
[0112] Based on the position of each fragment in the plane grid, the projection coordinates of each fragment are obtained.
[0113] The re-projection display device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.
[0114] The re-projection display device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
[0115] The reprojection display device provided in the embodiment of the present application can achieve Figure 1 , Figure 6 To avoid repetition, the various processes implemented by the method embodiment are not described here.
[0116] Alternatively, if Figure 8 As shown, an embodiment of the present application further provides an electronic device 800, including a processor 801 and a memory 802, wherein the memory 802 stores a program or instruction that can be executed on the processor 801, and when the program or instruction is executed by the processor 801, each step of the above-mentioned reprojection display method is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0117] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.
[0118] Fig. 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.
[0119] The electronic device 900 includes but is not limited to: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, and a processor 910 and other components.
[0120] Those skilled in the art will appreciate that the electronic device 900 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 910 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.
[0121] The processor 910 is used to create a plane mesh corresponding to a first frame image rendered by the extended reality head mounted device in a first pose, wherein the plane mesh includes a plurality of vertices, each of the vertices corresponds to a pixel of the first frame image, and the storage data of the first frame image includes a depth value of each pixel;
[0122] Processor 910, configured to obtain a three-dimensional space coordinate value corresponding to each vertex of the head mounted device in the first posture based on a depth value of a pixel corresponding to each vertex in the first frame of image;
[0123] Processor 910, configured to obtain, when the head mounted device changes to a second posture, a projection coordinate of each vertex on the two-dimensional screen based on a transformation matrix of the head mounted device in the second posture and a three-dimensional space coordinate value of each vertex;
[0124] Processor 910, configured to update the two-dimensional screen coordinates of each vertex of the plane mesh according to the projection coordinates of each vertex;
[0125] The processor 910 is used to perform fragment shading rendering on the updated plane mesh based on the first frame image to obtain a reprojected second frame image.
[0126] Optionally, when the processor 910 is used for creating a plane mesh corresponding to a first frame image rendered by the augmented reality head mounted device at a first pose, the processor 910 is specifically used for:
[0127] A plane grid is created that is consistent with a resolution of the first frame of image, wherein the plane grid is parallel to a near clipping plane of the head mounted device, and a size of the plane grid is the same as a two-dimensional screen of the head mounted device.
[0128] Optionally, when the processor 910 is used to obtain the three-dimensional space coordinate value corresponding to each vertex of the head mounted device in the first posture based on the depth value of the pixel corresponding to each vertex in the first frame image, it is specifically used to:
[0129] Obtaining a depth value corresponding to each vertex in the first frame image;
[0130] Based on the projection relationship between the plane grid and the two-dimensional screen, obtaining the two-dimensional screen coordinates of the vertex projected onto the two-dimensional screen;
[0131] Based on the two-dimensional screen coordinates of the vertices projected onto the two-dimensional screen and the transformation matrix of the head mounted device in the first posture, the three-dimensional space coordinates corresponding to each vertex of the head mounted device in the first posture are obtained.
[0132] Optionally, when the processor 910 is used to update the two-dimensional screen coordinates of each vertex of the plane mesh according to the projection coordinates of each vertex, it is specifically used to:
[0133] When the two-dimensional plane coordinates of a plurality of vertices are the same, the plurality of vertices in the plane mesh are updated to vertices having the smallest depth value among the plurality of vertices.
[0134] Optionally, when the processor 910 is used to perform fragment shading rendering on the updated plane mesh based on the first frame image to obtain a reprojected second frame image, it is specifically used to:
[0135] Obtaining updated two-dimensional screen coordinates of a plurality of fragments of the planar network;
[0136] Based on the two-dimensional screen coordinates of each fragment, sampling the color value of the first frame of image;
[0137] Based on the sampling result, each of the fragments is shaded and rendered to obtain the reprojected second frame image.
[0138] Optionally, when the processor 910 is used to obtain the projection coordinate values of the plurality of fragments of the planar network, it is specifically used to:
[0139] Performing rasterization processing on the updated plane grid to obtain a plurality of fragments;
[0140] Based on the position of each fragment in the plane grid, the projection coordinates of each fragment are obtained.
[0141] It should be understood that in the embodiment of the present application, the input unit 904 may include a graphics processor (Graphics Processing Unit, GPU) 9041 and a microphone 9042, and the graphics processor 9041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 906 may include a display panel 9061, and the display panel 9061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 907 includes a touch panel 9071 and at least one of other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include two parts: a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.
[0142] The memory 909 can be used to store software programs and various data. The memory 909 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 909 may include a volatile memory or a non-volatile memory, or the memory 909 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 909 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.
[0143] The processor 910 may include one or more processing units; optionally, the processor 910 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 910.
[0144] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned reprojection display method is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0145] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0146] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned reprojection display method, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0147] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0148] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned reprojection display method, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0149] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0150] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0151] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A reprojection display method, characterized in that: include: Creating a plane mesh corresponding to a first frame of image rendered by the head mounted device at a first pose, wherein the plane mesh includes a plurality of vertices, each of the vertices corresponds to a pixel of the first frame of image, and the stored data of the first frame of image includes a depth value of each pixel; Based on the depth value of the pixel corresponding to each vertex in the first frame image, obtaining the three-dimensional space coordinate value corresponding to each vertex of the head mounted device in the first posture; When the head mounted device changes to a second posture, based on a transformation matrix of the head mounted device in the second posture and a three-dimensional space coordinate value of each vertex, obtaining a projection coordinate of each vertex on the two-dimensional screen; Update the two-dimensional screen coordinates of each vertex of the plane mesh according to the projection coordinates of each vertex; Based on the first frame image, fragment shading rendering is performed on the updated plane mesh to obtain a reprojected second frame image.
2. The method according to claim 1, characterized in that The transformation matrix is used to transform the three-dimensional view acquired by the head mounted device in the second posture into a projection view displayed on the two-dimensional screen of the head mounted device.
3. The method according to claim 1 or 2, characterized in that: The step of creating a plane mesh corresponding to a first frame of an image rendered by the head mounted device at a first pose includes: A plane grid is created that is consistent with a resolution of the first frame of image, wherein the plane grid is parallel to a near clipping plane of the head mounted device, and a size of the plane grid is the same as a two-dimensional screen of the head mounted device.
4. The method according to any one of claims 1 to 3, characterized in that: The step of obtaining the three-dimensional space coordinate value corresponding to each vertex of the head mounted device in the first posture based on the depth value of the pixel corresponding to each vertex in the first frame image includes: Obtaining a depth value corresponding to each vertex in the first frame image; Based on the projection relationship between the plane grid and the two-dimensional screen, obtaining the two-dimensional screen coordinates of the vertex projected onto the two-dimensional screen; Based on the two-dimensional screen coordinates of the vertices projected onto the two-dimensional screen and the transformation matrix of the head mounted device in the first posture, the three-dimensional space coordinates corresponding to each vertex of the head mounted device in the first posture are obtained.
5. The method according to any one of claims 1 to 4, characterized in that: The two-dimensional screen coordinates include: two-dimensional plane coordinates and depth values; the projection coordinates include: two-dimensional plane coordinates and depth values; the updating of the two-dimensional screen coordinates of each vertex of the plane mesh according to the projection coordinates of each vertex includes: When the two-dimensional plane coordinates of a plurality of vertices are the same, the plurality of vertices in the plane mesh are updated to vertices having the smallest depth value among the plurality of vertices.
6. The method according to any one of claims 1 to 5, characterized in that: The method of performing fragment shading rendering on the updated plane mesh based on the first frame image to obtain a reprojected second frame image includes: Obtaining updated two-dimensional screen coordinates of a plurality of fragments of the planar network; Based on the two-dimensional screen coordinates of each fragment, sampling the color value of the first frame of image; Based on the sampling result, each of the fragments is shaded and rendered to obtain the reprojected second frame image.
7. The method according to claim 6, characterized in that The obtaining of projection coordinate values of a plurality of fragments of the planar network includes: Performing rasterization processing on the updated plane grid to obtain a plurality of fragments; Based on the position of each fragment in the plane grid, the projection coordinates of each fragment are obtained.
8. A reprojection display device, characterized in that: include: A creation module, configured to create a plane mesh corresponding to a first frame image rendered by the head mounted device in a first posture, wherein the plane mesh includes a plurality of vertices, each of the vertices corresponds to a pixel of the first frame image, and the storage data of the first frame image includes a depth value of each pixel; A first acquisition module, configured to acquire a three-dimensional space coordinate value corresponding to each vertex of the head mounted device in a first posture based on a depth value of a pixel corresponding to each vertex in the first frame image; A second acquisition module is used to obtain the projection coordinates of each vertex on the two-dimensional screen based on the transformation matrix of the head mounted device in the second posture and the three-dimensional space coordinate value of each vertex when the head mounted device changes to a second posture; An updating module, used for updating the two-dimensional screen coordinates of each vertex of the plane mesh according to the projection coordinates of each vertex; A shading module is used to perform fragment shading rendering on the updated plane mesh based on the first frame image to obtain a reprojected second frame image.
9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the reprojection display method according to any one of claims 1 to 7 are implemented.
10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the re-projection display method according to any one of claims 1 to 7 are implemented.
