Scene rendering method, device, AR device and storage medium

Abstract

This application relates to the field of rendering technology and provides a scene rendering method, apparatus, AR device, and storage medium. The method comprises: obtaining first depth data corresponding to a related scene, where the first depth data is depth data in a coordinate system corresponding to a user's observation perspective; obtaining second depth data corresponding to a target scene; and rendering the target scene based on the first depth data and the second depth data, where the depth of objects in the rendered target scene is less than the depth of objects in the related scene at the corresponding position. Embodiments of the present application enhance the realism of scene rendering.

Description

Technical Field

[0001] The present application relates to the field of rendering technology, and in particular to a scene rendering method, apparatus, AR device, and storage medium. Background Art

[0002] Rendering technology is the process of converting three-dimensional scenes or models into two-dimensional images. It is widely used in fields such as computer graphics, game development, film and television production, virtual reality (VR), and augmented reality (AR). Taking AR scene rendering as an example, it superimposes and merges the virtual environment with the real environment, presenting it to the user for a realistic visual experience. During the rendering process, if the front-to-back position relationship between objects in the virtual environment and the real environment is incorrect, the occlusion relationship between the virtual and real objects will be inaccurate, resulting in rendering distortion, causing a sense of visual unreality for the user and affecting the visual experience. Summary of the Invention

[0003] This application provides a scene rendering method, apparatus, AR device and storage medium, aiming to solve the rendering distortion problem and enhance the realism of scene rendering.

[0004] To achieve the above objectives, the present application provides a scene rendering method, which includes:

[0005] Acquire first depth data corresponding to the relevant scene, where the first depth data is depth data in a coordinate system corresponding to the user's observation perspective;

[0006] Acquire second depth data corresponding to the target scene;

[0007] The target scene is rendered based on the first depth data and the second depth data, and the depth of the rendered object in the target scene is less than the depth of the object in the related scene at the corresponding position.

[0008] In addition, to achieve the above-mentioned purpose, the present application also provides a scene rendering device, which includes a memory and a processor;

[0009] The memory is used to store computer programs;

[0010] The processor is used to execute the computer program and implement the steps of the scene rendering method as described above when executing the computer program.

[0011] In addition, to achieve the above-mentioned purpose, the present application also provides an AR device, which includes the scene rendering device as described above.

[0012] In addition, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned scene rendering method are implemented.

[0013] The present application discloses a scene rendering method, apparatus, AR device and storage medium, which obtain first depth data corresponding to a related scene, wherein the first depth data is depth data in a coordinate system corresponding to a user's observation perspective, and obtain second depth data corresponding to a target scene. Based on the first depth data and the second depth data, the front-to-back position relationship between objects in the target scene and objects in the related scene is accurately judged, and the target scene is rendered. The depth of the rendered object in the target scene is smaller than the depth of the object in the related scene at the corresponding position, thereby avoiding the problem of inaccurate occlusion relationship between the object in the target scene and the objects in the related scene, thereby enhancing the realism effect of the target scene rendering. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0015] Figure 1 A schematic diagram of a flow chart of a scene rendering method provided in an embodiment of the present application;

[0016] Figure 2 A schematic diagram of a sub-process of a scene rendering method provided in an embodiment of the present application;

[0017] Figure 3 A schematic diagram of a flow chart of another scene rendering method provided in an embodiment of the present application;

[0018] Figure 4 A schematic diagram of a sub-process of another scene rendering method provided in an embodiment of the present application;

[0019] Figure 5 A schematic diagram of a sub-process of another scene rendering method provided in an embodiment of the present application;

[0020] Figure 6 A schematic diagram of a process for rendering a virtual scene under a mobile perspective provided in an embodiment of the present application;

[0021] Figure 7 A schematic block diagram of a scene rendering device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0024] It should be understood that the terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0025] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0026] The embodiments of the present application provide a scene rendering method, apparatus, AR device, and storage medium for enhancing the realism of scene rendering.

[0027] See also Figure 1 , Figure 1 This is a flow chart of a scene rendering method provided by an embodiment of the present application. This method can be applied to a scene rendering device or an AR device, and the application scenario of this method is not limited in this application.

[0028] like Figure 1 As shown, the scene rendering method specifically includes steps S101 to S103.

[0029] S101: Acquire first depth data corresponding to a relevant scene, where the first depth data is depth data in a coordinate system corresponding to a user's observation perspective.

[0030] Among them, the related scene is a scene that does not need to be rendered, and the related scene will be superimposed and merged with the scene to be rendered. For example, the related scene can be a real scene in the real physical environment that the user is currently in. It should be noted that the related scene can also be other scenes, which is not limited in the embodiments of this application.

[0031] Generally speaking, the relevant scene contains one or more objects, for example, a real scene contains multiple real objects. Depth data of the objects in the relevant scene in the coordinate system corresponding to the user's observation perspective is obtained. For ease of description, the depth data in the coordinate system corresponding to the user's observation perspective is referred to as first depth data below. Because the first depth data is the depth data in the coordinate system corresponding to the user's observation perspective, the first depth data can accurately correspond to the user's observation perspective. Therefore, subsequent determination of occlusion relationships between objects based on the first depth data is more accurate and reliable.

[0032] In some embodiments, as Figure 2 As shown, step S101 may include sub-step S1011 and sub-step S1012.

[0033] S1011. Acquire scene depth data corresponding to the relevant scene, and acquire a transformation matrix between a coordinate system corresponding to the device and a coordinate system corresponding to the user's observation perspective, where the scene depth data is depth data in the coordinate system corresponding to the device;

[0034] S1012: Obtain first depth data based on the scene depth data and the transformation matrix.

[0035] For example, a depth map of a relevant scene may be captured by a device such as a depth camera, and scene depth data of objects in the relevant scene in a coordinate system corresponding to the device may be obtained based on the depth map. Alternatively, scene depth data of objects in the relevant scene in a coordinate system corresponding to the device may be acquired by a sensor device such as a depth detector. This application does not limit the method for obtaining scene depth data corresponding to the relevant scene.

[0036] It is understood that since scene depth data is obtained by devices such as depth cameras, depth cameras, and depth detectors, the scene depth data is depth data in the coordinate system corresponding to the corresponding device. For example, if the scene depth data is obtained by taking a depth image from a depth camera, the scene depth data is depth data in the depth camera coordinate system.

[0037] In actual applications, the coordinate system corresponding to the device is likely to be inconsistent with the coordinate system corresponding to the user's observation perspective. In order to make the depth data corresponding to the objects in the relevant scene accurately correspond to the user's observation perspective, the obtained scene depth data is converted into a coordinate system through the transformation matrix between the coordinate system corresponding to the device and the coordinate system corresponding to the user's observation perspective, so as to obtain the first depth data corresponding to the user's observation perspective. Compared with using the third depth data corresponding to the depth camera and other devices to determine the occlusion relationship between objects, the first depth data corresponding to the user's observation perspective is used to determine the occlusion relationship between objects, and the determination result is more consistent with the user's visual observation.

[0038] Exemplarily, the transformation matrix between the device coordinate system and the user's perspective coordinate system includes a rotation matrix, a projection matrix, and a translation vector. The rotation matrix describes the rotation relationship between the device coordinate system and the user's perspective coordinate system. In computer graphics, the projection matrix is ​​often used to project points in three-dimensional space onto a two-dimensional plane. The projection matrix in this application is used to perform a suitable projection transformation on scene depth data. The translation vector describes the positional offset of the device coordinate system relative to the user's perspective coordinate system.

[0039] For example, the transformation matrix between the device coordinate system and the user's viewing angle coordinate system is shown in formula (1):

[0040] Tobscam=R·P+t(1)

[0041] Where Tobscam is the transformation matrix between the coordinate system of the device and the coordinate system of the user's observation angle, R is the rotation matrix, P is the projection matrix, and t is the translation vector.

[0042] It should be noted that the transformation matrix between the device's corresponding coordinate system and the user's observation perspective's corresponding coordinate system can be currently established or pre-established and saved. Currently, you only need to call it to obtain the transformation matrix between the device's corresponding coordinate system and the user's observation perspective's corresponding coordinate system.

[0043] Based on the transformation matrix between the device corresponding coordinate system and the user observation perspective corresponding coordinate system, the scene depth data in the device corresponding coordinate system is converted into first depth data in the user observation perspective corresponding coordinate system.

[0044] For example, the first depth data in the coordinate system corresponding to the user's observation angle is calculated by formula (2):

[0045] Dobsdepth=Tobscam·Dcamdepth(2)

[0046] Among them, Tobscam is the transformation matrix between the coordinate system corresponding to the device and the coordinate system corresponding to the user's observation perspective, Dcamdepth is the scene depth data in the coordinate system corresponding to the device, and Dobsdepth is the first depth data in the coordinate system corresponding to the user's observation perspective.

[0047] For example, scene depth data includes a depth map matrix in the depth camera coordinate system obtained based on the depth map. Based on this depth map matrix and the transformation matrix between the depth camera coordinate system and the coordinate system corresponding to the user's observation angle, each pixel in the depth map is remapped to the coordinate system corresponding to the user's observation angle, resulting in a remapped depth map that adapts to the user's observation angle. This allows subsequent rendering and other operations to accurately handle object occlusion relationships based on the remapped depth map, thereby improving the realism and interactivity of scene rendering.

[0048] Exemplarily, obtaining the first depth data based on the scene depth data and the transformation matrix includes: converting the scene depth data into a coordinate system corresponding to the user's observation perspective based on the transformation matrix and bilinear interpolation to obtain the first depth data.

[0049] The transformation matrix describes the conversion relationship from the device coordinate system to the user's perspective coordinate system. Bilinear interpolation calculates the depth value at the new coordinate position by taking a weighted average of the depth values ​​at adjacent coordinate positions. Each pixel in the depth map is remapped to the user's perspective coordinate system. There may be pixels whose new coordinates in the user's perspective coordinate system are not integers. By using bilinear interpolation to calculate the weighted average of the depth values ​​corresponding to the pixels at their adjacent coordinate positions, this is used as the depth value corresponding to the pixel, thereby obtaining a more accurate and reliable remapped depth map.

[0050] It should be noted that in addition to the above-mentioned method of obtaining the first depth data through the transformation matrix, the first depth data can also be obtained through other methods, for example, obtaining the first depth data through point cloud registration. This application does not specifically limit the method of obtaining the first depth data.

[0051] S102: Acquire second depth data corresponding to the target scene.

[0052] The target scene is a scene to be rendered, for example, the target scene may be a virtual scene. It should be noted that the target scene may also be other scenes, which is not limited in the embodiments of the present application.

[0053] The target scene also includes one or more objects, for example, the virtual scene includes multiple virtual objects. For the target scene, depth data corresponding to the objects in the target scene is obtained, which is referred to as second depth data below for ease of description.

[0054] For example, taking a virtual scene as an example, based on the perspective projection matrix, the vertex coordinates of the virtual object in the virtual scene are multiplied by the perspective projection matrix, and the vertices of the virtual object are appropriately projected and transformed to obtain the depth value of the vertex of the virtual object in the virtual space. This depth value represents the position of the vertex of the virtual object relative to the near clipping plane and the far clipping plane, thereby determining the depth data of the virtual object in the virtual scene, that is, obtaining the second depth data.

[0055] It should be noted that step S102 may be executed after step S101, step S102 may also be executed before step S101, or step S102 and step S101 may be executed simultaneously, and there is no limitation on the order in which step S102 and step S101 are executed.

[0056] S103: Rendering a target scene based on the first depth data and the second depth data, wherein the depth of objects in the rendered target scene is smaller than the depth of objects in the related scene at the corresponding position.

[0057] By comparing the first depth data and the second depth data, the front-to-back positional relationship between the relevant scene object and the target scene object at the corresponding position is determined. If the depth of the target scene object is less than the depth of the relevant scene object at the corresponding position, that is, the target scene object is in front and the relevant scene object is behind, and the target scene object is not blocked, then the corresponding pixel of the target scene object is displayed. Conversely, if the depth of the target scene object is greater than or equal to the depth of the relevant scene object at the corresponding position, that is, the target scene object is behind and the relevant scene object is in front, and the target scene object is blocked, then the corresponding pixel of the target scene object is hidden.

[0058] By obtaining first depth data corresponding to the relevant scene that is adapted to the user's observation perspective, and comparing the first depth data with the second depth data corresponding to the target scene, the depth comparison result is more accurate and reliable, avoiding the problem of inaccurate occlusion relationship between the target scene object and the related scene object, thereby improving the realism and interactivity of the target scene rendering.

[0059] For example, assuming the relevant scene is a real scene and the target scene is a virtual scene, the first depth data includes a first depth value corresponding to a real object in the real scene, and the second depth data includes a second depth value corresponding to a virtual object in the virtual scene. Rendering the target scene based on the first depth data and the second depth data includes comparing the second depth value of the virtual object with the first depth value of the real object at the corresponding location, and rendering the pixel corresponding to the virtual object if the second depth value is less than the first depth value.

[0060] By comparing the second depth value of the virtual object in the virtual scene with the first depth value of the real object in the corresponding real scene, if the second depth value is less than the first depth value, that is, the virtual object is in front and the real object is behind, and the virtual object is not blocked, then the pixel points corresponding to the virtual object are rendered and displayed. Conversely, if the second depth value is greater than or equal to the first depth value, that is, the virtual object is behind and the real object is in front, and the virtual object is blocked, then the pixel points corresponding to the virtual object are hidden.

[0061] By comparing the second depth value with the first depth value, the front-to-back position relationship between the virtual object and the real object can be accurately determined, thereby avoiding the problem of inaccurate occlusion relationship between the virtual object and the real object, thereby improving the realism of virtual scene rendering.

[0062] In some embodiments, the first depth data and the second depth data are obtained by parallel processing based on different graphics processing unit (GPU) channels.

[0063] For example, a dual-channel processing architecture is constructed in the GPU shader. For example, the dual-channel processing architecture is as follows:

[0064] void main() {float realDepth = texture2D(realDepthTex, uv).r; floatvirtualDepth = computeVirtualDepth();gl_FragColor = (virtualDepth <realDepth)? virtualObj : realObj;}

[0065] The dual-channel processing architecture divides the depth comparison task into two different channels for parallel processing. One channel can be dedicated to processing the second depth data corresponding to the target scene, while the other channel is dedicated to processing the first depth data corresponding to the relevant scene. For example, one channel is dedicated to processing the depth data of virtual objects in a virtual scene, while the other channel is dedicated to processing the depth data of real objects in a real scene.

[0066] Exemplarily, the obtained first depth data and second depth data are cached in a cache area, for example, the first depth data and second depth data are cached in a GPU buffer, and the occlusion relationship between the target scene object and the related scene objects is determined by comparing the first depth data and the second depth data in the GPU buffer.

[0067] Through dual-channel parallel processing, the efficiency of depth data comparison is greatly improved, and the occlusion relationship between the target scene object and related scene objects can be determined in real time, thereby performing rendering operations more efficiently and deciding which pixels should be displayed and which should be hidden.

[0068] It should be noted that in addition to the above-mentioned method of parallel processing through GPU dual channels, the first depth data and the second depth data can also be processed in parallel by other methods, for example, by using CPU multi-threading to simultaneously process the first depth data and the second depth data, thereby improving data processing efficiency.

[0069] In some embodiments, as Figure 3 As shown, step S104 may be included before step S101, and step S101 may include sub-step S1013.

[0070] S104: Determine the user's displacement within a preset time period;

[0071] S1013: If the displacement is greater than a preset displacement threshold, first depth data corresponding to the relevant scene is obtained, where the first depth data is depth data in a coordinate system corresponding to the user's observation perspective.

[0072] In actual applications, the user may be in a fixed perspective, meaning the user does not move, or in a mobile perspective, meaning the user moves. In the fixed perspective, the first depth data in the coordinate system corresponding to the user's observation perspective is constant, so it is sufficient to obtain the first depth data in the coordinate system corresponding to the user's observation perspective once and then perform rendering operations. In the mobile perspective, however, the first depth data in the coordinate system corresponding to the user's observation perspective changes, requiring repeated acquisition of the first depth data in the coordinate system corresponding to the user's observation perspective, dynamic updating of the first depth data, and rendering of the target scene based on the dynamically updated first depth data.

[0073] The first depth data can be dynamically updated by presetting an update cycle and periodically performing the operation of acquiring the first depth data based on the update cycle. The operation of acquiring the first depth data each time can be referred to the above embodiment, so it will not be repeated here.

[0074] By dynamically updating the first depth data, the problem of inaccurate occlusion relationship between the target scene object and related scene objects under a mobile perspective is avoided, thereby improving the realism and interactivity of the target scene rendering under a mobile perspective.

[0075] Considering that if the user only moves very slightly, the change in the first depth data is very small and will not change the occlusion relationship between the target scene object and the related scene objects, there is actually no need to re-acquire the first depth data and update the first depth data.

[0076] Therefore, a preset time period for updating the first depth data is pre-set, and the user's displacement within the preset time period is determined at intervals of the preset time period. It should be noted that the preset time period can be flexibly set according to actual conditions and is not specifically limited in this application.

[0077] Furthermore, a preset displacement threshold is pre-set for determining whether to update the first depth data, for example, the preset displacement threshold is set to 5 cm. It should be noted that the preset displacement threshold can be flexibly set according to actual conditions and is not specifically limited in this application.

[0078] If the user's displacement within a preset duration exceeds a preset displacement threshold, indicating that the user has moved a certain distance, the first depth data corresponding to the relevant scene is acquired to ensure the accuracy of the occlusion relationship between the target scene object and related scene objects. For example, if the user's displacement within a preset duration exceeds a preset displacement threshold, a depth map recalculation is triggered to obtain a remapped depth map that adapts to the user's observation perspective.

[0079] If the user's displacement within the preset time is less than or equal to the preset displacement threshold, it means that the user's movement distance is small and will not change the occlusion relationship between the target scene object and the related scene objects. At this time, the operation of obtaining the first depth data corresponding to the related scene is not performed, thereby avoiding unnecessary depth map calculation process and reducing energy consumption.

[0080] Therefore, whether to re-acquire the first depth data is triggered based on the user's displacement, which not only avoids the problem of misalignment between the target scene object and the related scene objects under the moving perspective, but also reduces energy consumption.

[0081] In some embodiments, as Figure 4 As shown, step S104 may include sub-step S1041 and sub-step S1042.

[0082] S1041. Obtain user posture information corresponding to a preset duration, where the user posture information includes at least one of inertial measurement unit (IMU) data and simultaneous localization and mapping (SLAM) data;

[0083] S1042: Determine displacement based on user posture information.

[0084] For example, the IMU data such as the acceleration and angular velocity of the user's movement can be obtained through the IMU measurement of the AR glasses worn by the user. Based on the IMU data, the user's motion state and position changes can be calculated to obtain the user's displacement within a preset time period.

[0085] For example, images are captured by the camera of AR glasses worn by the user, and through feature extraction, matching and other algorithm processing, a map of the user's surrounding environment is constructed in real time, and the user's own position (such as x, y, z three-dimensional coordinates) and posture (such as rotation angle) in the map are determined. SLAM data, then the user's displacement within a preset time period is determined based on the SLAM data.

[0086] For example, IMU data and SLAM data can be obtained and integrated to give full play to the advantages of both data. IMU data can provide high-frequency motion information, while SLAM data can provide more accurate position and posture information. By integrating these two data, the user's displacement within a preset time period can be calculated more accurately.

[0087] In some embodiments, as Figure 5 As shown, step S1041 may include sub-step S10411 and sub-step S10412.

[0088] S10411. Capturing an image of a relevant scene with a camera device based on a preset refresh rate, wherein the preset duration is determined by the preset refresh rate;

[0089] S10412: Obtain first SLAM data corresponding to the current frame image and second SLAM data corresponding to the previous frame image.

[0090] The camera device includes but is not limited to a camera, a still camera, etc. The preset refresh rate of the camera device is, for example, 100 Hz, but can also be other values, which is not specifically limited in this application.

[0091] For example, by using a 100Hz refresh rate camera on AR glasses worn by a user to capture an image of a relevant scene, based on the current frame image obtained by the capture, SLAM data such as the user's position and posture corresponding to the timestamp of the current frame image can be obtained. For the sake of distinction, this will be referred to as the first SLAM data below. And based on the previous frame image, SLAM data such as the user's position and posture corresponding to the timestamp of the previous frame image can be obtained. For the sake of distinction, this will be referred to as the second SLAM data below.

[0092] The user's displacement within the time period corresponding to the timestamps of two adjacent frames of image can be calculated using the first and second SLAM data. Alternatively, the user's displacement within the preset time period can be more accurately calculated by fusing the first and second SLAM data with the IMU data.

[0093] The following is an example of virtual scene rendering under mobile perspective. Figure 6 As shown in the figure, the rendering process of the virtual scene under the mobile perspective is as follows:

[0094] stepA: start;

[0095] Step B: Use a depth camera to obtain a depth map of the user's real scene;

[0096] Step C: Obtain the transformation matrix between the coordinate system corresponding to the depth camera and the coordinate system corresponding to the user's observation angle;

[0097] stepD: If the user's displacement exceeds a preset displacement threshold (e.g., 5 cm), the transformation matrix is ​​used to remap each pixel in the depth map to the coordinate system corresponding to the user's observation angle, generating a remapped depth map adapted to the user's observation angle.

[0098] stepE: Based on the perspective projection matrix, calculate the depth data of the virtual object in the virtual scene;

[0099] stepF: remaps the depth data of real objects in the real scene and the depth data of virtual objects in the virtual scene corresponding to the depth map through the GPU buffer cache;

[0100] stepG: Based on the depth data, determine the occlusion relationship between the real object in the real scene and the virtual object in the virtual scene, and determine whether the depth of the virtual object is less than the depth of the real object; if so, execute stepH; if not, execute stepI;

[0101] stepH: Renders the pixel corresponding to the virtual object;

[0102] stepI: End.

[0103] By comparing the depth of virtual objects in the virtual scene with the depth of real objects in the real scene, the front-to-back position relationship between the virtual objects in the virtual scene and the real objects in the real scene can be accurately judged, avoiding the problem of inaccurate occlusion relationship between virtual objects and real objects, thereby improving the realism of virtual scene rendering.

[0104] See also Figure 7 , Figure 7It is a schematic block diagram of a scene rendering device provided in an embodiment of the present application. The scene rendering device can be configured in an AR device to execute the aforementioned scene rendering method.

[0105] like Figure 7 As shown, the scene rendering device 200 may include a processor 210 and a memory 220 , wherein the processor 210 and the memory 220 are connected via a bus, such as an I2C (Inter-integrated Circuit) bus.

[0106] Specifically, the processor 210 may be a micro-controller unit (MCU), a central processing unit (CPU), or a digital signal processor (DSP).

[0107] Specifically, the memory 220 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk, etc. The memory 220 stores various computer programs for execution by the processor 210 .

[0108] The processor 210 is configured to run a computer program stored in the memory and implement the following steps when executing the computer program:

[0109] Acquire first depth data corresponding to the relevant scene, where the first depth data is depth data in a coordinate system corresponding to the user's observation perspective;

[0110] Acquire second depth data corresponding to the target scene;

[0111] The target scene is rendered based on the first depth data and the second depth data, and the depth of the rendered object in the target scene is less than the depth of the object in the related scene at the corresponding position.

[0112] In some embodiments, when acquiring the first depth data corresponding to the relevant scene, the processor 210 is configured to implement:

[0113] Obtaining scene depth data corresponding to the relevant scene, and obtaining a transformation matrix between a coordinate system corresponding to a device and a coordinate system corresponding to the user's observation perspective, wherein the scene depth data is depth data in the coordinate system corresponding to the device;

[0114] The first depth data is obtained based on the scene depth data and the transformation matrix.

[0115] In some embodiments, when implementing the obtaining of the first depth data based on the scene depth data and the transformation matrix, the processor 210 is configured to implement:

[0116] Based on the transformation matrix and bilinear interpolation, the scene depth data is converted into a coordinate system corresponding to the user observation perspective to obtain the first depth data.

[0117] In some embodiments, before acquiring the first depth data corresponding to the relevant scene, the processor 210 is configured to:

[0118] Determine the user's movement within a preset time period;

[0119] When acquiring the first depth data corresponding to the relevant scene, the processor 210 is configured to:

[0120] If the displacement is greater than a preset displacement threshold, the first depth data corresponding to the relevant scene is acquired.

[0121] In some embodiments, when determining the displacement of the user within a preset time period, the processor 210 is configured to implement:

[0122] Obtaining user posture information corresponding to the preset duration, the user posture information including at least one of IMU inertial measurement unit data and SLAM synchronous positioning and mapping data;

[0123] The displacement is determined based on the user posture information.

[0124] In some embodiments, when obtaining the user posture information corresponding to the preset duration, the processor 210 is configured to implement:

[0125] capturing an image of the relevant scene using a camera device based on a preset refresh rate, wherein the preset duration is determined by the preset refresh rate;

[0126] Obtain first SLAM data corresponding to the current frame image and second SLAM data corresponding to the previous frame image.

[0127] In some embodiments, the first depth data and the second depth data are obtained by parallel processing based on different GPU graphics processor channels.

[0128] In some embodiments, the related scene is a real scene, the target scene is a virtual scene, the first depth data includes a first depth value corresponding to a real object in the real scene, and the second depth data includes a second depth value corresponding to a virtual object in the virtual scene. When rendering the target scene based on the first depth data and the second depth data, the processor 210 is configured to implement:

[0129] The second depth value of the virtual object is compared with the first depth value of the real object at the corresponding position. If the second depth value is less than the first depth value, the pixel point corresponding to the virtual object is rendered.

[0130] The scene rendering device 200 can execute the scene rendering method provided in the embodiment of the present application, and therefore can achieve the beneficial effects that can be achieved by the scene rendering method provided in the embodiment of the present application. Please refer to the previous embodiment for details and will not be repeated here.

[0131] An embodiment of the present application further provides an AR device, which includes a scene rendering device. The scene rendering device can be Figure 7 Therefore, the AR device can achieve the beneficial effects that can be achieved by the scene rendering method provided in the embodiment of the present application. Please refer to the previous embodiment for details and will not be repeated here.

[0132] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the scene rendering method as described above are implemented.

[0133] The computer-readable storage medium may be an internal storage unit of the scene rendering device or AR device described in the aforementioned embodiment, such as a hard disk or memory of the scene rendering device or AR device. The computer-readable storage medium may also be an external storage device of the scene rendering device or AR device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital card (Secure Digital Card, SD Card), a flash card (Flash Card), etc. equipped on the scene rendering device or AR device.

[0134] Since the computer program stored in the storage medium can execute any scene rendering method provided in the embodiments of the present application, the beneficial effects that can be achieved by any scene rendering method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0135] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0136] The above description is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the scope of protection of the present application.

Claims

1. A scene rendering method, characterized in that: The scene rendering method comprises: Determine the user's movement within a preset time period; If the displacement is greater than a preset displacement threshold, first depth data corresponding to the relevant scene is obtained, where the first depth data is depth data in a coordinate system corresponding to the user's observation angle; Acquire second depth data corresponding to the target scene; Rendering the target scene based on the first depth data and the second depth data, wherein the depth of objects in the rendered target scene is smaller than the depth of objects in the related scene at corresponding positions; The obtaining of first depth data corresponding to the relevant scene includes: Obtaining scene depth data corresponding to the relevant scene, and obtaining a transformation matrix between a coordinate system corresponding to a device and a coordinate system corresponding to the user's observation perspective, wherein the scene depth data is depth data in the coordinate system corresponding to the device; Based on the transformation matrix and bilinear interpolation, the scene depth data is converted into a coordinate system corresponding to the user observation perspective to obtain the first depth data.

2. The scene rendering method according to claim 1, wherein: Determining the user's displacement within a preset time period includes: Obtaining user posture information corresponding to the preset duration, the user posture information including at least one of IMU inertial measurement unit data and SLAM synchronous positioning and mapping data; The displacement is determined based on the user posture information.

3. The scene rendering method according to claim 2, wherein: The obtaining of user posture information corresponding to the preset duration includes: capturing an image of the relevant scene using a camera device based on a preset refresh rate, wherein the preset duration is determined by the preset refresh rate; Obtain first SLAM data corresponding to the current frame image and second SLAM data corresponding to the previous frame image.

4. The scene rendering method according to claim 1, wherein: The first depth data and the second depth data are obtained by parallel processing based on different GPU graphics processor channels.

5. The scene rendering method according to claim 1, wherein: The related scene is a real scene, the target scene is a virtual scene, the first depth data includes a first depth value corresponding to a real object in the real scene, and the second depth data includes a second depth value corresponding to a virtual object in the virtual scene. Rendering the target scene based on the first depth data and the second depth data includes: The second depth value of the virtual object is compared with the first depth value of the real object at the corresponding position. If the second depth value is less than the first depth value, the pixel point corresponding to the virtual object is rendered.

6. A scene rendering device, characterized in that: The scene rendering device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and implement the steps of the scene rendering method according to any one of claims 1 to 5 when executing the computer program.

7. An AR device, characterized in that: The AR device includes the scene rendering apparatus according to claim 6.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the scene rendering method according to any one of claims 1 to 5 are implemented.

Images