Scene rendering method, device, equipment, computer readable storage medium and product

Abstract

Embodiments of the present disclosure provide a scene rendering method, device, equipment, computer readable storage medium and product, the method comprising: in response to a three-dimensional object display operation triggered by a user, obtaining a target three-dimensional object to be displayed and an image to be processed corresponding to a target scene associated with the target three-dimensional object; determining a scene depth value corresponding to the target scene according to the image to be processed, and determining a three-dimensional object depth value corresponding to the target three-dimensional object; determining an occlusion relationship between the target three-dimensional object and the target scene based on the scene depth value and the three-dimensional object depth value; constructing an augmented reality scene corresponding to the target scene based on the image to be processed, and rendering the target three-dimensional object in the augmented reality scene based on the occlusion relationship. The target three-dimensional object can be accurately rendered in the augmented reality scene based on the occlusion relationship, and a virtual-real fusion display effect is achieved.

Description

Technical Field

[0001] This disclosure relates to the field of augmented reality technology, and in particular to a scene rendering method, apparatus, device, computer-readable storage medium, and product. Background Technology

[0002] With the development of science and technology, concepts such as AR, VR, MR, and metaverse have been proposed and realized one by one. AR technology is mainly a technology to enhance the sense of reality. It can build virtual scenes into the real world through head-mounted devices and allow people to interact with virtual scenes through the devices.

[0003] Users can add 3D objects to the constructed augmented reality scene according to their actual needs. However, during the movement of 3D objects, they may occlude or be occluded with other objects in the augmented reality scene. How to accurately determine the occlusion relationship of 3D objects has become an urgent problem to be solved. Summary of the Invention

[0004] This disclosure provides a scene rendering method, apparatus, device, computer-readable storage medium, and product, which solves the current technical problem of being unable to accurately determine the occlusion relationship between 3D objects and augmented reality scenes.

[0005] In a first aspect, embodiments of this disclosure provide a scene rendering method, including:

[0006] In response to a user-triggered 3D object display operation, the system acquires the target 3D object to be displayed and the image to be processed corresponding to the target scene associated with the target 3D object.

[0007] The scene depth value corresponding to the target scene is determined based on the image to be processed, and the 3D object depth value corresponding to the target 3D object is determined.

[0008] The occlusion relationship between the target 3D object and the target scene is determined based on the scene depth value and the 3D object depth value;

[0009] An augmented reality scene corresponding to the target scene is constructed based on the image to be processed, and the target 3D object is rendered in the augmented reality scene based on the occlusion relationship.

[0010] Secondly, embodiments of this disclosure provide a scene rendering apparatus, including:

[0011] The acquisition module is used to acquire the target 3D object to be displayed and the image to be processed corresponding to the target scene associated with the target 3D object in response to the user-triggered 3D object display operation.

[0012] The determining module is used to determine the scene depth value corresponding to the target scene based on the image to be processed, and to determine the 3D object depth value corresponding to the target 3D object;

[0013] The processing module is used to determine the occlusion relationship between the target 3D object and the target scene based on the scene depth value and the 3D object depth value;

[0014] The rendering module is used to construct an augmented reality scene corresponding to the target scene based on the image to be processed, and to render the target 3D object in the augmented reality scene based on the occlusion relationship.

[0015] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor and a memory;

[0016] The memory stores computer-executed instructions;

[0017] The processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the scene rendering method as described in the first aspect and various possible designs of the first aspect.

[0018] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the scene rendering method described in the first aspect and various possible designs of the first aspect.

[0019] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the scene rendering method described in the first aspect and various possible designs of the first aspect.

[0020] The scene rendering method, apparatus, device, computer-readable storage medium, and product provided in this embodiment can accurately determine the occlusion relationship between the target 3D object and the target scene by determining the scene depth value corresponding to the target scene and the 3D object depth value corresponding to the target 3D object, and then accurately render the target 3D object in the augmented reality scene based on the occlusion relationship to achieve a virtual-real fusion display effect. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A schematic flowchart of the scene rendering method provided in the embodiments of this disclosure;

[0023] Figure 2 A flowchart illustrating a scene rendering method provided in yet another embodiment of this disclosure;

[0024] Figure 3 A flowchart illustrating a scene rendering method provided in yet another embodiment of this disclosure;

[0025] Figure 4 This is a schematic diagram of the structure of the scene rendering apparatus provided in the embodiments of this disclosure;

[0026] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0028] To address the current technical problem of being unable to accurately determine the occlusion relationship between 3D objects and augmented reality scenes, this disclosure provides a scene rendering method, apparatus, device, computer-readable storage medium, and product.

[0029] It should be noted that the scene rendering methods, apparatus, devices, computer-readable storage media and products provided in this disclosure can be applied to any augmented reality scene.

[0030] Augmented Reality (AR) technology is a technique that cleverly integrates virtual information with the real world. It widely utilizes multimedia, 3D modeling, real-time tracking and registration, intelligent interaction, and sensing technologies to simulate computer-generated text, images, 3D models, music, and videos, applying these virtual elements to the real world. The two types of information complement each other, thus "enhancing" the real world. To enrich the content displayed in AR scenes, users can select 3D objects and place them within the AR scene according to their needs. However, because the occlusion relationship between 3D objects and the AR scene is difficult to determine, directly placing 3D objects into the AR scene may result in poor display quality.

[0031] In addressing the aforementioned technical problems, the inventors discovered that, to accurately place a 3D object within an augmented reality (AR) scene, both the AR scene's depth value and the 3D object's depth value can be determined separately. By comparing these two depth values, the occlusion relationship between the AR scene and the 3D object can be accurately determined. This occlusion relationship then enables the accurate rendering of the 3D object within the AR scene.

[0032] The system architecture disclosed herein includes at least an augmented reality device, a terminal device, and a server. The augmented reality device can acquire image frames corresponding to a target scene. The user can trigger a placement request for a 3D object on the terminal device. After receiving the placement request, the server can acquire both the image frame to be processed and the 3D object, and determine the depth values ​​corresponding to the augmented reality scene and the 3D object, respectively. Based on these depth values, the 3D object is displayed correctly.

[0033] Figure 1 This is a flowchart illustrating the scene rendering method provided in the embodiments of this disclosure, as shown below. Figure 1 As shown, the method includes:

[0034] Step 101: In response to the user-triggered 3D object display operation, obtain the target 3D object to be displayed and the image to be processed corresponding to the target scene associated with the target 3D object.

[0035] In this embodiment, the user can place 3D objects in the augmented reality scene according to actual needs. These 3D objects can be displayed statically or dynamically within the augmented reality scene. For example, the 3D object could be a glowing whale that can rotate around the user in the augmented reality scene.

[0036] Optionally, various 3D objects can be pre-set. Users can select a target 3D object for display on the terminal device according to their actual needs. In response to the user's selection of a 3D object, a 3D object display operation can be generated. In response to the user-triggered 3D object display operation, the target 3D object to be displayed and the corresponding image to be processed for the target scene associated with the target 3D object can be obtained.

[0037] Step 102: Determine the scene depth value corresponding to the target scene based on the image to be processed, and determine the 3D object depth value corresponding to the target 3D object.

[0038] In this embodiment, when a 3D object is displayed in an augmented reality scene, it often causes occlusion of the target scene, or the content in the target scene occludes the 3D object. Continuing the previous example, when the glowing whale swims in front of the user, the glowing whale occludes the user in the augmented reality scene. When the glowing whale swims behind the user, the user occludes the glowing whale.

[0039] Therefore, in order to accurately determine the occlusion relationship between the 3D object and the augmented reality scene, the scene depth value corresponding to the target scene and the 3D object depth value corresponding to the target 3D object can be determined based on the image to be processed.

[0040] Step 103: Determine the occlusion relationship between the target 3D object and the target scene based on the scene depth value and the 3D object depth value.

[0041] In this implementation, a smaller depth value indicates a closer distance between the object and the camera, potentially causing occlusion of objects behind it. Conversely, a larger depth value indicates a greater distance between the object and the camera, potentially causing occlusion by objects in front. Therefore, occlusion relationships can be accurately determined based on scene depth values ​​and 3D object depth values.

[0042] Therefore, after determining the scene depth value corresponding to the target scene and the 3D object depth value corresponding to the target 3D object, the occlusion relationship between the target 3D object and the target scene can be determined based on the scene depth value and the 3D object depth value.

[0043] Step 104: Construct an augmented reality scene corresponding to the target scene based on the image to be processed, and render the target 3D object in the augmented reality scene based on the occlusion relationship.

[0044] In this embodiment, after determining the occlusion relationship between the target 3D object and the target scene, the rendering operation of the target 3D object can be accurately performed based on the occlusion relationship.

[0045] Optionally, an augmented reality scene corresponding to the target scene can be constructed based on the image to be processed, and the target 3D object can be rendered in the augmented reality scene based on occlusion relationships. This enables accurate display of the target 3D object and the target scene in the augmented reality scene, improving the realism of the augmented reality scene.

[0046] The scene rendering method provided in this embodiment determines the scene depth value corresponding to the target scene and the 3D object depth value corresponding to the target 3D object, thereby accurately determining the occlusion relationship between the target 3D object and the target scene based on the scene depth value and the 3D object depth value. Based on this occlusion relationship, the target 3D object can be accurately rendered in the augmented reality scene to achieve a virtual-real fusion display effect.

[0047] Furthermore, based on any of the above embodiments, step 102 includes:

[0048] The first depth map corresponding to the target scene is determined based on the image to be processed and a preset depth estimation algorithm.

[0049] A linear transformation operation is performed on the first depth map based on a preset linear transformation algorithm to obtain the scene depth value corresponding to the target scene.

[0050] In this embodiment, a depth estimation algorithm can be preset so that after the image to be processed is acquired, the image to be processed can be processed by the depth estimation algorithm to obtain the first depth map corresponding to the target scene.

[0051] Furthermore, in order to more intuitively determine the occlusion relationship between the target 3D object and the target scene, the content of the first depth map can be digitized to obtain the scene depth value corresponding to the target scene.

[0052] Optionally, the depth values ​​can be evenly distributed across the near and far planes of the camera using a preset linear transformation algorithm. This linear transformation operation can then be performed on the first depth map to obtain the scene depth values ​​corresponding to the target scene. The linear transformation algorithm can be as shown in Equation 1:

[0053]

[0054] Where zNear and zFar are the z values ​​of the camera's near and far planes, respectively.

[0055] Furthermore, based on any of the above embodiments, step 102 includes:

[0056] The second depth map corresponding to the target 3D object is determined based on the target 3D object and a preset depth estimation algorithm.

[0057] A linear transformation operation is performed on the second depth map based on a preset linear transformation algorithm to obtain the scene depth value corresponding to the target 3D object.

[0058] In this embodiment, a depth estimation algorithm can be preset so that after the image to be processed is acquired, the target 3D object can be processed by the depth estimation algorithm to obtain the second depth map corresponding to the target 3D object.

[0059] Furthermore, in order to more intuitively determine the occlusion relationship between the target 3D object and the target scene, the content of the second depth map can be digitized to obtain the scene depth value corresponding to the target 3D object.

[0060] Optionally, a preset linear transformation algorithm can be used to evenly distribute the depth values ​​within the near and far planes of the camera, and then perform a linear transformation operation on the second depth map to obtain the scene depth value corresponding to the target 3D object.

[0061] Optionally, based on any of the above embodiments, step 103 includes:

[0062] If the scene depth value is greater than the 3D object depth value, then the target 3D object is determined to be located in front of the target scene, thus occluding the target scene.

[0063] If the scene depth value is less than the 3D object depth value, it is determined that the target scene is located in front of the target 3D object, thus occluding the target 3D object.

[0064] In this embodiment, after obtaining the scene depth value and the 3D object depth value respectively, a comparison operation can be performed between the scene depth value and the 3D object depth value to obtain the comparison result. Then, the occlusion relationship between the target 3D object and the target scene can be accurately determined based on the comparison result.

[0065] Understandably, a smaller depth value indicates a closer distance between the object and the camera, while a larger depth value indicates a farther distance. Therefore, after obtaining the comparison results, if the scene depth value is greater than the 3D object depth value, the target 3D object is determined to be in front of the target scene, occluding the target scene. If the scene depth value is less than the 3D object depth value, the target scene is determined to be in front of the target 3D object, occluding the target 3D object.

[0066] After determining the occlusion relationship, you can render content that is closer to the camera, while not rendering content that is occluded.

[0067] The scene rendering method provided in this embodiment determines the scene depth value corresponding to the target scene and the 3D object depth value corresponding to the target 3D object. This allows for the accurate determination of the occlusion relationship between the target 3D object and the target scene based on the scene depth value and the 3D object depth value. Consequently, it enables the accurate display of the correct display effect between the target 3D object and the target scene in augmented reality scenes, thereby improving the realism of augmented reality scenes.

[0068] Figure 2 This is a flowchart illustrating a scene rendering method provided in yet another embodiment of the present disclosure. Based on any of the above embodiments, such as... Figure 2 As shown, after step 103, the following steps are also included:

[0069] Step 201: Calculate the difference between the scene depth value and the 3D object depth value.

[0070] Step 202: Determine the occlusion edge region between the target scene and the target 3D object based on the difference information.

[0071] Step 203: Perform a feathering operation on the occluded edge area to obtain the feathering result.

[0072] In this embodiment, when displaying a target 3D object in an augmented reality scene, the target 3D object and the target scene may overlap as the target 3D object or objects in the augmented reality scene move. To make the overlapping parts appear more realistic, occlusion edge regions can be identified and feathered.

[0073] Furthermore, the difference between the scene depth value and the 3D object depth value can be calculated. Understandably, the larger the difference, the farther the target 3D object is from the target scene, and therefore, overlap is generally unlikely. Conversely, the smaller the difference, the closer the target 3D object is to the target scene, and the greater the likelihood of overlap. Therefore, after determining the difference, the occlusion edge regions of the target scene and the target 3D object can be identified based on this difference. A feathering operation is then performed on these occlusion edge regions to obtain the feathered results.

[0074] Furthermore, based on any of the above embodiments, step 202 includes:

[0075] Based on a preset difference range, a target difference value matching the difference range is determined from the difference information.

[0076] The set of pixels corresponding to the target difference is determined as the occlusion edge region.

[0077] In this embodiment, to accurately identify occlusion edge regions with overlapping phenomena, a difference range can be preset. Pixels whose difference information falls within this difference range are the occlusion edge regions.

[0078] Therefore, after determining the difference information, a target difference that matches the preset difference range can be identified from the difference information. The set of pixels corresponding to the target difference is then defined as the occlusion edge region.

[0079] It should be noted that this difference range can be mapped to the alpha value of a pixel. Therefore, after determining the occlusion edge region, the color fusion operation between the target scene and the target 3D object can be performed based on the alpha value of the pixel corresponding to the occlusion edge region.

[0080] The scene rendering method provided in this embodiment identifies overlapping occlusion edge regions and feathers these occlusion edge regions, thereby making the augmented reality scene more closely resemble the real scene, improving the realism of the augmented reality scene, and enhancing the user experience.

[0081] Figure 3 This is a flowchart illustrating a scene rendering method provided in yet another embodiment of the present disclosure. Based on any of the above embodiments, such as... Figure 3 As shown, after step 101, the following steps are also included:

[0082] Step 301: Generate a normal map corresponding to the image to be processed, and generate a projection region corresponding to the target 3D object.

[0083] Step 302: Based on the normal map, perform an offset operation on the projection area to obtain a projection area with a distortion effect.

[0084] Step 303: Display the projection area of ​​the distortion effect in the display area associated with the target 3D object in the augmented reality scene generated based on the image to be processed.

[0085] In this embodiment, the current shadow processing method generally renders the shadow of a 3D object onto a transparent surface and simultaneously places the surface in 3D space. However, since the surface itself has a size, using the above method for shadow processing can easily result in the shadow being clipped outside the surface, leading to poor shadow display.

[0086] In augmented reality scenarios, projections can include shadows and the aperture of luminous objects. After acquiring the target 3D object, a projection region corresponding to the target 3D object can be generated. This projection region is then offset based on the normal map of the image to be processed, preventing the projection region from coinciding with the target 3D object and improving the realism of the augmented reality scene.

[0087] Optionally, a normal map corresponding to the image to be processed can be generated, as well as a projection region corresponding to the target 3D object. An offset operation is performed on the projection region based on the normal map to obtain a distorted projection region. The distorted projection region is then displayed in the display area associated with the target 3D object in the augmented reality scene generated based on the image to be processed.

[0088] The scene rendering method provided in this embodiment, after generating the projection area corresponding to the target 3D object, performs an offset operation on the projection area based on the normal map corresponding to the image to be processed, thereby obtaining a more realistic projection of the target 3D object, avoiding the projection area from coinciding with the target 3D object, and improving the realism in augmented reality scenes.

[0089] Furthermore, based on any of the above embodiments, step 301 includes:

[0090] The target 3D object is rendered onto a preset rendering target to obtain a first projection rendering target.

[0091] Obtain the preset channel in the first projection rendering target, and convert the first projection rendering target into a solid color second projection rendering target.

[0092] The second projection rendering target is blurred to obtain the projection area.

[0093] In this embodiment, to render the projection area, it is first necessary to construct the projection area. First, the target 3D object can be rendered onto a preset rendering target to obtain a first projection rendering target. A preset channel in the first projection rendering target is obtained, and the first projection rendering target is converted into a solid-color second projection rendering target. This preset channel can be an alpha channel. The second projection rendering target is then blurred to obtain the projection area. Any blurring algorithm can be used to blur the second projection rendering target; for example, a Gaussian blur algorithm can be used. This disclosure does not limit this approach.

[0094] Furthermore, based on any of the above embodiments, step 302 includes:

[0095] Read the color value of the color channel corresponding to the normal map, and determine the color value as the normal texture sampling value corresponding to the normal map.

[0096] The value range corresponding to the normal texture sampling value is converted to a preset target range to obtain the target normal sampling value.

[0097] The values ​​of two preset directions in the target normal sampling value are used as the offset base directions. Based on the offset base directions and the preset offset algorithm, the projection area is offset to obtain a projection area with a distortion effect.

[0098] In this embodiment, to perform the offset operation on the shadow region, the color value of the color channel corresponding to the normal map can be read, and the color value can be determined as the normal texture sampling value corresponding to the normal map. For example, the RGB color value of the normal map can be read as the xyz (packed normal) of the normal. The value range corresponding to the normal texture sampling value is transformed to a preset target range to obtain the target normal sampling value. Specifically, the xyz of [0,1] can be restored to [-1,1] to obtain the target normal sampling value. The values ​​of two preset directions in the target normal sampling value are used as the offset base directions. Based on the offset base directions and the preset offset algorithm, the projection area is offset to obtain the projection area with the distortion effect.

[0099] Alternatively, since uv is a two-dimensional vector, the xy of the target normal sample value can be used as the offset base direction, and the distorted uv can be used to sample the target projection RT.

[0100] The offset operation of the projection area can be achieved using formula 2-3:

[0101] uvOffset(x,y)=(sceneNormal.xy+unitOffset)*-0.12-pos(x,y) (2)

[0102] uvDist(x,y)=textureCoordinate+unitOffset*distort*scale (3)

[0103] Where sceneNormal is the scene normal (x,y,z), textureCoordinate is the texture coordinate, distort is the degree of distortion of the projected area, scale is the UV scaling, unitOffset is the unit distortion offset value, and -0.12 and pos(x,y) are preset constants.

[0104] Furthermore, based on any of the above embodiments, step 104 includes:

[0105] The scene depth map is offset based on the normal map to obtain the offset result.

[0106] The color to be rendered for the target 3D object is determined based on the offset result, the projection area of ​​the distortion effect, the color of the projection area, and the color of the target 3D object.

[0107] The target 3D object is rendered in the augmented reality scene based on the occlusion relationship and the color to be rendered.

[0108] In this embodiment, to further enhance the realism of the augmented reality scene, an offset processing step can be performed on the depth map during projection. This involves offsetting the scene depth map based on the normal map to obtain the offset result. The color to be rendered for the target 3D object is determined based on the offset result, the projection area of ​​the distortion effect, the color of the projection area, and the color of the target 3D object. The target 3D object is then rendered in the augmented reality scene based on the occlusion relationship and the color to be rendered.

[0109] The scene rendering method provided in this embodiment, after generating the projection area corresponding to the target 3D object, performs an offset operation on the projection area based on the normal map corresponding to the image to be processed, thereby obtaining a more realistic projection of the target 3D object, avoiding the projection area from coinciding with the target 3D object, and improving the realism in augmented reality scenes.

[0110] Figure 4 This is a schematic diagram of the structure of the scene rendering apparatus provided in the embodiments of this disclosure, such as... Figure 4 As shown, the device includes: an acquisition module 41, a determination module 42, a processing module 43, and a rendering module 44. The acquisition module 41 is used to acquire, in response to a user-triggered 3D object display operation, a target 3D object to be displayed and a corresponding image to be processed for a target scene associated with the target 3D object. The determination module 42 is used to determine the scene depth value corresponding to the target scene based on the image to be processed, and to determine the 3D object depth value corresponding to the target 3D object. The processing module 43 is used to determine the occlusion relationship between the target 3D object and the target scene based on the scene depth value and the 3D object depth value. The rendering module 44 is used to construct an augmented reality scene corresponding to the target scene based on the image to be processed, and to render the target 3D object in the augmented reality scene based on the occlusion relationship.

[0111] Furthermore, based on any of the above embodiments, the determining module is configured to: determine a first depth map corresponding to the target scene based on the image to be processed and a preset depth estimation algorithm; and perform a linear transformation operation on the first depth map based on a preset linear transformation algorithm to obtain a scene depth value corresponding to the target scene.

[0112] Furthermore, based on any of the above embodiments, the determining module is configured to: determine a second depth map corresponding to the target 3D object based on the target 3D object and a preset depth estimation algorithm; and perform a linear transformation operation on the second depth map based on a preset linear transformation algorithm to obtain the scene depth value corresponding to the target 3D object.

[0113] Furthermore, based on any of the above embodiments, the processing module is configured to: if the scene depth value is greater than the 3D object depth value, determine that the target 3D object is located in front of the target scene and occlude the target scene; if the scene depth value is less than the 3D object depth value, determine that the target scene is located in front of the target 3D object and occlude the target 3D object.

[0114] Furthermore, based on any of the above embodiments, the apparatus further includes: a calculation module, used to calculate the difference information between the scene depth value and the 3D object depth value; a determination module, used to determine the occlusion edge region between the target scene and the target 3D object based on the difference information; and a processing module, used to perform a feathering operation on the occlusion edge region to obtain a feathering result.

[0115] Furthermore, based on any of the above embodiments, the determining module is configured to: determine a target difference value matching the preset difference range in the difference information; and determine the set of pixels corresponding to the target difference value as the occlusion edge region.

[0116] Furthermore, based on any of the above embodiments, the apparatus further includes: a generation module, configured to generate a normal map corresponding to the image to be processed, and to generate a projection region corresponding to the target 3D object; a processing module, configured to perform an offset operation on the projection region based on the normal map to obtain a projection region with a distortion effect; and a display module, configured to display the projection region with the distortion effect in a display area associated with the target 3D object in an augmented reality scene generated based on the image to be processed.

[0117] Further, based on any of the above embodiments, the generation module is configured to: render the target 3D object onto a preset rendering target to obtain a first projection rendering target; obtain a preset channel in the first projection rendering target and convert the first projection rendering target into a solid-color second projection rendering target; and perform blurring processing on the second projection rendering target to obtain the projection area.

[0118] Further, based on any of the above embodiments, the processing module is configured to: read the color value of the color channel corresponding to the normal map, and determine the color value as the normal texture sampling value corresponding to the normal map; convert the value range corresponding to the normal texture sampling value to a preset target range to obtain a target normal sampling value; use the values ​​of two preset directions in the target normal sampling value as the offset base directions, and perform an offset operation on the projection area based on the offset base directions and a preset offset algorithm to obtain a projection area with a distortion effect.

[0119] Further, based on any of the above embodiments, the rendering module is configured to: perform an offset operation on the scene depth map based on the normal map to obtain an offset result; determine the color to be rendered corresponding to the target 3D object based on the offset result, the projection area of ​​the distortion effect, the color of the projection area, and the color of the target 3D object; and render the target 3D object in the augmented reality scene based on the occlusion relationship and the color to be rendered.

[0120] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0121] To implement the above embodiments, this disclosure also provides an electronic device, including: a processor and a memory;

[0122] The memory stores computer-executed instructions;

[0123] The processor executes computer execution instructions stored in the memory, causing the processor to perform the scene rendering method as described in any of the above embodiments.

[0124] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this disclosure, such as... Figure 5 As shown, the electronic device 500 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0125] like Figure 5 As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0126] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0127] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0128] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0129] To implement the above embodiments, this disclosure also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the scene rendering method as described in any of the above embodiments.

[0130] To implement the above embodiments, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the scene rendering method as described in any of the above embodiments.

[0131] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0132] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0133] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0135] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0136] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0137] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0138] In a first aspect, according to one or more embodiments of this disclosure, a scene rendering method is provided, comprising:

[0139] In response to a user-triggered 3D object display operation, the system acquires the target 3D object to be displayed and the image to be processed corresponding to the target scene associated with the target 3D object.

[0140] The scene depth value corresponding to the target scene is determined based on the image to be processed, and the 3D object depth value corresponding to the target 3D object is determined.

[0141] The occlusion relationship between the target 3D object and the target scene is determined based on the scene depth value and the 3D object depth value;

[0142] An augmented reality scene corresponding to the target scene is constructed based on the image to be processed, and the target 3D object is rendered in the augmented reality scene based on the occlusion relationship.

[0143] According to one or more embodiments of this disclosure, determining the scene depth value corresponding to the target scene based on the image to be processed includes:

[0144] The first depth map corresponding to the target scene is determined based on the image to be processed and a preset depth estimation algorithm;

[0145] A linear transformation operation is performed on the first depth map based on a preset linear transformation algorithm to obtain the scene depth value corresponding to the target scene.

[0146] According to one or more embodiments of this disclosure, determining the depth value of the three-dimensional object corresponding to the target three-dimensional object includes:

[0147] A second depth map corresponding to the target 3D object is determined based on the target 3D object and a preset depth estimation algorithm;

[0148] A linear transformation operation is performed on the second depth map based on a preset linear transformation algorithm to obtain the scene depth value corresponding to the target 3D object.

[0149] According to one or more embodiments of this disclosure, determining the occlusion relationship between the target 3D object and the target scene based on the scene depth value and the 3D object depth value includes:

[0150] If the scene depth value is greater than the 3D object depth value, then the target 3D object is determined to be located in front of the target scene, thus occluding the target scene;

[0151] If the scene depth value is less than the 3D object depth value, it is determined that the target scene is located in front of the target 3D object, thus occluding the target 3D object.

[0152] According to one or more embodiments of this disclosure, after determining the occlusion relationship between the target 3D object and the target scene based on the scene depth value and the 3D object depth value, the method further includes:

[0153] Calculate the difference between the scene depth value and the 3D object depth value;

[0154] Based on the difference information, the occlusion edge region between the target scene and the target 3D object is determined;

[0155] Perform a feathering operation on the occluded edge region to obtain the feathering result.

[0156] According to one or more embodiments of this disclosure, determining the occlusion edge region between the target scene and the target 3D object based on the difference information includes:

[0157] A target difference that matches the preset difference range is determined from the difference information.

[0158] The set of pixels corresponding to the target difference is determined as the occlusion edge region.

[0159] According to one or more embodiments of this disclosure, after obtaining the target 3D object to be displayed and the image to be processed corresponding to the target scene associated with the target 3D object, the method further includes:

[0160] Generate a normal map corresponding to the image to be processed, and generate a projection region corresponding to the target 3D object;

[0161] The projection region is offset based on the normal map to obtain a projection region with a distorted effect;

[0162] In the augmented reality scene generated based on the image to be processed, the projection area of ​​the distortion effect is displayed in the display area associated with the target 3D object.

[0163] According to one or more embodiments of this disclosure, generating a projection region corresponding to the target 3D object includes:

[0164] The target 3D object is rendered onto a preset rendering target to obtain a first projection rendering target;

[0165] Obtain a preset channel from the first projection rendering target, and convert the first projection rendering target into a solid color second projection rendering target;

[0166] The second projection rendering target is blurred to obtain the projection area.

[0167] According to one or more embodiments of this disclosure, the step of offsetting the projection region based on the normal map to obtain a distorted projection region includes:

[0168] Read the color value of the color channel corresponding to the normal map, and determine the color value as the normal texture sampling value corresponding to the normal map;

[0169] The value range corresponding to the normal texture sampling value is transformed into a preset target range to obtain the target normal sampling value;

[0170] The values ​​of two preset directions in the target normal sampling value are used as the offset base directions. Based on the offset base directions and the preset offset algorithm, the projection area is offset to obtain a projection area with a distortion effect.

[0171] According to one or more embodiments of this disclosure, rendering the target 3D object in the augmented reality scene based on the occlusion relationship includes:

[0172] Based on the normal map, the scene depth map is offset to obtain the offset result;

[0173] The color to be rendered for the target 3D object is determined based on the offset result, the projection area of ​​the distortion effect, the color of the projection area, and the color of the target 3D object.

[0174] The target 3D object is rendered in the augmented reality scene based on the occlusion relationship and the color to be rendered.

[0175] Secondly, according to one or more embodiments of this disclosure, a scene rendering apparatus is provided, comprising:

[0176] The acquisition module is used to acquire the target 3D object to be displayed and the image to be processed corresponding to the target scene associated with the target 3D object in response to the user-triggered 3D object display operation.

[0177] The determining module is used to determine the scene depth value corresponding to the target scene based on the image to be processed, and to determine the 3D object depth value corresponding to the target 3D object;

[0178] The processing module is used to determine the occlusion relationship between the target 3D object and the target scene based on the scene depth value and the 3D object depth value;

[0179] The rendering module is used to construct an augmented reality scene corresponding to the target scene based on the image to be processed, and to render the target 3D object in the augmented reality scene based on the occlusion relationship.

[0180] According to one or more embodiments of this disclosure, the determining module is configured to:

[0181] The first depth map corresponding to the target scene is determined based on the image to be processed and a preset depth estimation algorithm;

[0182] A linear transformation operation is performed on the first depth map based on a preset linear transformation algorithm to obtain the scene depth value corresponding to the target scene.

[0183] According to one or more embodiments of this disclosure, the determining module is configured to:

[0184] A second depth map corresponding to the target 3D object is determined based on the target 3D object and a preset depth estimation algorithm;

[0185] A linear transformation operation is performed on the second depth map based on a preset linear transformation algorithm to obtain the scene depth value corresponding to the target 3D object.

[0186] According to one or more embodiments of this disclosure, the processing module is configured to:

[0187] If the scene depth value is greater than the 3D object depth value, then the target 3D object is determined to be located in front of the target scene, thus occluding the target scene;

[0188] If the scene depth value is less than the 3D object depth value, it is determined that the target scene is located in front of the target 3D object, thus occluding the target 3D object.

[0189] According to one or more embodiments of this disclosure, the apparatus further includes:

[0190] The calculation module is used to calculate the difference information between the scene depth value and the 3D object depth value;

[0191] The determination module is used to determine the occlusion edge region between the target scene and the target 3D object based on the difference information;

[0192] The processing module is used to perform a feathering operation on the occluded edge region to obtain the feathering result.

[0193] According to one or more embodiments of this disclosure, the determining module is configured to:

[0194] A target difference that matches the preset difference range is determined from the difference information.

[0195] The set of pixels corresponding to the target difference is determined as the occlusion edge region.

[0196] According to one or more embodiments of this disclosure, the apparatus further includes:

[0197] The generation module is used to generate a normal map corresponding to the image to be processed, and to generate a projection region corresponding to the target 3D object;

[0198] The processing module is used to perform an offset operation on the projection area based on the normal map to obtain a projection area with a distortion effect;

[0199] The display module is used to display the projection area of ​​the distortion effect in the display area associated with the target 3D object in the augmented reality scene generated based on the image to be processed.

[0200] According to one or more embodiments of this disclosure, the generation module is configured to:

[0201] The target 3D object is rendered onto a preset rendering target to obtain a first projection rendering target;

[0202] Obtain a preset channel from the first projection rendering target, and convert the first projection rendering target into a solid color second projection rendering target;

[0203] The second projection rendering target is blurred to obtain the projection area.

[0204] According to one or more embodiments of this disclosure, the processing module is configured to:

[0205] Read the color value of the color channel corresponding to the normal map, and determine the color value as the normal texture sampling value corresponding to the normal map;

[0206] The value range corresponding to the normal texture sampling value is transformed into a preset target range to obtain the target normal sampling value;

[0207] The values ​​of two preset directions in the target normal sampling value are used as the offset base directions. Based on the offset base directions and the preset offset algorithm, the projection area is offset to obtain a projection area with a distortion effect.

[0208] According to one or more embodiments of this disclosure, the rendering module is configured to:

[0209] Based on the normal map, the scene depth map is offset to obtain the offset result;

[0210] The color to be rendered for the target 3D object is determined based on the offset result, the projection area of ​​the distortion effect, the color of the projection area, and the color of the target 3D object.

[0211] The target 3D object is rendered in the augmented reality scene based on the occlusion relationship and the color to be rendered.

[0212] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0213] The memory stores computer-executed instructions;

[0214] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the scene rendering method as described in the first aspect and various possible designs of the first aspect.

[0215] Fourthly, according to one or more embodiments of this disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the scene rendering method described in the first aspect and various possible designs of the first aspect is implemented.

[0216] Fifthly, according to one or more embodiments of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the scene rendering method described in the first aspect and various possible designs of the first aspect.

[0217] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0218] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0219] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A scene rendering method, characterized in that, include: In response to a user-triggered 3D object display operation, the system acquires the target 3D object to be displayed and the image to be processed corresponding to the target scene associated with the target 3D object; wherein the target 3D object is a computer-generated virtual 3D object, and the target scene is a real scene; The scene depth value corresponding to the target scene is determined based on the image to be processed, and the 3D object depth value corresponding to the target 3D object is determined; the 3D object depth value represents the distance between each point in the virtual 3D object and the virtual camera; The occlusion relationship between the target 3D object and the target scene is determined based on the scene depth value and the 3D object depth value; An augmented reality scene corresponding to the target scene is constructed based on the image to be processed, and the target 3D object is rendered in the augmented reality scene based on the occlusion relationship; After acquiring the target 3D object to be displayed and the image to be processed corresponding to the target scene associated with the target 3D object, the process further includes: Generate a normal map corresponding to the image to be processed, and generate a projection region corresponding to the target 3D object; wherein, the normal map is used to characterize the surface orientation information of real objects in the target scene; Based on the normal map, the projection region is offset to obtain a distorted projection region, and the distorted effect matches the surface geometry of the real object in the image to be processed. In the augmented reality scene generated based on the image to be processed, the projection area of ​​the distortion effect is displayed in the display area associated with the target 3D object.

2. The method according to claim 1, characterized in that, Determining the scene depth value corresponding to the target scene based on the image to be processed includes: The first depth map corresponding to the target scene is determined based on the image to be processed and a preset depth estimation algorithm; A linear transformation operation is performed on the first depth map based on a preset linear transformation algorithm to obtain the scene depth value corresponding to the target scene.

3. The method according to claim 1, characterized in that, Determining the depth value of the three-dimensional object corresponding to the target three-dimensional object includes: A second depth map corresponding to the target 3D object is determined based on the target 3D object and a preset depth estimation algorithm; A linear transformation operation is performed on the second depth map based on a preset linear transformation algorithm to obtain the scene depth value corresponding to the target 3D object.

4. The method according to claim 1, characterized in that, Determining the occlusion relationship between the target 3D object and the target scene based on the scene depth value and the 3D object depth value includes: If the scene depth value is greater than the 3D object depth value, then the target 3D object is determined to be located in front of the target scene, thus occluding the target scene; If the scene depth value is less than the 3D object depth value, it is determined that the target scene is located in front of the target 3D object, thus occluding the target 3D object.

5. The method according to claim 1, characterized in that, After determining the occlusion relationship between the target 3D object and the target scene based on the scene depth value and the 3D object depth value, the method further includes: Calculate the difference between the scene depth value and the 3D object depth value; Based on the difference information, the occlusion edge region between the target scene and the target 3D object is determined; Perform a feathering operation on the occluded edge region to obtain the feathering result.

6. The method according to claim 5, characterized in that, Determining the occlusion edge region between the target scene and the target 3D object based on the difference information includes: A target difference that matches the preset difference range is determined from the difference information. The set of pixels corresponding to the target difference is determined as the occlusion edge region.

7. The method according to claim 1, characterized in that, The generation of the projection region corresponding to the target 3D object includes: The target 3D object is rendered onto a preset rendering target to obtain a first projection rendering target; Obtain a preset channel from the first projection rendering target, and convert the first projection rendering target into a solid color second projection rendering target; The second projection rendering target is blurred to obtain the projection area.

8. The method according to claim 1, characterized in that, The step of offsetting the projection region based on the normal map to obtain a distorted projection region includes: Read the color value of the color channel corresponding to the normal map, and determine the color value as the normal texture sampling value corresponding to the normal map; The value range corresponding to the normal texture sampling value is transformed into a preset target range to obtain the target normal sampling value; The values ​​of two preset directions in the target normal sampling value are used as the offset base directions. Based on the offset base directions and the preset offset algorithm, the projection area is offset to obtain a projection area with a distortion effect.

9. The method according to claim 1, characterized in that, Rendering the target 3D object in the augmented reality scene based on the occlusion relationship includes: Based on the normal map, the scene depth map is offset to obtain the offset result; The color to be rendered for the target 3D object is determined based on the offset result, the projection area of ​​the distortion effect, the color of the projection area, and the color of the target 3D object. The target 3D object is rendered in the augmented reality scene based on the occlusion relationship and the color to be rendered.

10. A scene rendering device, characterized in that, include: The acquisition module is used to acquire, in response to a user-triggered 3D object display operation, a target 3D object to be displayed and an image to be processed corresponding to a target scene associated with the target 3D object; wherein, the target 3D object is a computer-generated virtual 3D object and the target scene is a real scene; The determining module is used to determine the scene depth value corresponding to the target scene based on the image to be processed, and to determine the 3D object depth value corresponding to the target 3D object; the 3D object depth value represents the distance between each point in the virtual 3D object and the virtual camera; The processing module is used to determine the occlusion relationship between the target 3D object and the target scene based on the scene depth value and the 3D object depth value; The rendering module is used to construct an augmented reality scene corresponding to the target scene based on the image to be processed, and to render the target 3D object in the augmented reality scene based on the occlusion relationship; A generation module is used to generate a normal map corresponding to the image to be processed, and to generate a projection region corresponding to the target 3D object; wherein, the normal map is used to characterize the surface orientation information of real objects in the target scene; The processing module is used to perform an offset operation on the projection area based on the normal map to obtain a projection area with a distortion effect, wherein the distortion effect matches the surface geometry of the real object in the image to be processed. The display module is used to display the projection area of ​​the distortion effect in the display area associated with the target 3D object in the augmented reality scene generated based on the image to be processed.

11. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the scene rendering method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the scene rendering method as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the scene rendering method as described in any one of claims 1 to 9.

Images