A method and terminal device for image rendering in panoramic application
By setting depth and depth reference values in panoramic applications, the foreground image is rendered first, followed by the rendering of unoccupied pixels. This solves the problem of resource waste caused by repeated rendering in panoramic applications and achieves a more efficient rendering process.
Patent Information
- Application Number
- CN201911403810.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-12-27
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2039-12-27
AI Technical Summary
Existing technologies in panoramic applications require repeated rendering of overlapping areas, resulting in high rendering overhead and significant resource waste.
By setting the depth value and depth reference value of the foreground image, the foreground image is rendered first, and then the panoramic image is rendered for the pixels not occupied by the foreground image, thus avoiding repeated rendering.
It reduces resource waste, lowers rendering overhead, improves rendering efficiency, and ensures rendering accuracy.
Smart Images

Figure CN113129417B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a method and terminal device for image rendering in a panoramic application. Background Art
[0002] Panoramic images have the advantages of strong sense of reality, large amount of information and good interactivity, which can give users an immersive feeling. Therefore, panoramic images are widely used in many fields, such as panoramic car viewing, panoramic house viewing, panoramic hotel experience, map street view and artificial intelligence (AI) video background replacement for video chat.
[0003] In these panoramic applications, panoramic images are not simply displayed, but involve superimposing one or more foreground elements on the panoramic image. The foreground elements can be human images, ground signs or other images, etc. In panoramic applications, the panoramic image is usually the background. To achieve superimposing the foreground image on the panoramic image, the layer principle is currently used.
[0004] When rendering a frame of background and foreground images using the layer principle, the background image is rendered first, and the color information of each pixel in the background image is filled into the display buffer. Then, the color information of the corresponding pixel of the foreground image is filled into the display buffer. The color information of the pixel of the foreground image will cover the color information of the pixel of the background image that overlaps with the foreground image. It can be seen that the overlapping part needs to be rendered repeatedly, resulting in a very high rendering overhead. Summary of the invention
[0005] The embodiment of the present application provides a method for image rendering in a panoramic application, which is used to reduce rendering overhead and avoid waste of resources caused by repeated rendering. The embodiment of the present application also provides a corresponding terminal device.
[0006] The first aspect of the present application provides a method for image rendering in a panoramic application, and the panoramic application may include panoramic car viewing, panoramic house viewing, panoramic hotel experience, map street view and artificial intelligence (AI) video background replacement for video chat, as well as other applications applied to panoramic pictures. The method includes: in response to the operation of the panoramic application, obtaining a foreground picture and a panoramic picture, the panoramic picture is the background picture of the foreground picture; rendering the foreground picture, and setting the depth value of each pixel on the canvas, the depth value of each pixel includes a first depth value and a depth reference value, the first depth value is the depth value of the pixel corresponding to the foreground picture, the depth reference value is the depth value of the pixel except the pixel corresponding to the foreground picture, the canvas is the area on the screen of the terminal device for displaying the panoramic picture and the foreground picture, it should be understood that the canvas can refer to a part of the display area on the screen, or it can refer to the display area of the entire screen; according to the first depth value and the depth reference value, the color value corresponding to the first pixel on the panoramic picture is rendered at the first pixel, and the first pixel is the pixel corresponding to the depth reference value.
[0007] In the first aspect, the foreground image may be a person, a ground icon or other image different from the background image. The background image may also be a non-panoramic image, and the size of the background image is larger than the size of the foreground image. For a canvas, the number of pixels contained in the canvas is fixed, and the pixels on the canvas can be regarded as a pixel set A, wherein the set of pixels occupied by the foreground image can be described as a subset B, and the set of pixels remaining after removing subset B can be described as a subset C, and the pixels of subset B and subset C do not overlap, and subset B + subset C = set A. For ease of description, the "first depth value" refers to the depth value of the pixel corresponding to the foreground image, and the depth reference value is a reference value. The first depth value and the depth reference value are respectively used to represent different states of the pixel, and the first depth value is used to indicate that the corresponding pixel does not need to be repeatedly rendered, and the depth reference value indicates that the corresponding pixel has not been rendered. The first depth value can be represented by a specific numerical value, such as -0.5, and the depth reference value can be represented by 0. Of course, it can also be other numerical values. In addition, the first depth value and the depth reference value in the present application are not limited to numerical values, and other characters that can indicate whether the pixel needs to be rendered can also be used, for example: T means that repeated rendering is not required, and F means that the pixel has not been rendered. Of course, it can also be represented by other characters as long as the indication is clear. The depth value does not represent a specific physical concept, and it can be expressed by other names, as long as it can express whether the pixel can be rendered according to the panoramic image. If the depth reference value is 0, it can also be understood that before each rendering of the foreground image, a clearing operation will be performed on the depth value of the pixel on the canvas, and then the first depth value of the foreground image is written to the pixel of the corresponding foreground image when the foreground image is rendered. If the depth reference value is not 0, it can also be understood that before each rendering of the foreground image, a unified assignment operation of the depth value of the pixel on the canvas is performed, and the depth values of the pixels on the canvas are first assigned to the depth reference value, and then when the foreground image is rendered, the depth reference value is modified to the first depth value at the pixel of the corresponding foreground image. From the first aspect, it can be seen that the foreground image is rendered first, and then the corresponding content of the panoramic image is rendered for the pixels not occupied by the foreground image. In this way, repeated rendering of the overlapping parts of the foreground image and the panoramic image is avoided, which not only reduces resource waste and rendering overhead, but also improves rendering efficiency.
[0008] In a possible implementation manner of the first aspect, the above-mentioned steps: rendering the color value corresponding to the first pixel point on the panoramic image at the first pixel point according to the first depth value and the depth reference value, include: if the depth value of the second pixel point is the first depth value, interrupting the rendering of the color value corresponding to the second pixel point on the panoramic image at the second pixel point, the second pixel point being any pixel point on the canvas; if the depth value of the second pixel point is the depth reference value, rendering the color value corresponding to the second pixel point on the panoramic image at the second pixel point.
[0009] In this possible implementation, after the foreground image is rendered, the panoramic image is rendered. The panoramic image can be rendered pixel by pixel. The rendered pixel can be referred to as the "second pixel". If the depth value of the second pixel is the first depth value, it means that the second pixel has been occupied by the foreground image and does not need to be rendered repeatedly, so the rendering of the second pixel is interrupted. If the depth value of the second pixel is the depth reference value, it means that the second pixel is not occupied by the foreground image, so the rendering is performed according to the content of the panoramic image corresponding to the second pixel. In this possible implementation, the indication of whether the pixel needs to be rendered by the first depth value and the depth reference value can ensure that the pixel occupied by the foreground image will not be rendered repeatedly according to the panoramic image, thereby ensuring the accuracy of the rendering.
[0010] In a possible implementation manner of the first aspect, the above-mentioned step: rendering the color value corresponding to the first pixel on the panoramic image at the first pixel according to the first depth value and the depth reference value, includes: if the depth value of the second pixel is less than the second depth value, interrupting the rendering of the color value corresponding to the second pixel on the panoramic image at the second pixel, the second depth value is greater than the first depth value and less than the depth reference value, and the second pixel is any pixel on the canvas; if the depth value of the second pixel is greater than the second depth value, rendering the color value corresponding to the second pixel on the panoramic image at the second pixel.
[0011] In this possible implementation, the "second depth value" refers to the depth value of the panoramic image. The first depth value and the second depth value are relative. Because the panoramic image is the background image of the foreground image, the second depth value must be greater than the first depth value. The first depth value and the second depth value are both less than the depth reference value. The first depth value, the second depth value and the depth reference value are usually pre-configured. The rendering of the panoramic image can be performed pixel by pixel. The rendered pixel can be referred to as the "second pixel". If the depth value of the second pixel is less than the second depth value, it means that the second pixel has been occupied by the foreground image and does not need to be repeatedly rendered, so the rendering of the second pixel is interrupted. If the depth value of the second pixel is greater than the second depth value, it means that the second pixel is not occupied by the foreground image, and the content of the panoramic image corresponding to the second pixel is rendered. In this possible implementation, by comparing the first depth value, the depth reference value and the second depth value to determine whether the second pixel needs to be rendered, it can be ensured that the pixel occupied by the foreground image will not be repeatedly rendered according to the panoramic image, thereby ensuring the accuracy of the rendering.
[0012] In a possible implementation manner of the first aspect, the method further includes: comparing the depth value of the second pixel and the second depth value through a vertex shader, wherein the second depth value is configured in the vertex shader.
[0013] In this possible implementation, a graphics processing unit (GPU) generally includes a vertex shader and a fragment shader, wherein the vertex shader is generally used to locate pixel points, and the fragment shader is used to color, that is, to write color values into a color buffer. The second depth value is configured in the vertex shader, and the second depth value can be compared with the depth value of the second pixel point when locating the pixel point, so as to determine whether the second pixel point needs to be rendered according to the panoramic image.
[0014] In a possible implementation of the first aspect, the above steps: rendering a foreground image and setting a depth value for each pixel on the canvas include: detecting a transparency value of an alpha channel of a pixel corresponding to the foreground image, the transparency value being used to indicate a degree of transparency of the pixel; if the transparency value is non-zero, writing a color value of the foreground image to a first position in a color buffer, and writing a first depth value to a second position in a depth buffer, the first position being a position of a pixel corresponding to the foreground image in the color buffer, and the second position being a position of a pixel corresponding to the foreground image in the depth buffer.
[0015] In this possible implementation, the display buffer (frame buffer) related to rendering can be divided into a color buffer and a depth buffer, wherein the color buffer is used to cache the color values of pixels on the canvas, and the depth buffer is used to cache the depth values of pixels on the canvas. For example: caching the first depth value, the depth reference value, or the second depth value. Each pixel has an alpha channel, and the alpha channel has 256 levels of color values, from 0 to 255, where 0 represents a transparency value of 0, that is, the pixel needs to remain transparent in the foreground image and does not need to be rendered. Other non-zero transparency values from 1 to 255 need to be rendered according to the color value of the foreground image. When rendering the foreground image, the depth values of the pixels in the depth buffer are all depth reference values. If the transparency value is 0, it means that the pixel needs to remain transparent in the foreground image, so a panoramic image is needed to fill it. The depth value of the pixel in the depth buffer does not need to be modified. If the transparency value is not 0, it means that the pixel needs to be filled with the color in the foreground image and does not need to be filled with a panoramic image. In this case, the depth value of the pixel in the depth buffer needs to be modified from the depth reference value to the first depth value, for example: from 0 to -0.5.
[0016] In a possible implementation manner of the first aspect, the method further includes: if the transparency value is 0, interrupting the operation of writing the color value to the color buffer.
[0017] In this possible implementation, if the transparency value is 0, it means that there is no need to write the color value of the foreground image to the color buffer. Writing the color value is usually performed by the fragment shader. When the transparency value is 0, just interrupt the fragment shader to write the color value of the pixel.
[0018] In a possible implementation manner of the first aspect, the above step: writing the first depth value to the second position in the depth buffer includes: writing the first depth value to the second position in the depth buffer through a fragment shader, and the fragment shader is configured with the first depth value.
[0019] In this possible implementation, a first depth value is configured in a fragment shader. During the shading process, if it is determined through judgment of the color value that the depth reference value in the depth buffer needs to be modified to the first depth value, it can be written according to the first depth value configured in the fragment shader, thereby improving the efficiency of writing the first depth value.
[0020] In a possible implementation manner of the first aspect, the method further includes: in response to a position selection operation on the foreground image, acquiring a rendering position of the foreground image; and determining a pixel point corresponding to the foreground image according to the rendering position.
[0021] In this possible implementation, the user can flexibly place the overlapping position of the foreground image in the panoramic image according to needs. After the user selects the position, the terminal device will determine the corresponding pixel points of the foreground image on the canvas according to the user-selected position, and then render it, which can improve the rendering efficiency.
[0022] The second aspect of the present application provides a terminal device, which has the function of implementing the method of the first aspect or any possible implementation of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as: an input unit and a processing unit.
[0023] A third aspect of the present application provides a terminal device, which includes at least one central processing unit (CPU), a GPU, a memory, a transceiver, and computer execution instructions stored in the memory and executable on the CPU. When the computer execution instructions are executed by the CPU, the CPU and the GPU execute the method described in the first aspect or any possible implementation of the first aspect.
[0024] A fourth aspect of the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.
[0025] The fifth aspect of the present application provides a computer program product (or computer program) storing one or more computer execution instructions. When the computer execution instructions are executed by the central processing unit, the central processing unit and the GPU execute the method of the above-mentioned first aspect or any possible implementation method of the first aspect.
[0026] In a sixth aspect of the present application, a chip system is provided, which includes a central processing unit and a GPU, and is used to support a terminal device to implement the functions involved in the above-mentioned first aspect or any possible implementation of the first aspect. In a possible design, the chip system may also include a memory, which is used to store program instructions and data necessary for the terminal device. The chip system may be composed of a chip, or may include a chip and other discrete devices.
[0027] Among them, the technical effects brought about by the second to sixth aspects or any possible implementation methods thereof can refer to the technical effects brought about by the first aspect or different possible implementation methods of the first aspect, and will not be repeated here.
[0028] The embodiment of the present application renders the foreground image first, and then renders the corresponding content of the panoramic image for the pixels not occupied by the foreground image. In this way, repeated rendering of the overlapping parts of the foreground image and the panoramic image is avoided, which not only reduces resource waste and rendering overhead, but also improves rendering efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 This is a schematic diagram of a scene of a panoramic application provided by an embodiment of the present application;
[0030] Figure 2 is a schematic diagram of an embodiment of a method for image rendering in a panoramic application provided by an embodiment of the present application;
[0031] Figure 3 is a schematic diagram of another embodiment of the method for image rendering in a panoramic application provided by an embodiment of the present application;
[0032] Figure 4A is a schematic diagram of another embodiment of the method for image rendering in a panoramic application provided by an embodiment of the present application;
[0033] Figure 4B is a schematic diagram of another embodiment of the method for image rendering in a panoramic application provided by an embodiment of the present application;
[0034] Figure 4C is a schematic diagram of another embodiment of the method for image rendering in a panoramic application provided by an embodiment of the present application;
[0035] Figure 5 This is a schematic diagram of a scene of image rendering in a panoramic application provided by an embodiment of the present application;
[0036] Figure 6 is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application;
[0037] Figure 7 This is another structural diagram of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0038] The following describes the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the described embodiments are only embodiments of a part of the present application, rather than all embodiments. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0039] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0040] The embodiment of the present application provides an image rendering method for reducing rendering overhead and avoiding waste of resources caused by repeated rendering. The embodiment of the present application also provides a corresponding terminal device. The following are detailed descriptions.
[0041] The panoramic applications involved in the embodiments of the present application may include panoramic car viewing, panoramic house viewing, panoramic hotel experience, map street view and artificial intelligence (AI) video background replacement for video chat, as well as other applications applied to panoramic images.
[0042] Figure 1 This is a schematic diagram of a panoramic application scenario.
[0043] like Figure 1As shown, in a panoramic application, panoramic pictures and foreground pictures are used. Foreground pictures take people as an example, and panoramic pictures take landscape pictures as an example. Users or program developers need to superimpose the foreground pictures into the panoramic pictures. Taking the operation of a user in a panoramic application through a terminal device such as a mobile phone as an example, the user opens the panoramic application on the mobile phone, selects the panoramic picture, and can determine the part to be selected by viewing angle selection. Then, the mobile phone will render the foreground picture to the display buffer, and then render the panoramic picture determined by viewing angle selection to the display buffer, so as to realize the superposition of the foreground picture and the panoramic picture.
[0044] Figure 1 The example description only takes a mobile phone as an example. In fact, the terminal device loaded with the panoramic application is not limited to a mobile phone, but can also be a tablet computer (pad), a personal computer (PC), a virtual reality (VR) terminal, an augmented reality (AR) terminal, a wireless terminal in industrial control, a wireless terminal in self driving, a wireless terminal in remote medical, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, etc.
[0045] In the above Figure 1 In the example of , the foreground image is rendered first, and then the panoramic image is rendered. The pixels occupied by the foreground image will not be repeatedly rendered when rendering the panoramic image. The embodiment of this process can be referred to Figure 2 To understand.
[0046] Figure 2 A schematic diagram of an embodiment of a method for image rendering in a panoramic application provided in an embodiment of the present application.
[0047] like Figure 2 As shown, an embodiment of the method for image rendering in a panoramic application provided by an embodiment of the present application may include:
[0048] 101. The terminal device responds to the operation on the panoramic application and obtains the foreground image and the panoramic image.
[0049] The panoramic image is the background image of the foreground image.
[0050] The foreground image can be a person, a ground icon, or other images that are different from the background image. The background image can also be a non-panoramic image, and the size of the background image is larger than the size of the foreground image. When the background image is a non-panoramic image, the implementation principles and steps of the panoramic image in the rendering process of the superimposed foreground image are basically the same as those of the present application, and the perspective selection may not be performed.
[0051] 102. The terminal device renders the foreground image and sets the depth value of each pixel on the canvas.
[0052] The depth value of each pixel includes a first depth value and a depth reference value. The first depth value is the depth value of the pixel corresponding to the foreground image, and the depth reference value is the depth value of the pixel excluding the pixel corresponding to the foreground image. The canvas is the area on the screen of the terminal device used to display the panoramic image and the foreground image.
[0053] For a canvas, the number of pixels on the canvas is fixed. The pixels on the canvas can be regarded as a pixel set A, where the set of pixels occupied by the foreground image can be described as subset B, and the set of pixels remaining after removing subset B can be described as subset C. The pixels of subset B and subset C do not overlap, and subset B + subset C = set A.
[0054] For the convenience of description, the "first depth value" refers to the depth value of the pixel corresponding to the foreground image, and the depth reference value is a reference value. The first depth value and the depth reference value are respectively used to indicate different states of the pixel. The first depth value is used to indicate that the corresponding pixel does not need to be repeatedly rendered, and the depth reference value indicates that the corresponding pixel has not been rendered. The first depth value can be represented by a specific numerical value, such as -0.5, and the depth reference value can be represented by 0. Of course, other numerical values can also be used. In addition, the first depth value and the depth reference value in the present application are not limited to numerical values. Other characters that can indicate whether the pixel needs to be rendered are also acceptable. For example: T means that repeated rendering is not required, and F means that the pixel has not been rendered. Of course, other characters can also be used to indicate it, as long as the indication is clear. The depth value does not represent a specific physical concept, and it can be expressed by other names, as long as it can express whether the pixel can be rendered according to the panoramic image.
[0055] If the depth reference value is 0, it can also be understood that before each foreground image is rendered, the depth values of the pixels on the canvas are cleared, and then the first depth value of the foreground image is written to the corresponding pixel points of the foreground image when the foreground image is rendered. If the depth reference value is not 0, it can also be understood that before each foreground image is rendered, a unified assignment operation of the depth values of the pixels on the canvas is performed, and the depth values of the pixels on the canvas are first assigned to the depth reference value, and then when the foreground image is rendered, the depth reference value is modified to the first depth value at the corresponding pixel points of the foreground image.
[0056] 103. The terminal device renders a color value corresponding to the first pixel point on the panoramic image at the first pixel point according to the first depth value and the depth reference value.
[0057] The first pixel point is a pixel point corresponding to the depth reference value.
[0058] The solution provided in the embodiment of the present application first renders the foreground image, and then renders the corresponding content of the panoramic image for the pixels not occupied by the foreground image. In this way, repeated rendering of the overlapping parts of the foreground image and the panoramic image is avoided, which not only reduces resource waste and rendering overhead, but also improves rendering efficiency.
[0059] The above step 103 can be implemented by two solutions, which are introduced below respectively.
[0060] Solution 1: A solution implemented by identifying a first depth value and a depth reference value.
[0061] After the foreground image is rendered, the panoramic image is rendered. The panoramic image may be rendered pixel by pixel. The rendered pixel may be referred to as the "second pixel".
[0062] When rendering pixel by pixel, if the depth value of the second pixel is the first depth value, the color value corresponding to the second pixel on the panoramic image is interrupted, and the second pixel is any pixel on the canvas; if the depth value of the second pixel is the depth reference value, the color value corresponding to the second pixel on the panoramic image is rendered at the second pixel.
[0063] If the depth value of the second pixel is the first depth value, it means that the second pixel has been occupied by the foreground image and does not need to be rendered again, so the rendering of the second pixel is interrupted. If the depth value of the second pixel is the depth reference value, it means that the second pixel is not occupied by the foreground image, so the rendering is performed according to the content of the panoramic image corresponding to the second pixel.
[0064] For example, if the first depth value is -0.5 and the depth reference value is 0, if the depth value of the second pixel is -0.5, it means that the second pixel has been occupied by the foreground image and does not need to be rendered again, so the rendering of the second pixel is interrupted. If the depth value of the second pixel is 0, it means that the second pixel is not occupied by the foreground image, so the rendering is performed according to the content of the panoramic image corresponding to the second pixel.
[0065] In this way, the indication of whether a pixel needs to be rendered by the first depth value and the depth reference value can ensure that the pixel already occupied by the foreground image will not be repeatedly rendered according to the panoramic image, thereby ensuring the accuracy of the rendering.
[0066] Solution 2: A solution implemented by comparing the depth value of the second pixel with the second depth value.
[0067] The "second depth value" refers to the depth value of the panoramic image. The first depth value and the second depth value are relative. Because the panoramic image is the background image of the foreground image, the second depth value must be greater than the first depth value. The first depth value and the second depth value are both less than the depth reference value. The first depth value, the second depth value and the depth reference value are usually pre-configured. The panoramic image can be rendered pixel by pixel, and the rendered pixel can be referred to as the "second pixel".
[0068] When rendering pixel by pixel, if the depth value of the second pixel is less than the second depth value, the color value corresponding to the second pixel on the panoramic image is interrupted, the second depth value is greater than the first depth value and less than the depth reference value, and the second pixel is any pixel on the canvas; if the depth value of the second pixel is greater than the second depth value, the color value corresponding to the second pixel on the panoramic image is rendered at the second pixel.
[0069] If the depth value of the second pixel is less than the second depth value, it means that the second pixel has been occupied by the foreground image and does not need to be rendered again, so the rendering of the second pixel is interrupted. If the depth value of the second pixel is greater than the second depth value, it means that the second pixel is not occupied by the foreground image, so the rendering is performed according to the content of the panoramic image corresponding to the second pixel.
[0070] For example, if the first depth value is -0.5, the depth reference value is 0, and the second depth value is -0.2. If the depth value of the second pixel is less than -0.2, it means that the depth value of the second pixel is -0.5, which means that the second pixel has been occupied by the foreground image and does not need to be rendered repeatedly, so the rendering of the second pixel is interrupted. If the depth value of the second pixel is greater than -0.2, it means that the depth value of the second pixel is 0, which means that the second pixel is not occupied by the foreground image, so the content of the panoramic image corresponding to the second pixel is rendered.
[0071] In this way, by comparing the first depth value, the depth reference value and the second depth value to determine whether the second pixel needs to be rendered, it can be ensured that the pixels occupied by the foreground image will not be repeatedly rendered according to the panoramic image, thereby ensuring the accuracy of rendering.
[0072] The rendering process is usually completed by a graphics processing unit (GPU), which includes a vertex shader and a fragment shader. The vertex shader is usually used to locate pixels, and the fragment shader is used to color, that is, to write color values into a color buffer. The second depth value is configured in the vertex shader, and the second depth value can be compared with the depth value of the second pixel when locating the pixel, so as to determine whether the second pixel needs to be rendered according to the panoramic image.
[0073] That is to say, a second depth value, for example -0.2, can be written in the vertex shader script. In this way, the vertex shader can compare -0.2 with -0.5 when locating the pixel, or compare the size relationship between -0.2 and 0 to determine whether to render the second pixel. If rendering is required, the fragment shader will perform the writing of the color value.
[0074] The above step 102 can be implemented by the following embodiments: detecting the transparency value of the alpha channel of the pixel point corresponding to the foreground image, and the transparency value is used to indicate the degree of transparency of the pixel point; if the transparency value is non-zero, writing the color value of the foreground image to the first position in the color buffer, and writing the first depth value to the second position in the depth buffer, the first position is the position of the pixel point corresponding to the foreground image in the color buffer, and the second position is the position of the pixel point corresponding to the foreground image in the depth buffer.
[0075] If the transparency value is 0, the color value writing operation to the color buffer is interrupted.
[0076] In this possible embodiment, the display buffer (frame buffer) related to rendering can be divided into a color buffer and a depth buffer, wherein the color buffer is used to cache the color values of the pixels on the canvas, and the depth buffer is used to cache the depth values of the pixels on the canvas. For example: caching the first depth value, the depth reference value, or the second depth value. Each pixel has an alpha channel, and the alpha channel has 256 levels of color values, from 0 to 255, where 0 represents a transparency value of 0, that is, the pixel needs to remain transparent in the foreground image and does not need to be rendered. Other non-zero transparency values from 1 to 255 need to be rendered according to the color value of the foreground image.
[0077] When rendering the foreground image, the depth values of the pixels in the depth buffer are all depth reference values. If the transparency value is 0, it means that the pixel needs to remain transparent in the foreground image, so a panoramic image is needed to fill it. The depth value of the pixel in the depth buffer does not need to be modified. If the transparency value is not 0, it means that the pixel needs to be filled with the color in the foreground image and does not need to be filled with a panoramic image. In this case, the depth value of the pixel in the depth buffer needs to be modified from the depth reference value to the first depth value, for example: from 0 to -0.5.
[0078] If the transparent value is 0, it means that there is no need to write the color value to the color buffer. Writing the color value is usually performed by the fragment shader. When the transparent value is 0, just interrupt the fragment shader to write the color value of the pixel.
[0079] The method for image rendering in a panoramic application provided by an embodiment of the present application also includes: obtaining a rendering position of the foreground image in response to a position selection operation on the foreground image; and determining a pixel point corresponding to the foreground image according to the rendering position.
[0080] In an embodiment of the present application, the user can flexibly place the overlapping position of the foreground image in the panoramic image according to needs. After the user selects the position, the terminal device will determine the pixel points corresponding to the foreground image on the canvas according to the position selected by the user, and then render it, which can improve the efficiency of rendering.
[0081] The above describes the method of image rendering in a panoramic application from the perspective of the terminal device as a whole. The following describes the process of calling the panoramic application to the GPU to perform rendering and display from a modular perspective.
[0082] Figure 3 It is a flowchart of the rendering process provided by an embodiment of the present application.
[0083] like Figure 3As shown, the rendering process involves a panoramic application (APP) 201, a panoramic (Panorama) software development kit (SDK) 202, a panoramic function library 203, a system graphics library 204, a graphics processor (GPU) 205 display buffer 206 and a display device 207, wherein the panoramic application 201 is used to interact with the user. The panoramic SDK 202 is equivalent to an interface, which can be connected to the panoramic function library 203. The panoramic function library 203 may include an overlay algorithm for the foreground image and the panoramic image, and the overlay algorithm may include clearing the depth value and the color value in the display buffer, identifying the depth value, and rendering the foreground image and the panoramic image according to the depth value. The panoramic function library 203 includes the algorithm, and the specific process may be that the panoramic function library 203 calls the system graphics library 204 to write the depth value and the color value into the display buffer 206 through the GPU 205, and displays the overlay rendering result through the display device 207.
[0084] Above Figure 3 The panoramic function library 203 in Figure 4A As shown, a rendering initialization module 2031 , a foreground image rendering module 2032 and a panoramic image rendering module 2033 may be included.
[0085] like Figure 4A In the process shown, the panoramic application 201 implements specific panoramic application functions by calling the panoramic SDK 202, such as realizing street view browsing and real-time video background replacement. The panoramic function library 203 is a dynamic library of panoramic picture display functions deployed on the terminal device. The rendering initialization module 2031, the rendering foreground picture module 2032 and the rendering panoramic picture module 2033 included in the panoramic function library 203 call these three modules once to complete the display buffer clearing when the panoramic application renders a frame of the picture, and then renders the foreground picture, and finally renders the panoramic picture as the background to complete the superposition of a frame of panoramic pictures. The panoramic function library 203 calls the system graphics library 204 to operate the GPU 205, and records each step of the operation in the display buffer 206. After that, the system will display the content in the display buffer 206 on the display device 207 at a fixed time, thereby completing the result of the superposition of the panoramic picture and the foreground picture.
[0086] The display buffer 206 may include a color buffer and a depth buffer. Before rendering the foreground image, the rendering initialization module 2031 may clear the color value in the color buffer and the depth value in the depth buffer to avoid interference of the previous result on the current rendering.
[0087] The rendering process of the foreground image in combination with the above modules may include: the panoramic application 201 passes the foreground image to the panoramic function library 203 through the interface of the panoramic SDK 202. The foreground image rendering module 2032 can call the fragment shader in the GPU through the system graphics library 204 to determine whether the transparency value of the alpha channel of the current pixel (the pixel to be rendered, refer to the second pixel mentioned above) is 0. If the transparency value is zero, it means that the current pixel is transparent, and it is necessary to interrupt the fragment shader to continue execution, and keep the depth reference value of the pixel in the depth buffer (for example: 0) unchanged. If it is determined that the transparency value of the alpha channel of the current pixel is not 0, it means that the pixel has color, then the color value of the foreground image is written into the color buffer, and the depth value corresponding to the pixel is written as the first depth value (for example -0.5). In this way, the color value of the foreground image has been saved in the corresponding position in the display buffer, and the depth buffer value of the non-transparent part is set to -0.5. Among them, the logic in the foreground image rendering module 2032 may include Figure 4B The following steps are shown.
[0088] 2032-1. The foreground image rendering module 2032 receives the foreground image.
[0089] 2032-2. Rendering foreground image module 2032 starts rendering of the foreground image.
[0090] 2032-3. The foreground image rendering module 2032 can call the fragment shader in the GPU through the system graphics library 204 to determine whether the transparency value of the alpha channel of the current pixel is 0. If it is 0, execute step 2032-4; if not 0, execute step 2032-5.
[0091] 2032-4. If the transparency value is zero, the foreground image rendering module 2032 interrupts the fragment shader to continue executing. The foreground image rendering module 2032 keeps the depth reference value of the pixel point in the depth buffer unchanged.
[0092] 2032-5. If the transparency value of the alpha channel of the current pixel is not 0, the foreground image rendering module 2032 writes the color value of the foreground image into the color buffer through the fragment shader, and writes the depth value corresponding to the pixel in the depth buffer into the first depth value.
[0093] After the foreground image is rendered, the panoramic image will be rendered.
[0094] The panoramic image rendering module 2033 receives the panoramic image transmitted by the panoramic application 201 through the interface of the panoramic SDK 202. The panoramic image rendering module 2033 calls the vertex shader in the GPU for rendering through the unified graphics library 204. The depth value of the panoramic image is set to -0.2 in the vertex shader script. The GPU's Early-Depth-Test is turned on. The GPU will calculate the depth value of the current pixel (refer to the second pixel mentioned above). If the depth value of the foreground image is less than the depth value of the pixel in the depth buffer, it means that the pixel in the depth buffer does not render the color of the foreground image. The GPU will write the color value of the corresponding pixel in the panoramic image to the color buffer. If the depth value of the current pixel is greater than the depth value of the pixel in the depth buffer (for example: -0.2>-0.5), the GPU Early-Depth-Test fails, the fragment shader is not executed, and the color value of the corresponding pixel in the color buffer and the depth value in the depth buffer remain unchanged. Among them, the logic in the panoramic image rendering module 2033 may include Figure 4C The following steps are shown.
[0095] 2033-1. The panoramic image rendering module 2033 receives the panoramic image.
[0096] 2033-2. The panoramic image rendering module 2033 simulates the two-dimensional plane into a three-dimensional space.
[0097] 2033 - 3. The panoramic image rendering module 2033 sets the depth value of the panoramic image to the second depth value in the vertex shader script.
[0098] 2033-4. The panoramic image rendering module 2033 confirms that the Early-Depth-Test function of the GPU is turned on.
[0099] If it is not enabled, the GPU's Early-Depth-Test function will be enabled.
[0100] 2033-5. The panoramic image rendering module 2033 calls the vertex shader in the GPU to compare the depth value of the current pixel with the second depth value. If the depth value of the current pixel is less than the second depth value, step 2033-6 is executed. If the depth value of the current pixel is greater than the second depth value, step 2033-7 is executed.
[0101] 2033-6. If the depth value of the current pixel is less than the second depth value, the panoramic image rendering module 2033 triggers a GPU Early-Depth-Test failure and does not execute the fragment shader.
[0102] 2033-7. If the depth value of the current pixel is greater than the second depth value, the panoramic image rendering module 2033 triggers the GPU to write the color value of the corresponding pixel in the panoramic image into the color buffer.
[0103] The following describes the above-mentioned overlay rendering process in conjunction with the video stream background replacement application. The portrait in the video can be extracted as the foreground image through AI cutout technology, and then the 360-degree panoramic image is used as the background image and the portrait as the foreground image is merged. The rendering process of this process in the embodiment of this application can be referred to. Figure 5 To understand, such as Figure 5 As shown, the component structure diagram of this embodiment mainly includes a panoramic picture 301 and a portrait 302 with the background removed as a foreground picture.
[0104] The rendering initialization module 2031 clears both the color buffer and the depth buffer. The cleared conditions in the color buffer and the depth buffer can be understood by referring to 3031 and 3032 .
[0105] The rendering foreground image module 2032 renders the character image 302 input by the user into the display buffer at the position specified by the user, and modifies the depth value of the pixel point at the position corresponding to the character image in the depth buffer to a first depth value (e.g., -0.5). In conjunction with this scene, after rendering the character image 302, the clearing of the color buffer and the depth buffer can be understood by referring to 3033 and 3034.
[0106] The panoramic picture rendering module 2033 continues the process of rendering the panoramic picture after the foreground picture rendering is completed. After the previous step is completed, all pixel values in the depth buffer are converted to the depth reference value (for example: 0) or the first depth value (for example: -0.5). When rendering the panoramic picture, the calculated depth calculation result is set to the second depth value (for example: -0.2) in the vertex shader. Combined with this scene, after rendering the panoramic picture 301, the cleared situation in the color buffer and the depth buffer can be understood by referring to 3035 and 3036.
[0107] The embodiment of the present application renders the foreground image first, so that the foreground image blocks the panoramic image through the depth value. Due to the function of GPU Early-Depth-Test, the GPU will automatically optimize and abandon the rendering of pixels whose depth value is greater than the depth value in the depth buffer. Compared with repeated rendering of the superimposed area, the solution provided by the embodiment of the present application is equivalent to rendering one less foreground image, avoiding repeated rendering, reducing resource waste, reducing rendering overhead, and improving rendering efficiency.
[0108] The above example only describes one foreground image as an example. Two or more foreground images can be superimposed on a panoramic image. No matter how many foreground images there are, the rendering principle is the same as the principle described in the above embodiment. If two or more foreground images serving as panoramic images overlap, different depth values can be assigned according to the arrangement of the two or more foreground images. When rendering, the foreground image with the smallest depth value is rendered first, and then the other foreground images are rendered in order of depth values. The rendering principle is the same as the rendering principle of the above embodiment.
[0109] The above describes the method of image rendering in a panoramic application. The following introduces the terminal device provided in an embodiment of the present application in conjunction with the accompanying drawings.
[0110] like Figure 6 As shown, an embodiment of the terminal device 40 provided in the embodiment of the present application includes:
[0111] An input unit 401, used to respond to operations on the panoramic application;
[0112] The processing unit 402 is used for:
[0113] Acquire a foreground image and a panoramic image, wherein the panoramic image is a background image of the foreground image;
[0114] Rendering the foreground image, and setting a depth value for each pixel on a canvas, wherein the depth value for each pixel includes a first depth value and a depth reference value, wherein the first depth value is a depth value of a pixel corresponding to the foreground image, and the depth reference value is a depth value of a pixel excluding a pixel corresponding to the foreground image, wherein the canvas is an area on a screen of a terminal device for displaying the panoramic image and the foreground image;
[0115] According to the first depth value and the depth reference value, a color value corresponding to the first pixel point is rendered on the panoramic image at the first pixel point, where the first pixel point is the pixel point corresponding to the depth reference value.
[0116] The solution provided in the embodiment of the present application is equivalent to rendering one less foreground image, avoiding repeated rendering, thereby reducing resource waste, reducing rendering overhead, and improving rendering efficiency.
[0117] In a possible embodiment, the processing unit 402 is configured to:
[0118] If the depth value of the second pixel is the first depth value, interrupting rendering of the color value corresponding to the second pixel on the panoramic image at the second pixel, where the second pixel is any pixel on the canvas;
[0119] If the depth value of the second pixel is the depth reference value, a color value corresponding to the second pixel on the panoramic image is rendered at the second pixel.
[0120] In a possible embodiment, the processing unit 402 is configured to:
[0121] If the depth value of the second pixel is less than a second depth value, interrupting rendering the color value corresponding to the second pixel on the panoramic image at the second pixel, the second depth value is greater than the first depth value and less than the depth reference value, and the second pixel is any pixel on the canvas;
[0122] If the depth value of the second pixel is greater than the second depth value, a color value corresponding to the second pixel on the panoramic image is rendered at the second pixel.
[0123] In a possible embodiment, the processing unit 402 is further configured to:
[0124] The depth value of the second pixel and the second depth value are compared through a vertex shader, and the second depth value is configured in the vertex shader.
[0125] In a possible embodiment, the processing unit 402 is configured to:
[0126] Detecting a transparency value of an alpha channel of a pixel corresponding to the foreground image, where the transparency value is used to indicate a degree of transparency of the pixel;
[0127] If the transparency value is not 0, the color value of the foreground image is written to the first position in the color buffer, and the first depth value is written to the second position in the depth buffer, where the first position is the position of the pixel corresponding to the foreground image in the color buffer, and the second position is the position of the pixel corresponding to the foreground image in the depth buffer.
[0128] In a possible embodiment, the processing unit 402 is further configured to:
[0129] If the transparency value is 0, the color value writing operation to the color buffer is interrupted.
[0130] In a possible embodiment, the processing unit 402 is configured to:
[0131] The first depth value is written into a second position in a depth buffer through a fragment shader, and the first depth value is configured in the fragment shader.
[0132] In a possible embodiment, the processing unit 402 is further configured to:
[0133] In response to a position selection operation on the foreground image, obtaining a rendering position of the foreground image;
[0134] A pixel point corresponding to the foreground image is determined according to the rendering position.
[0135] It should be noted that the information interaction, execution process, etc. between the various units of the above-mentioned terminal device 40 are based on the same concept as the method embodiment of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. The specific contents can be found in the description of the method embodiment shown in the previous description of the present application, and will not be repeated here.
[0136] Figure 7 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal provided in an embodiment of the present application. Figure 7 The mobile phone includes: a radio frequency (RF) circuit 510, a memory 520, an input unit 530, a display unit 540, a graphics processor (GPU) 550, an audio circuit 560, a wireless fidelity (WiFi) module 570, a processor 580, and a power supply 590. Those skilled in the art will understand that Figure 7 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0137] Combine the following Figure 7 A detailed introduction to the various components of the mobile phone:
[0138] The RF circuit 510 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 580 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 510 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the RF circuit 510 can also communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.
[0139] The memory 520 can be used to store software programs and modules. The processor 580 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 520. The memory 520 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 520 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0140] The input unit 530 can be used to receive user operation instructions and generate key signal inputs related to the user settings and function control of the mobile phone. Specifically, the input unit 530 may include a touch panel 531 and other input devices 532. The touch panel 531, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 531 or near the touch panel 531 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 531 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 580, and can receive and execute commands sent by the processor 580. In addition, the touch panel 531 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 531, the input unit 530 may also include other input devices 532. Specifically, other input devices 532 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, etc.
[0141] The display unit 540 can be used to display the superposition result of the foreground image and the panoramic image. The display unit 540 may include a display panel 541. Optionally, the display panel 541 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 531 may cover the display panel 541. When the touch panel 531 detects a touch operation on or near it, it is transmitted to the processor 580 to determine the type of touch event. Subsequently, the processor 580 provides a corresponding visual output on the display panel 541 according to the type of touch event. Although in Figure 7 In the embodiment, the touch panel 531 and the display panel 541 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 531 and the display panel 541 can be integrated to realize the input and output functions of the mobile phone.
[0142] The mobile phone may also include at least one sensor, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 541 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 541 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.
[0143] The graphics processing unit (GPU) 550 is used to execute the corresponding steps in the above-mentioned method embodiment of image rendering in panoramic application, such as: steps 10, 102 and 103, as well as steps such as clearing the color buffer and the depth buffer, foreground image rendering and panoramic image rendering.
[0144] The audio circuit 560, the speaker 561, and the microphone 562 can provide an audio interface between the user and the mobile phone. The audio circuit 560 can transmit the received audio data to the speaker 561 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 562 converts the collected sound signal into an electrical signal, which is received by the audio circuit 560 and converted into audio data, and then the audio data is output to the processor 580 for processing, and then sent to another mobile phone through the RF circuit 510, or the audio data is output to the memory 520 for further processing.
[0145] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 570. It provides users with wireless broadband Internet access. Figure 7 A WiFi module 570 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.
[0146] The processor 580 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 520, and calling data stored in the memory 520, it executes various functions of the mobile phone and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 580 may include one or more processing units; preferably, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 580.
[0147] The mobile phone also includes a power source 590 (such as a battery) for supplying power to various components. Preferably, the power source can be logically connected to the processor 580 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions.
[0148] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.
[0149] The present application also provides a chip system, which includes a central processing unit and a GPU, for supporting the above-mentioned terminal device to implement the functions involved, such as rendering foreground images and rendering panoramic images. In a possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the computer device. The chip system can be composed of a chip, or it can include a chip and other discrete devices.
[0150] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.
[0151] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.
[0152] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0153] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0154] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0155] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0156] In this application, "first", "second", "third", etc. are only used to distinguish expressions and do not have the meaning of limiting the order; in addition, the first object and the second object may refer to the same object in some cases; furthermore, since there is no limited order, there can be a second or third without a first.
[0157] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.
[0158] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for image rendering in a panoramic application, characterized in that: include: In response to an operation on the panoramic application, a foreground image and a panoramic image are acquired, wherein the panoramic image is a background image of the foreground image; Rendering the foreground image, and setting a depth value for each pixel on a canvas, wherein the depth value for each pixel includes a first depth value and a depth reference value, wherein the first depth value is a depth value of a pixel corresponding to the foreground image, and the depth reference value is a depth value of a pixel excluding a pixel corresponding to the foreground image, and the canvas is an area on a screen of a terminal device for displaying the panoramic image and the foreground image; wherein the first depth value is used to indicate that the corresponding pixel does not need to be repeatedly rendered, and the depth reference value indicates that the corresponding pixel needs to be rendered; According to the first depth value and the depth reference value, a color value corresponding to the first pixel point on the panoramic image is rendered at a first pixel point, where the first pixel point is a pixel point corresponding to the depth reference value.
2. The method according to claim 1, characterized in that The rendering, at a first pixel point, of a color value corresponding to the first pixel point on the panoramic image according to the first depth value and the depth reference value includes: If the depth value of a second pixel is the first depth value, interrupting rendering of the color value corresponding to the second pixel on the panoramic image at the second pixel, where the second pixel is any pixel on the canvas; If the depth value of the second pixel is the depth reference value, a color value corresponding to the second pixel on the panoramic image is rendered at the second pixel.
3. The method according to claim 1, characterized in that: The rendering, at a first pixel point, of a color value corresponding to the first pixel point on the panoramic image according to the first depth value and the depth reference value includes: If the depth value of the second pixel is less than a second depth value, interrupting rendering the color value corresponding to the second pixel on the panoramic image at the second pixel, the second depth value is greater than the first depth value and less than the depth reference value, and the second pixel is any pixel on the canvas; If the depth value of the second pixel is greater than the second depth value, a color value corresponding to the second pixel on the panoramic image is rendered at the second pixel.
4. The method according to claim 3, characterized in that The method further comprises: The depth value of the second pixel and the second depth value are compared through a vertex shader, and the second depth value is configured in the vertex shader.
5. The method according to any one of claims 1 to 4, characterized in that: The rendering of the foreground image and setting the depth value of each pixel on the canvas includes: Detecting a transparency value of an alpha channel of a pixel point corresponding to the foreground image, wherein the transparency value is used to indicate a degree of transparency of the corresponding pixel point; If the transparency value is not 0, the color value of the foreground image is written to the first position in the color buffer, and the first depth value is written to the second position in the depth buffer, the first position being the position of the pixel corresponding to the foreground image in the color buffer, and the second position being the position of the pixel corresponding to the foreground image in the depth buffer.
6. The method according to claim 5, characterized in that The method further comprises: If the transparency value is 0, the color value writing operation to the color buffer is interrupted.
7. The method according to claim 5, characterized in that Writing the first depth value to a second position in the depth buffer comprises: The first depth value is written into a second position in a depth buffer through a fragment shader, and the first depth value is configured in the fragment shader.
8. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: In response to a position selection operation on the foreground image, obtaining a rendering position of the foreground image; A pixel point corresponding to the foreground image is determined according to the rendering position.
9. A terminal device, characterized in that: It includes a transceiver, a central processing unit, a memory and a graphics processor, wherein the central processing unit is coupled to the memory, and the memory is used to store a program; The central processor is used to respond to the operation of the panoramic application, obtain a foreground image and a panoramic image, and the panoramic image is the background image of the foreground image; The central processing unit is used to control the graphics processor to perform the following steps: Rendering the foreground image, and setting a depth value for each pixel on the canvas, wherein the depth value for each pixel includes a first depth value and a depth reference value, wherein the first depth value is a depth value of a pixel corresponding to the foreground image, and the depth reference value is a depth value of a pixel excluding a pixel corresponding to the foreground image, wherein the canvas is an area on a screen of a terminal device for displaying the panoramic image and the foreground image, wherein the first depth value is used to indicate that the corresponding pixel does not need to be repeatedly rendered, and the depth reference value indicates that the corresponding pixel needs to be rendered; According to the first depth value and the depth reference value, a color value corresponding to the first pixel point on the panoramic image is rendered at a first pixel point, where the first pixel point is a pixel point corresponding to the depth reference value.
10. The terminal device according to claim 9, characterized in that: The graphics processor is used for: If the depth value of a second pixel is the first depth value, interrupting rendering of the color value corresponding to the second pixel on the panoramic image at the second pixel, where the second pixel is any pixel on the canvas; If the depth value of the second pixel is the depth reference value, a color value corresponding to the second pixel on the panoramic image is rendered at the second pixel.
11. The terminal device according to claim 9, characterized in that: The graphics processor is used for: If the depth value of the second pixel is less than a second depth value, interrupting rendering the color value corresponding to the second pixel on the panoramic image at the second pixel, the second depth value is greater than the first depth value and less than the depth reference value, and the second pixel is any pixel on the canvas; If the depth value of the second pixel is greater than the second depth value, a color value corresponding to the second pixel on the panoramic image is rendered at the second pixel.
12. The terminal device according to claim 11, characterized in that: The graphics processor is used to compare the depth value of the second pixel with the second depth value through a vertex shader, and the second depth value is configured in the vertex shader.
13. The terminal device according to any one of claims 9 to 12, characterized in that: The graphics processor is used for: Detecting a transparency value of an alpha channel of a pixel corresponding to the foreground image, wherein the transparency value is used to indicate a degree of transparency of the pixel; If the transparency value is not 0, the color value of the foreground image is written to the first position in the color buffer, and the first depth value is written to the second position in the depth buffer, the first position being the position of the pixel corresponding to the foreground image in the color buffer, and the second position being the position of the pixel corresponding to the foreground image in the depth buffer.
14. The terminal device according to claim 13, characterized in that: The graphics processor is further configured to interrupt the operation of writing the color value to the color buffer if the transparency value is 0.
15. The terminal device according to claim 13, characterized in that: The graphics processor is used to write the first depth value into a second position in the depth buffer through a fragment shader, and the first depth value is configured in the fragment shader.
16. The terminal device according to any one of claims 9 to 12, characterized in that: The graphics processor is also used for: In response to a position selection operation on the foreground image, obtaining a rendering position of the foreground image; A pixel point corresponding to the foreground image is determined according to the rendering position.
17. A computer-readable storage medium comprising a program, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 8.
Citation Information
Patent Citations
Picture rendering method and device, storage medium, and electronic device
CN109461199A
Cited By
Image rendering method for panorama application, and terminal device
WO2021128929A1
