Image rendering method and device and electronic equipment

Abstract

The embodiment of the invention discloses an image rendering method and device and electronic equipment. The method comprises the steps that an image picture collected by a target camera is written into a target rendering texture; obtaining a width value and a height value of the target rendering texture in a three-dimensional space according to the resolution of the target rendering texture, the pixel ratio per unit and the zoom factor; constructing a cylinder model according to the width value and the height value of the target rendering texture in the three-dimensional space, and obtaining the horizontal resolution and the vertical resolution corresponding to the cylinder model; constructing a three-dimensional grid according to the horizontal resolution and the vertical resolution corresponding to the cylinder model; performing image rendering on the three-dimensional grid according to textures in the target rendering textures; according to the method, scene rendering is performed in combination with the rendering texture, and scene content does not need to be drawn on a buffer, so that one-time additional texture sampling is reduced, the whole rendering process is simplified, and meanwhile, the rendering quality can be improved.

Description

Technical Field

[0001] The present application relates to the field of image rendering technology; specifically, to an image rendering method, device and electronic device. Background Art

[0002] With the rapid development of virtual reality technology, users' requirements for immersive experience are increasing, especially in the design of user interface (UI). In a virtual reality environment, UI not only needs to be beautiful and practical, but also must be able to adapt to the characteristics of three-dimensional space and ensure good readability and ease of use at different viewing angles.

[0003] Currently, when rendering UI images in a 3D engine, the camera first renders the captured scene content to a buffer, then samples and synthesizes the image in the buffer through an asynchronous time warp thread, and then renders the scene content to the display; such multiple image renderings reduce the image rendering quality. Summary of the invention

[0004] To solve the above technical problems, the embodiments of the present application provide an image rendering method, device and electronic device, which solve the problem of low image rendering quality in the related art.

[0005] According to one aspect of an embodiment of the present application, an image rendering method is provided, the method comprising: constructing a target rendering texture according to a canvas size and a scaling factor of a three-dimensional engine; writing an image screen acquired by a target camera into the target rendering texture; obtaining a width value and a height value of the target rendering texture in a three-dimensional space according to a resolution of the target rendering texture, a pixel ratio per unit, and the scaling factor; constructing a cylinder model according to the width value and the height value of the target rendering texture in the three-dimensional space, and obtaining a horizontal resolution and a vertical resolution corresponding to the cylinder model; constructing a three-dimensional mesh according to the horizontal resolution and the vertical resolution corresponding to the cylinder model; and performing image rendering on the three-dimensional mesh according to the texture in the target rendering texture.

[0006] Optionally, constructing a target rendering texture according to a canvas size and a scaling factor of a three-dimensional engine includes: obtaining length and width values ​​of the canvas in three-dimensional space according to the canvas size and the scaling factor of the three-dimensional engine; obtaining an initial resolution according to the length and width values ​​and a preset ratio of pixels per unit; obtaining a resolution of a rendering texture according to the initial resolution and a preset rendering scale coefficient; and constructing the target rendering texture according to the resolution of the rendering texture.

[0007] Optionally, writing the image captured by the target camera into the target rendering texture includes: obtaining the texture coordinates corresponding to each pixel point in the image captured by the target camera in the target rendering texture; rendering the pixel value corresponding to each pixel point to the corresponding texture coordinates; obtaining the adjacent texture points corresponding to each sampling point in the target rendering texture; and updating the color value of the sampling point according to the color values ​​of the adjacent texture points.

[0008] Optionally, a cylindrical model is constructed according to the width value and height value of the target rendering texture in the three-dimensional space, and the horizontal resolution and vertical resolution corresponding to the cylindrical model are obtained, including: calculating the positive and negative boundary values ​​of the X-axis and the positive and negative boundary values ​​of the Y-axis of the world coordinate system according to the width value and height value of the target rendering texture in the three-dimensional space; placing the center point of the cylindrical model of the three-dimensional engine at the origin of the world coordinate axis to obtain the actual radius of the cylindrical model; obtaining the number of vertices per unit length by multiplying the number of vertices per degree by the conversion coefficient from radians to degrees; multiplying the number of vertices per unit length by the ratio of the positive boundary value of the X-axis to the actual radius of the cylindrical model to obtain the horizontal resolution corresponding to the cylindrical model; multiplying the horizontal resolution by the ratio of the positive boundary of the Y-axis to the positive boundary of the X-axis to obtain the vertical resolution corresponding to the cylindrical model.

[0009] Optionally, a three-dimensional mesh is constructed according to the horizontal resolution and the vertical resolution corresponding to the cylindrical model, including: calculating the UV coordinates corresponding to each pixel according to the horizontal resolution and the vertical resolution; calculating the corresponding vertex coordinates on the cylindrical model according to each UV coordinate; looping through the horizontal resolution and the vertical resolution to generate triangle indexes; and constructing a three-dimensional mesh according to the UV coordinates, vertex coordinates and triangle indexes.

[0010] Optionally, according to each UV coordinate, the corresponding vertex coordinates on the cylindrical model are calculated, including: obtaining the length value of the cylindrical model in the vertical direction according to the linear interpolation between the positive boundary value of the X-axis and the negative boundary value of the X-axis according to the parameter U of the UV coordinate; obtaining the height value of the cylindrical model in the vertical direction according to the linear interpolation between the positive boundary value of the Y-axis and the negative boundary value of the Y-axis according to the parameter V of the UV coordinate; obtaining the angle from the center of the cylinder to the vertex according to the ratio of the length value of the cylindrical model in the vertical direction to the actual radius of the cylindrical model; obtaining the X coordinate of the vertex according to the product of the sine value of the angle and the actual radius of the cylindrical model; using the height value of the cylindrical model in the vertical direction as the Y coordinate of the vertex; and obtaining the Z coordinate of the vertex according to the product of the cosine value of the angle and the actual radius of the cylindrical model minus the actual radius of the cylindrical model.

[0011] Optionally, constructing a three-dimensional mesh according to the UV coordinates, vertex coordinates and triangle indexes includes: creating a Mesh object and assigning the vertex coordinates to a vertex set of the Mesh object; mapping a texture to a Mesh surface according to the UV coordinates; defining the Mesh surface according to the triangle index; calculating an axis-aligned bounding box according to the vertex positions of the Mesh; and calculating the normal of each vertex according to the vertex positions of the Mesh and the triangle definition to form the three-dimensional mesh.

[0012] Optionally, image rendering is performed on the three-dimensional mesh according to the texture in the target rendering texture, including: converting the target rendering texture into a basic texture map; creating a material ball, and assigning the material ball to the mesh renderer of the cylinder model; setting the material properties of the material ball to the basic texture map; applying the material property block to the mesh renderer of the cylinder model to achieve image rendering of the three-dimensional mesh.

[0013] According to one aspect of an embodiment of the present application, an image rendering device is provided, the device comprising: a rendering texture construction module, used to construct a target rendering texture according to a canvas size and a scaling factor of a three-dimensional engine; a texture writing module, used to write an image screen acquired by a target camera into the target rendering texture; a width and height value acquisition module, used to obtain a width value and a height value of the target rendering texture in a three-dimensional space according to a resolution of the target rendering texture, a pixel ratio per unit and the scaling factor; a resolution acquisition module, used to construct a cylinder model according to the width value and the height value of the target rendering texture in the three-dimensional space, and obtain a horizontal resolution and a vertical resolution corresponding to the cylinder model; a three-dimensional grid construction module, used to construct a three-dimensional grid according to the horizontal resolution and the vertical resolution corresponding to the cylinder model; and an image rendering module, used to perform image rendering on the three-dimensional grid according to the texture in the target rendering texture.

[0014] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image rendering method in the above technical solution is implemented.

[0015] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions so that the electronic device implements an image rendering method as in the above technical solution.

[0016] According to one aspect of an embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the image rendering method in the above technical solution is implemented.

[0017] The technical solution provided by this application includes at least the following beneficial effects:

[0018] 1. The present application writes the scene content captured by the target camera into a rendering texture, constructs a cylinder model according to the resolution of the rendering texture, and uses the image texture in the rendering texture to render the three-dimensional grid corresponding to the cylinder model; therefore, the present invention combines the rendering texture for scene rendering without drawing the scene content to a buffer, thereby reducing an additional texture sampling, simplifying the entire rendering process, and improving the rendering quality.

[0019] 2. This application uses rendering textures in Unity to dynamically adjust the resolution, render the image at a high resolution and then scale it to the final display resolution, so that the edges are smoother, the jagged phenomenon is reduced, and the moiré phenomenon caused by the interference of the pixel grid and the texture details can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0021] Figure 1 The figure is a flow chart of an image rendering method provided by an embodiment of the present application;

[0022] Figure 2 Shown Figure 1 An exemplary flow chart of step S10 in FIG.

[0023] Figure 3 Shown Figure 1 An exemplary flow chart of step S40 in FIG.

[0024] Figure 4 Shown Figure 1 An exemplary flow chart of step S50 in FIG.

[0025] Figure 5 Shown Figure 4 An exemplary flow chart of step S52 in

[0026] Figure 6 The figure shows a schematic diagram of the relationship between UV coordinates and vertex coordinates provided in an embodiment of the present application;

[0027] Figure 7 The figure is a schematic diagram of the structure of an image rendering device provided by an embodiment of the present application;

[0028] Figure 8 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0029] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0030] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0031] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0032] It should also be noted that the "multiple" mentioned in this application refers to two or more than two. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship.

[0033] Figure 1 FIG. 1 is a flow chart of an image rendering method provided by an embodiment of the present application; Figure 1 As shown, the method specifically comprises the following steps:

[0034] Step S10: construct a target rendering texture according to the canvas size and scaling factor of the 3D engine.

[0035] like Figure 2 As described above, this embodiment constructs the target rendering texture according to the canvas size and scaling factor of the 3D engine and specifically includes the following steps:

[0036] Step S11: Obtain the length and width of the canvas in the three-dimensional space according to the canvas size and scaling factor of the three-dimensional engine.

[0037] In this embodiment, the three-dimensional engine includes the Unity game engine, and the canvas represents a container for all UI components in the three-dimensional engine; in one scene, multiple canvas objects are allowed to exist, and canvases can be nested.

[0038] In this embodiment, a new script is created in the canvas of the Unity game engine, and the script obtains the canvas size and scaling factor through the RectTransform component; wherein the RectTransform component is a component used in Unity to control the position and size of UI elements, and the scaling factor is the scaling factor of the parent element; in Unity, the scaling of UI elements is affected not only by their own scaling settings, but also by the scaling settings of all their parent elements, because Unity's UI system uses a mechanism called "relative layout", in which the transformation of child elements (including position, rotation and scaling) is calculated relative to their parent elements; therefore, the final size of the canvas is also affected by the scaling factor of its parent element.

[0039] Optionally, the canvas size includes an X component (length) and a Y component (width), and the scaling factor includes a first scaling factor corresponding to the X component and a second scaling factor corresponding to the Y component. Each component of the canvas size is multiplied by the corresponding scaling factor to obtain the actual length and width of the canvas in the three-dimensional space; for example, if the canvas size is 1×1, the first scaling factor is 0.5, and the second scaling factor is 1, then the length and width of the canvas in the three-dimensional space are 0.5×1.

[0040] Step S12: obtaining an initial resolution according to the length and width values ​​and a preset ratio of pixels per unit.

[0041] In this embodiment, the ratio of pixels per unit that drives the texture size is defined by a script in the canvas. The ratio of pixels per unit is used to convert the unit size of the canvas into a pixel size (i.e., the resolution size). The actual length and width of the canvas in three-dimensional space are multiplied by the ratio of pixels per unit to obtain its pixel size, i.e., the initial resolution. The API Mathf.RoundToInt (an API for converting floating-point numbers to integers) is used to round the converted floating-point numbers to integers, thereby obtaining the initial resolution of the integer.

[0042] Step S13: Obtaining the resolution of the rendered texture according to the initial resolution and a preset rendering scale factor.

[0043] In this embodiment, the script in the canvas defines a rendering scale factor. If additional resolution is required, this scale factor can be multiplied by the initial resolution to be used as the resolution of the final rendered texture. For example: the pixel size of a 1m*1m canvas object is 100*100, but it is hoped that this canvas can be rendered at a high resolution, that is, the initial resolution × rendering scale factor can be used. In this way, the resolution of the rendered image can be dynamically adjusted without changing the camera settings. By adjusting the integer rendering scale factor, a balance can be found between picture quality and rendering performance. For example, in a scene that needs to display a lot of details but has insufficient performance, the integer rendering scale factor can be appropriately reduced to improve performance; in a scene that has sufficient performance but requires higher picture quality, the value of the integer rendering scale factor can be increased.

[0044] Step S14: construct the target rendering texture according to the resolution of the rendering texture.

[0045] The image rendering method of this embodiment is applied to a head-mounted display; wherein the head-mounted display includes a UI camera and a main camera, and the script sets the rendering level of the UI camera to the UI layer through LayerMask (layer mask), and sets the rendering layer of the main camera to not render the UI; this step mainly uses the rendering level management to prevent the VR main camera (user perspective) from directly seeing the canvas and the UI elements on it.

[0046] Create new global variables, target render texture and camera parameters, and initialize the camera parameters. The initialization content is as follows: Set the camera's projection mode to orthogonal mode, which means that the camera has no perspective effect, limit the camera's visible distance to save performance, set the camera's clear flag to solid color, which means setting the camera to a solid color background, set the camera's background color to completely transparent to ensure that the UI screen is not affected by the background color, and check the UI layer in the camera rendering mask to ensure that this camera will only render UI elements.

[0047] In this embodiment, the width and height of the target rendering texture are equal to the size of the pixels to be rendered, and the color format of the rendering texture is specified to be ARGB32, indicating that each pixel uses 32 colors, including 8-bit red, 8-bit green, 8-bit blue, and 8-bit transparency; the read and write mode of the target rendering texture is specified to be "sRGB", and the target texture parameters of the UI camera are set to the above-mentioned newly created target rendering texture. In this way, the UI camera will not render the captured image content onto the screen of the head-mounted display, but will write the content into the constructed target rendering texture.

[0048] Step S20: writing the image captured by the camera into the target rendering texture.

[0049] In this embodiment, writing the image screen captured by the target camera into the target rendering texture specifically includes: obtaining the texture coordinates corresponding to each pixel point in the image screen captured by the target camera in the target rendering texture; rendering the pixel value corresponding to each pixel point to the corresponding texture coordinates; obtaining the adjacent texture points corresponding to each sampling point in the target rendering texture; and updating the color value of the sampling point according to the color value of the adjacent texture point.

[0050] It should be noted that the target camera is the UI camera, and the image captured by the target camera represents the UI image captured by the UI camera; the conversion from "sRGB" to linear space is applied when writing the texture, and the opposite conversion is applied when reading the texture to ensure that the color is displayed correctly on the display device, which is very important for processing lighting and color mixing; after the pixel is written to the texture, the texture pixels are sampled and interpolated, so as to produce smoother results when the texture is scaled; the specific writing process is as follows:

[0051] (1) Writing pixel values: During the texture mapping process, you first need to determine the texture coordinates corresponding to each pixel on the screen. Since there is usually no one-to-one correspondence between screen pixels and texture pixels, it is necessary to find the texture coordinates corresponding to each pixel in the image on the target rendering texture through UV coordinate mapping and other methods; then render the pixel value corresponding to each pixel point to the corresponding texture coordinates.

[0052] (2) Obtaining adjacent texture points: Take each texture point in the target rendering texture as a sampling point and find the four texture points closest to the sampling point on the texture. These four points are usually located at the upper left, upper right, lower left, and lower right positions of the sampling point.

[0053] (3) Horizontal interpolation: Linear interpolation is performed on the texture points on both sides of the sampling point in the horizontal direction. Specifically, the weighted sum of the color values ​​of the four pixels is calculated using the ratio of the x-coordinate (or U-coordinate) of the sampling point to the x-coordinates of the four adjacent texture points to obtain the intermediate color value of the sampling point in the horizontal direction.

[0054] (4) Vertical interpolation: The ratio of the y coordinate (or V coordinate) of the sampling point to the y coordinates of the four adjacent pixels is used to calculate the weighted sum of the color values ​​of the four pixels, thereby obtaining the intermediate color value of the sampling point in the vertical direction.

[0055] (4) Color value update: The sampled color value is updated by the intermediate color value of the sampling point in the horizontal direction and the intermediate color value in the vertical direction, thereby completing the texture rendering of the UI image.

[0056] It should also be noted that the height of the field of view in the orthogonal mode of the UI camera is set to half the height of the UI canvas from the center line to the top and bottom. The height of the pixel size is divided by the pixel ratio per unit (pixelsPerUnit) and then multiplied by 0.5 to ensure that the orthogonal camera field of view can just render the UI.

[0057] Step S30: Obtain a width value and a height value of the target rendering texture in a three-dimensional space according to the resolution, the pixel ratio per unit, and the scaling factor of the target rendering texture.

[0058] It should be noted that the resolution and pixel ratio of the target render texture are obtained through the script under the canvas, the column resolution of the target render texture is divided by the pixel ratio per unit to obtain the width value of the target render texture, the row resolution of the target render texture is divided by the pixel ratio per unit to obtain the height value of the target render texture, and the width and height values ​​of the target render texture are divided by the scaling factor respectively, so as to obtain the width and height values ​​of the target render texture in three-dimensional space; wherein, the scaling factor represents the scaling ratio of the current object and all its parent objects in the world space. If the canvas or any of its parent objects is scaled, the actual size of the render texture in the world space will also change accordingly. Therefore, the actual size of the target render texture without being affected by scaling is obtained by dividing by the scaling factor.

[0059] Step S40: construct a cylindrical model according to the width value and the height value of the target rendering texture in the three-dimensional space, and obtain the horizontal resolution and the vertical resolution corresponding to the cylindrical model.

[0060] like Figure 3 As shown, this embodiment constructs a cylindrical model according to the width value and height value of the target rendering texture in the three-dimensional space, and obtains the horizontal resolution and vertical resolution corresponding to the cylindrical model, specifically including the following steps:

[0061] Step S41, calculating the positive and negative boundary values ​​of the X axis and the positive and negative boundary values ​​of the Y axis of the world coordinate system according to the width value and the height value of the target rendering texture in the three-dimensional space.

[0062] It should be noted that the positive and negative boundary values ​​of the X-axis and Y-axis of the world coordinate system are calculated according to the width and height values ​​of the target rendering texture in the three-dimensional space. Assuming the origin coordinates of the world coordinate system are (0, 0), the positive boundary value of the X-axis = +half of the width value, the negative boundary value of the X-axis = -half of the width value, the positive boundary value of the Y-axis = +half of the height value, and the negative boundary value of the Y-axis = -half of the height value.

[0063] Step S42: Place the center point of the cylindrical model of the three-dimensional engine at the origin of the world coordinate axis to obtain the actual radius of the cylindrical model.

[0064] It should be noted that the center point of the built-in cylinder model in the 3D engine is placed at the origin of the world coordinate axis. However, the positive boundary value of the X-axis and the positive boundary value of the Y-axis of the world coordinate system represent the maximum size of the cylinder model in the horizontal direction and the vertical direction respectively. The actual radius of the cylinder model can be directly obtained through the script under the canvas.

[0065] Step S43: multiply the number of vertices per degree by the conversion coefficient from radians to degrees to obtain the number of vertices per unit length.

[0066] It should be noted that the number of vertices per degree is a preset parameter value, and the conversion factor from radians to degrees is equal to Multiply the number of vertices per degree by the radians to degrees conversion factor to get the number of vertices per unit length.

[0067] Step S44: multiply the number of vertices per unit length by the ratio of the positive boundary value of the X-axis to the actual radius of the cylindrical model to obtain the horizontal resolution corresponding to the cylindrical model.

[0068] It should be noted that The horizontal resolution corresponding to the cylindrical model is obtained; the positive boundary value of the X-axis is the maximum size of the cylindrical model in the horizontal direction, which can be understood as the projection of the radius of the cylinder in the X-axis direction, and the actual radius of the cylinder is the radius of the cylinder after scaling; this calculation method gives a preliminary estimate of the horizontal resolution based on the horizontal size of the cylinder and the number of vertices per degree.

[0069] Step S45: multiply the horizontal resolution by the ratio of the positive boundary of the Y axis to the positive boundary of the X axis to obtain the vertical resolution corresponding to the cylindrical model.

[0070] It should be noted that the vertical resolution is calculated based on the horizontal resolution. The vertical resolution is thus obtained; this calculation method adjusts the vertical resolution according to the relative size of the cylinder in the vertical direction.

[0071] Step S50: construct a three-dimensional grid according to the horizontal resolution and vertical resolution corresponding to the cylindrical model.

[0072] The UV coordinates in this embodiment are two-dimensional coordinates, which are used here to map the two-dimensional texture image to the surface of the three-dimensional model; the UV coordinates are generated by calculating the relative position (u and v) of each vertex in the mesh and are usually normalized to the range of 0 to 1.

[0073] The pixel in this embodiment is the basic unit of the texture image, representing a point in the image; when the texture is mapped to the model, the UV coordinates determine which parts of the model surface correspond to which pixels on the texture image.

[0074] The resolution in this embodiment refers to the width and height of the texture image (in pixels); the higher the resolution of the texture, the more pixels it contains and the richer the details; the way the UV coordinates are generated (based on the resolution of the grid) and the final texture mapping effect are closely related.

[0075] The vertices in this embodiment are basic elements constituting a three-dimensional mesh. The vertex list contains the three-dimensional coordinates of all vertices required to generate a cylindrical mesh. Each vertex has a corresponding UV coordinate for determining the position of the vertex on the texture image.

[0076] The triangles in this embodiment are the basic polygons that make up the three-dimensional mesh, and the triangle list contains the vertex indices of all triangles that make up the cylindrical mesh; through these triangles, the vertices can be connected to form the surface of the mesh; when the texture is mapped to the mesh, each triangle will map a part of the texture image according to the UV coordinates of its vertices.

[0077] UV coordinates play a vital role in mapping textures to 3D meshes. They determine which pixels on the texture image will be mapped to which vertices of the mesh, which in turn affects the visual effect of the entire model. Vertices, triangles, and texture resolution are all important factors that affect the final rendering effect.

[0078] like Figure 4 As shown, in this embodiment, constructing a three-dimensional grid according to the horizontal resolution and vertical resolution corresponding to the cylindrical model specifically includes the following steps:

[0079] Step S51, calculating the UV coordinates corresponding to each pixel according to the horizontal resolution and the vertical resolution.

[0080] It should be noted that the horizontal resolution and vertical resolution corresponding to the cylindrical model represent the horizontal and vertical resolutions (or the number of columns and rows of the grid) respectively. and Loop through, X represents a constant, that is, the horizontal resolution, x represents a variable, and its value range is from 0 to X-1; similarly, Y represents a constant, that is, the vertical resolution, y represents a variable, and its value range is from 0 to Y-1; the relative position of the current point x in its row (i.e., the U coordinate) is ensured by dividing x by X-1, so that when x traverses from 0 to X-1, the value of u changes smoothly from 0 to 1; similarly, the relative position of y in its column (i.e., the V coordinate) also ranges from 0 to 1, and by dividing y by Y-1, it is ensured that when y traverses from 0 to Y-1, the value of v changes smoothly from 0 to 1; this is done to match the common texture coordinate system, where the lower left corner of the texture is (0,0) and the upper right corner is (1,1), so as to calculate the UV coordinate corresponding to each pixel.

[0081] Step S52, calculating the corresponding vertex coordinates on the cylindrical model according to each UV coordinate;

[0082] like Figure 5 As shown, this embodiment calculates the corresponding vertex coordinates on the cylindrical model according to each UV coordinate and specifically includes the following steps:

[0083] Step S521: Obtain the length value of the cylindrical model in the vertical direction according to the linear interpolation of the parameter U of the UV coordinate between the positive boundary value of the X-axis and the negative boundary value of the X-axis.

[0084] Step S522: Obtain the height value of the cylindrical model in the vertical direction according to the linear interpolation of the parameter V of the UV coordinate between the positive boundary value of the Y axis and the negative boundary value of the Y axis.

[0085] Step S523: Obtain the angle from the center of the cylinder to the vertex according to the ratio of the length of the cylinder model in the vertical direction to the actual radius of the cylinder model.

[0086] Step S524: Obtain the X coordinate of the vertex according to the product of the sine value of the angle and the actual radius of the cylindrical model.

[0087] Step S525: Use the height value of the cylindrical model in the vertical direction as the Y coordinate of the vertex.

[0088] Step S526: Obtain the Z coordinate of the vertex according to the product of the cosine value of the angle and the actual radius of the cylindrical model, and then subtract the actual radius of the cylindrical model.

[0089] It should be noted that the vertex coordinates include X coordinates, Y coordinates and Z coordinates, such as Figure 6 As shown, Figure 6The rectangle in the figure is a texture image composed of several UV coordinates. The arc surface composed of dotted lines represents part of the outer surface of the cylinder model. The points on the outer surface of the cylinder model are called vertices, and their corresponding coordinates are vertex coordinates. This embodiment calculates the corresponding vertex coordinates according to each UV coordinate. The specific steps are as follows:

[0090] (1) Parameter interpolation: Through the API interpolation function, according to the linear interpolation of the UV coordinate parameter U between the positive boundary value of the X axis and the negative boundary value of the X axis, the length value of the cylinder model in the vertical direction is calculated, which is recorded as float.x. Then, through the API interpolation function, according to the linear interpolation of the parameter V between the positive boundary value of the Y axis and the negative boundary value of the Y axis, the height value of the cylinder model in the vertical direction is calculated, which is recorded as float.y.

[0091] (2) Angle calculation: According to the placement direction of the cylinder, calculate the angle angle from the center of the cylinder to the point; for a vertically placed cylinder, the angle is calculated based on the length value float.x, that is, angle = float.x / scaledRadius; where scaledRadius represents the actual radius of the cylinder model, also known as the scaled radius of the cylinder.

[0092] (3) Coordinate calculation: Use trigonometric functions (sine and cosine) to convert the angle into the coordinates of a point on the surface of the cylinder, specifically:

[0093] The X coordinate of the vertex point.x = Mathf.Sin(angle)×scaledRadius, which is located on the circumference of the cylinder.

[0094] The Y coordinate of the vertex point.y = float.y, and the height value is directly used as the y coordinate of the point.

[0095] The Z coordinate of the vertex point.z = Mathf.Cos(angle)*scaledRadius-scaledRadius, where scaledRadius is subtracted to adjust the position of the center of the cylinder on the z-axis (assuming that the cylinder is placed along the z-axis and the bottom center is at the origin).

[0096] Step S53: loop through the horizontal resolution and the vertical resolution to generate triangle indexes.

[0097] Note that the grid is based on a two-dimensional resolution, where resolution.x and resolution.y represent the number of vertices of the width (number of columns) and height (number of rows) of the grid respectively; each grid cell is traversed through a double loop, and two triangles are generated for each internal cell; the specific steps are as follows:

[0098] (1) Outer loop (y-axis): for(int y = 0; y < resolution.y - 1; y++) This loop traverses each row (y coordinate) of the grid, starting from 0 until resolution.y - 2 (because the last row does not need to be processed, it serves as the starting boundary for the next row).

[0099] (2) Inner loop (x-axis): for(int x = 0; x < resolution.x - 1; x++) This loop traverses each column (x coordinate) of each row, starting from 0 until resolution.x - 2 (because the last column does not need to be processed, it serves as the starting boundary for the next column).

[0100] (3) Vertex index calculation:

[0101] int v00 = x + y * resolution.x;: Calculate the vertex index (v00) of the lower left corner of the current cell.

[0102] int v10 = v00 + 1;: Calculate the vertex index (v10) of the lower right corner of the current cell.

[0103] int v01 = v00 + resolution.x;: Calculate the vertex index (v01) of the upper left corner of the current cell.

[0104] int v11 = v00 + 1 + resolution.x;: Calculate the vertex index (v11) of the upper right corner of the current cell.

[0105] These indices are based on the assumption that the vertices are linearly stored from left to right and top to bottom. v00 is the lower left vertex of the currently processed cell, v10 is the lower right, v01 is the upper left (start of the next row), and v11 is the upper right (next vertex of the next row).

[0106] (4) Generate triangle indices:

[0107] tris.Add(v00), tris.Add(v11), tris.Add(v10) means adding the first triangle, which is composed of v00 (lower left corner), v11 (upper right corner), and v10 (lower right corner); tris.Add(v00), tris.Add(v01), tris.Add(v11) means adding the second triangle, which is composed of v00 (lower left corner), v01 (upper left corner), and v11 (upper right corner); for example, the first quadrilateral (composed of vertices 0, 1, 100, 101) will generate two triangles (0, 101, 1) and (0, 1, 100).

[0108] Step S54: construct a three-dimensional mesh according to the UV coordinates, vertex coordinates and triangle indexes.

[0109] It should be noted that the specific method of building a 3D mesh is to use the C# API of the Unity engine to create a Mesh object. Mesh is one of the core components used in Unity to represent the geometric shape of 3D objects. It contains key information such as vertices, normals, UV coordinates, triangles, etc. The specific steps are:

[0110] (1) Create a Mesh object: Mesh mesh = new Mesh(). The new keyword creates an instance of the Mesh class. This instance is currently an empty Mesh without any vertices, triangles, UV coordinates, or normals.

[0111] (2) Set vertices: mesh.SetVertices(verts), call the SetVertices method, and assign the vertices array (expected to be an array of Vector3[] type, where each Vector3 represents the 3D coordinates of a vertex) to the vertex set of the Mesh. This is the basis for building the Mesh, because the vertices define the shape and size of the Mesh.

[0112] (3) Set UV coordinates: mesh.SetUVs(0,uvs), the first set of UV coordinates (usually called UV0) of the Mesh is set through the SetUVs method. uvs is an array of Vector2[] type, where each Vector2 represents the UV coordinates of a vertex. UV coordinates are used to map textures to the surface of the Mesh, so this set of coordinates is crucial for the texture rendering of the Mesh.

[0113] (4) Set triangles: mesh.SetTriangles(tris,0), ​​define the triangle set of Mesh through the SetTriangles method. tris is an array of type int[], where each element is an index pointing to a vertex in the vertex array. Every three indexes form a triangle, and these triangles define the surface of Mesh. The second parameter (here 0) is usually used to specify which set of UV coordinates to use, but in this example, since only one set of UV coordinates is set, the default value 0 is used.

[0114] (5) Recalculate bounds: mesh.RecalculateBounds(). The RecalculateBounds method automatically calculates the axis-aligned bounding box of the Mesh based on the vertex positions of the Mesh. This is useful for determining the position, size, and shape of the Mesh, especially in physical collisions, rendering optimization, etc.

[0115] (6) Recalculate Normals: mesh.RecalculateNormals(), the RecalculateNormals method automatically calculates the normals of each vertex based on the vertex position and triangle definition of the Mesh. Normals are crucial for determining the lighting and shadowing direction of the Mesh surface because they determine how light interacts with the Mesh surface.

[0116] Therefore, this embodiment creates a Mesh object, defines its shape and texture mapping by setting vertices, UV coordinates, and triangles, and then improves the geometry and rendering properties of the Mesh by recalculating the boundaries and normals.

[0117] Step S60: performing image rendering on the three-dimensional mesh according to the texture in the target rendering texture.

[0118] It should be noted that the target rendering texture is first converted into a base texture map (Texture), where the base texture map (Texture) represents a texture format that can be converted to and from the target rendering texture (Render Textur); the target rendering texture (Render Textur) is usually used for advanced rendering technologies such as screen post-processing and off-screen rendering. It is connected to a FrameBufferObject, allowing the rendering results to be saved in a storage area of ​​the GPU for later use; and the texture map Texture is usually loaded from the hard disk into the memory, and then uploaded to the GPU for rendering.

[0119] Secondly, create a new material ball, get the cylinder's mesh renderer MeshRender component, and assign the material to the cylinder's MeshRender component.

[0120] Then, create a new MaterialPropertyBlock object, set its texture map to the base texture map Texture rendered by RenderTexture, and set the clipping threshold of the Alpha channel of its shader attribute shader to 1.

[0121] Finally, the set material property block MaterialPropertyBlock is applied to the mesh renderer MeshRender of the cylinder to realize the image rendering of the three-dimensional mesh; among them, the material property block MaterialPropertyBlock allows temporary change of material properties without affecting the material instance itself, which is very useful for batch rendering and performance optimization. This can ensure that the material properties can be changed more efficiently, especially when the properties need to be updated frequently, maintaining the flexibility of the material and reducing performance overhead.

[0122] It should also be noted that after the image rendering of the three-dimensional grid is completed, the image can be processed with special effects, specifically: based on Unity's ShaderLab (a shader writing syntax) syntax, set the shader main texture to specify the receiving basic texture map Texture, control the clipping threshold of the Alpha channel between 0 and 1, and make useless pixels transparent (for example, a 10*10 pixel image has only 5*5 pixels and other colorless pixels are white by default, which are useless pixels of the image). Set SubShader (a key part in Unity Shader, used to define different stages or methods of rendering) to render in the 'AlphaTest' (rendering queue is usually used for objects that need to be Alpha tested) rendering queue, ignore the projector to prevent shadows or light projections from affecting the object, exclude the 3D scene from interfering with it, mark the shader as a transparency clipping type, allow the rendering engine to perform specific optimization or processing on it, use the source color for blending, ignore the target color, and set the detail level to 100 for high-quality rendering.

[0123] Optionally, the core calculation of the Shader is: receive vertex data as input, the texture coordinates passed by the vertex shader, use the tex2D function to sample the main texture. The sampling result is a float4 type color value, and then alpha clipping is performed. If the alpha is less than the clipping threshold, the current fragment is clipped. Finally, the final color value is returned for rendering. The standard CG library file of Unity is introduced and defined as CG code to enable the code to run on the GPU.

[0124] For example: Suppose the sampled color of a fragment is (0.5, 0.2, 0.8, 0.4) (RGBA), and _Cutoff is set to 0.5. Calculate color.a (alpha value) - _Cutoff: 0.4-0.5 = -0.1. Since the result is negative, the clip function will discard the fragment. On the contrary, if the sampled color of the fragment is (0.5, 0.2, 0.8, 0.6), calculate color.a (alpha value) - _Cutoff: 0.6-0.5 = 0.1, the result is positive, so the fragment is retained and the color (0.5, 0.2, 0.8, 0.6) is returned.

[0125] Optionally, in the LaterUpdate function (a lifecycle method in Unity, which is called after all other Update methods, which ensures that the logic executed in this method is performed after all other frame update logic), when performing texture mapping, map the texture of RenderTexture (target render texture) to the newly created cylindrical material ball, obtain the temporary render texture by using the API RenderTexture.GetTemporary (get a temporary render texture of the same size and format), and use Graphics.CopyTexture (GPU-level, efficient texture copying) to copy the texture to improve performance; and release it through RenderTexture.ReleaseTemporary (the temporary render texture is returned to Unity's temporary render texture pool) after use before the next frame update to avoid memory leaks; after processing the texture, set the internal use of supersampling to submit the layer to the VR device; the layer has priority during rendering and can be rendered directly to the display, thereby avoiding the degradation of image quality caused by multiple texture sampling, thus completing the optimization of aliasing and moiré.

[0126] To summarize, the present application writes the scene content captured by the target camera into the rendering texture, constructs a cylindrical model according to the resolution of the rendering texture, and uses the image texture in the rendering texture to render the three-dimensional mesh corresponding to the cylindrical model; therefore, the present application combines the rendering texture for scene rendering, and there is no need to draw the scene content to the cache, thereby reducing an additional texture sampling, simplifying the entire rendering process, and improving the rendering quality.

[0127] In addition, the present application uses rendering textures in the Unity browser to dynamically adjust the resolution, render the image at a high resolution and then scale it to the final display resolution, thereby making the edges smoother and reducing aliasing, and also reducing the moiré phenomenon caused by the interference of the pixel grid and texture details.

[0128] Figure 7FIG. 1 is a schematic diagram of the structure of an image rendering device provided in an embodiment of the present application; Figure 7 As shown, the device comprises:

[0129] A rendering texture construction module 710 is used to construct a target rendering texture according to the canvas size and scaling factor of the 3D engine;

[0130] The texture writing module 720 is used to write the image captured by the target camera into the target rendering texture;

[0131] A width and height value acquisition module 730 is used to acquire a width value and a height value of the target rendering texture in a three-dimensional space according to the resolution, the pixel ratio per unit and the scaling factor of the target rendering texture;

[0132] A resolution acquisition module 740 is used to construct a cylindrical model according to the width value and the height value of the target rendering texture in the three-dimensional space, and obtain the horizontal resolution and the vertical resolution corresponding to the cylindrical model;

[0133] A three-dimensional mesh construction module 750 is used to construct a three-dimensional mesh according to the horizontal resolution and vertical resolution corresponding to the cylindrical model;

[0134] The image rendering module 760 is used to perform image rendering on the three-dimensional mesh according to the texture in the target rendering texture.

[0135] Figure 8 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0136] It should be noted that Figure 8 The computer system 1000 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0137] like Figure 8 As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage part 1008 to the random access memory (RAM) 1003, such as executing the method described in the above embodiment. In the RAM 1003, various programs and data required for system operation are also stored. The CPU 1001, the ROM 1002 and the RAM 1003 are connected to each other through the bus 1004. The input / output (I / O) interface 1005 is also connected to the bus 1004.

[0138] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0139] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1009, and / or installed from a removable medium 1011. When the computer program is executed by a central processing unit (CPU) 1001, various functions defined in the system of the present application are executed.

[0140] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0141] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0143] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. An image rendering method, characterized in that: The method comprises: Construct the target rendering texture according to the canvas size and scaling factor of the 3D engine; Writing the image captured by the target camera into the target rendering texture; Obtaining a width value and a height value of the target rendering texture in a three-dimensional space according to the resolution, the pixel ratio per unit, and the scaling factor of the target rendering texture; Constructing a cylindrical model according to the width value and the height value of the target rendering texture in the three-dimensional space, and obtaining the horizontal resolution and the vertical resolution corresponding to the cylindrical model; Constructing a three-dimensional grid according to the horizontal resolution and the vertical resolution corresponding to the cylindrical model; An image is rendered on the three-dimensional mesh according to a texture in the target rendering texture.

2. The method according to claim 1, characterized in that Construct the target rendering texture according to the canvas size and scaling factor of the 3D engine, including: Obtaining the length and width of the canvas in the three-dimensional space according to the canvas size and scaling factor of the three-dimensional engine; Obtaining an initial resolution according to the length and width values ​​and a preset ratio of pixels per unit; Obtaining a resolution of a rendering texture according to the initial resolution and a preset rendering scale factor; The target rendering texture is constructed according to the resolution of the rendering texture.

3. The method according to claim 1, characterized in that Writing the image captured by the target camera into the target rendering texture includes: Get the texture coordinates of each pixel in the image captured by the target camera in the target rendering texture; Render the pixel value corresponding to each pixel point to the corresponding texture coordinate; Get the adjacent texture points corresponding to each sampling point in the target rendering texture; Update the color value of the sampling point according to the color values ​​of the adjacent texture points.

4. The method according to claim 1, characterized in that: Constructing a cylindrical model according to the width value and the height value of the target rendering texture in the three-dimensional space, and obtaining the horizontal resolution and the vertical resolution corresponding to the cylindrical model, including: Calculate the positive and negative boundary values ​​of the X axis and the positive and negative boundary values ​​of the Y axis of the world coordinate system according to the width and height of the target rendering texture in the three-dimensional space; Place the center point of the cylinder model of the 3D engine at the origin of the world coordinate axis to obtain the actual radius of the cylinder model; The number of vertices per unit length can be obtained by multiplying the number of vertices per degree by the radian to degree conversion factor; The horizontal resolution corresponding to the cylindrical model is obtained by multiplying the number of vertices per unit length by the ratio of the positive boundary value of the X-axis to the actual radius of the cylindrical model; The horizontal resolution is multiplied by the ratio of the positive boundary of the Y axis to the positive boundary of the X axis to obtain the vertical resolution corresponding to the cylindrical model.

5. The method according to claim 1, characterized in that: Constructing a three-dimensional grid according to the horizontal resolution and vertical resolution corresponding to the cylindrical model includes: Calculate the UV coordinates corresponding to each pixel according to the horizontal resolution and the vertical resolution; According to each UV coordinate, calculate the corresponding vertex coordinates on the cylindrical model; Loop through the horizontal resolution and the vertical resolution to generate triangle indexes; A three-dimensional mesh is constructed according to the UV coordinates, vertex coordinates and triangle indices.

6. The method according to claim 5, characterized in that According to each UV coordinate, calculate the corresponding vertex coordinates on the cylindrical model, including: According to the linear interpolation of the parameter U of the UV coordinate between the positive boundary value of the X axis and the negative boundary value of the X axis, the length value of the cylinder model in the vertical direction is obtained; According to the linear interpolation of the parameter V of the UV coordinate between the positive boundary value of the Y axis and the negative boundary value of the Y axis, the height value of the cylinder model in the vertical direction is obtained; Obtaining the angle from the center of the cylinder to the vertex according to the ratio of the length value of the cylinder model in the vertical direction to the actual radius of the cylinder model; Obtain the X coordinate of the vertex according to the product of the sine value of the angle and the actual radius of the cylinder model; The height value of the cylinder model in the vertical direction is used as the Y coordinate of the vertex; The Z coordinate of the vertex is obtained by multiplying the cosine value of the angle by the actual radius of the cylindrical model and subtracting the actual radius of the cylindrical model.

7. The method according to claim 5, characterized in that Constructing a three-dimensional mesh according to the UV coordinates, vertex coordinates and triangle indexes, including: Create a Mesh object and assign the vertex coordinates to the vertex set of the Mesh object; Mapping the texture to the Mesh surface according to the UV coordinates; Define the Mesh surface according to the triangle index; Calculate the axis-aligned bounding box based on the vertex positions of the Mesh; The normal of each vertex is calculated according to the vertex position and triangle definition of the Mesh to form the three-dimensional mesh.

8. The method according to claim 4, characterized in that Performing image rendering on the three-dimensional mesh according to the texture in the target rendering texture includes: Convert the target render texture into a base texture map; Create a material ball and assign the material ball to the mesh renderer of the cylinder model; Setting the material properties of the material ball to the base texture map; The material attribute block is applied to the mesh renderer of the cylinder model to achieve image rendering of the three-dimensional mesh.

9. An image rendering device, characterized in that: The device comprises: A rendering texture building module is used to build a target rendering texture according to the canvas size and scaling factor of the 3D engine; A texture writing module, used for writing the image captured by the target camera into the target rendering texture; A width and height value acquisition module, used to acquire a width value and a height value of the target rendering texture in a three-dimensional space according to the resolution of the target rendering texture, the pixel ratio per unit and the scaling factor; A resolution acquisition module, used to construct a cylindrical model according to the width value and height value of the target rendering texture in three-dimensional space, and obtain the horizontal resolution and vertical resolution corresponding to the cylindrical model; A three-dimensional grid construction module, used to construct a three-dimensional grid according to the horizontal resolution and vertical resolution corresponding to the cylindrical model; An image rendering module is used to perform image rendering on the three-dimensional grid according to the texture in the target rendering texture.

10. An electronic device, characterized in that: include: processor; as well as A memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to enable the electronic device to implement the image rendering method as described in any one of claims 1 to 8.

Images