Image rendering method, device and equipment and readable storage medium

Abstract

The invention discloses an image rendering method, device and equipment and a readable storage medium, and belongs to the technical field of computers. The method comprises the steps that a mirror surface mask is determined based on camera parameters of a main camera, the main camera is used for imaging a mirror surface model and an object model, and the mirror surface mask is used for shielding other areas except a mirror surface area in a blank image; obtaining camera parameters of a mirror camera, wherein the mirror camera is used for imaging the object model based on the mirror mask; based on camera parameters of the mirror camera and a mirror mask, the object model is rendered in the blank image to obtain a first image, and a mirror area in the first image comprises a mirror image of an object represented by the object model; and based on camera parameters of the main camera, rendering the mirror surface model and the object model in the first image to obtain a rendered image. According to the method and the device, the rendered image is obtained through rendering when the mirror surface model exists in the three-dimensional scene, and the rendered image comprises the object, the mirror surface and the virtual image of the object imaged in the mirror surface.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of computer technology, and in particular to an image rendering method, apparatus, device, and readable storage medium. Background Art

[0002] With the development of computer technology, image rendering technology has also developed. Image rendering technology is a technology that projects an object model in a three-dimensional scene to obtain a two-dimensional image according to set environments, lights, materials, rendering parameters, etc. Among them, the object model is a three-dimensional model used to represent an object, and the object model includes geometric surfaces, vertices, texture information, etc.

[0003] In some cases, the object model in the three-dimensional scene includes a mirror model for representing a mirror. Based on the principle of light reflection, the light emitted or reflected by an object enters the eye after being reflected by the mirror, and the virtual image of the object can be seen by the eye along the reverse extension line of the light, making the mirror have the physical property of mirror imaging. Based on this, when the two-dimensional image to be rendered contains objects represented by at least one object model in the three-dimensional scene, and these objects in the two-dimensional image include the mirror represented by the mirror model, the mirror in the two-dimensional image needs to include the virtual image of the object imaged in the mirror. Based on this, when there is a mirror model in the three-dimensional scene, how to perform image rendering becomes an urgent technical problem to be solved. Summary of the Invention

[0004] The present application provides an image rendering method, apparatus, device, and readable storage medium, which can be used to render a rendered image when there is a mirror model in a three-dimensional scene, and the rendered image includes an object, a mirror, and a virtual image of the object imaged in the mirror. The technical solution includes the following content.

[0005] On the one hand, an image rendering method is provided, and the method includes:

[0006] Determine a mirror mask based on the camera parameters of the main camera, where the main camera is used to perform virtual imaging on the mirror model and the object model included in the virtual scene, and the mirror mask is used to block other regions in the blank image except the mirror region, and the mirror region is the image region where the mirror represented by the mirror model is located;

[0007] Obtain the camera parameters of the mirror camera, where the mirror camera is used to perform virtual imaging on the object model based on the mirror mask;

[0008] Render the object model in the blank image based on the camera parameters of the mirror camera and the mirror mask to obtain a first image, where the mirror region in the first image includes the mirror image of the object represented by the object model;

[0009] Based on the camera parameters of the main camera, render the mirror model and the object model in the first image to obtain a rendered image.

[0010] On the other hand, an image rendering device is provided, and the device includes:

[0011] A determination module, configured to determine a mirror mask based on the camera parameters of a main camera, where the main camera is used for virtual imaging of a mirror model and an object model included in a virtual scene, and the mirror mask is used to block other areas in a blank image except for a mirror area, and the mirror area is an image area where the mirror represented by the mirror model is located;

[0012] An acquisition module, configured to acquire the camera parameters of a mirror camera, where the mirror camera is used for virtual imaging of the object model based on the mirror mask;

[0013] A rendering module, configured to render the object model in the blank image based on the camera parameters of the mirror camera and the mirror mask to obtain a first image, where the mirror area in the first image includes a mirror image of the object represented by the object model;

[0014] The rendering module is further configured to render the mirror model and the object model in the first image based on the camera parameters of the main camera to obtain a rendered image.

[0015] In a possible implementation manner, the determination module is configured to determine a mirror mask based on the camera parameters of the main camera if the included angle between the mirror orientation of the mirror model and the pointing orientation of the main camera pointing to the mirror model is an obtuse angle or a flat angle, and the mirror model is located within the viewing frustum of the main camera.

[0016] In a possible implementation manner, the acquisition module is further configured to acquire the position data of the main camera, the position data of the mirror model, and a first vector representing the mirror orientation;

[0017] The determination module is further configured to determine a second vector representing the pointing orientation based on the position data of the main camera and the position data of the mirror model;

[0018] The determination module is further configured to determine that the included angle between the mirror orientation and the pointing orientation is an obtuse angle or a flat angle based on the first vector and the second vector.

[0019] In a possible implementation manner, the acquisition module is further configured to acquire the vertex position data of the mirror model;

[0020] The determining module is further configured to, for any clipping plane on the frustum of the main camera, determine a judgment result of the any clipping plane based on the vertex position data and the plane parameters of the any clipping plane, where the judgment result of the any clipping plane represents whether the vertex corresponding to the vertex position data is located on the side of the frustum of the main camera where the any clipping plane is located;

[0021] The determining module is further configured to determine that the mirror model is located within the frustum of the main camera based on the judgment results of the respective clipping planes.

[0022] In a possible implementation manner, the mirror mask includes an unoccluded area, and the unoccluded area corresponds to the mirror area;

[0023] The determining module is configured to determine the vertex position data of the unoccluded area based on the camera parameters of the main camera and the vertex position data of the mirror model; and determine the mirror mask based on the vertex position data of the unoccluded area.

[0024] In a possible implementation manner, the camera parameters of the mirror camera include the viewing angle parameters and the projection matrix of the mirror camera;

[0025] The obtaining module is configured to obtain the viewing angle parameters of the mirror camera, where the viewing angle parameters of the mirror camera represent at least one of the position or orientation of the mirror camera; and determine the projection matrix of the mirror camera based on the viewing angle parameters of the mirror camera.

[0026] In a possible implementation manner, the viewing angle parameters of the mirror camera include the position data and the orientation data of the mirror camera;

[0027] The obtaining module is configured to determine the position data of the mirror camera based on the position data of the main camera, the position data of the mirror model, and a first vector representing the mirror orientation; and determine the orientation data of the mirror camera based on the first vector and the orientation data of the main camera.

[0028] In a possible implementation, the obtaining module is configured to determine a normal vector representing the mirror orientation based on the viewing angle parameter of the mirror camera and a first vector representing the mirror orientation, where the first vector is a vector based on the world coordinate system, and the normal vector is a vector based on the camera coordinate system of the mirror camera; determine a distance parameter between the mirror model and the origin based on the viewing angle parameter of the mirror camera, the normal vector, and the position data of the mirror model, where the position data of the mirror model is data based on the world coordinate system, and the distance parameter is a parameter based on the camera coordinate system of the mirror camera; and determine a projection matrix of the mirror camera based on the normal vector and the distance parameter.

[0029] In a possible implementation, the rendering module is configured to obtain an inversion command for mirror-inverting the viewing space of the mirror camera; and render the object model in the blank image based on the inversion command, the camera parameters of the mirror camera, and the mirror mask to obtain a first image.

[0030] In a possible implementation, the rendered image includes pixel information of each first point on the surface of the object model facing the main camera;

[0031] The rendering module is configured to determine the depth of each second point on the mirror model based on the camera parameters of the main camera and the vertex position data of the mirror model, where the depth of the second point represents the distance between the second point and the main camera; if the line of sight of the main camera passes through any one of the first points and any one of the second points, and the depth of any one of the first points is not greater than the depth of any one of the second points, write the pixel information of any one of the first points in the first image, where the depth of any one of the first points represents the distance between any one of the first points and the main camera; and if the line of sight of the main camera passes through any one of the first points and does not pass through any one of the second points, write the pixel information of any one of the first points in the first image.

[0032] On the other hand, an electronic device is provided, where the electronic device includes a processor and a memory, and at least one computer program is stored in the memory and is loaded and executed by the processor to enable the electronic device to implement the image rendering method described in any one of the above.

[0033] On the other hand, a computer-readable storage medium is further provided, where at least one computer program is stored in the computer-readable storage medium and is loaded and executed by a processor to enable an electronic device to implement the image rendering method described in any one of the above.

[0034] On the other hand, a computer program is also provided. There is at least one computer program, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above image rendering methods.

[0035] On the other hand, a computer program product is also provided. At least one computer program is stored in the computer program product, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above image rendering methods.

[0036] The technical solution provided by this application brings at least the following beneficial effects:

[0037] In the technical solution provided by this application, first, a specular mask is determined based on the camera parameters of the main camera, and then based on the camera parameters of the specular camera and the specular mask, an object model is rendered in a blank image to obtain a first image, realizing the rendering of an object in the specular area of the blank image, so that the first image includes a specular surface and a mirror image of the object imaged in the specular surface. In addition, the specular camera only renders within the specular area, reducing the rendering area and the occupation of rendering resources. Then, based on the camera parameters of the main camera, the specular model and the object model are rendered in the first image to obtain a rendered image, realizing the rendering of an object in the first image, so that the rendered image includes a specular surface, an object, and a virtual image of the object imaged in the specular surface, improving the accuracy of the rendered image. In addition, the main camera renders in the first image instead of in an additional blank image, which is beneficial to reducing the storage resources occupied by the image. By reducing the occupation of rendering resources in the rendering stage of the specular camera and reducing the occupation of storage resources in the rendering stage of the main camera, an electronic device with a lower configuration can also have better rendering performance, improving the applicability. Description of the Drawings

[0038] To more clearly illustrate the technical solutions in the embodiments of this application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0039] Figure 1 It is a schematic diagram of the implementation environment of an image rendering method provided by an embodiment of this application;

[0040] Figure 2 It is a flowchart of an image rendering method provided by an embodiment of this application;

[0041] Figure 3 It is a schematic diagram of a virtual scene provided by an embodiment of this application;

[0042] Figure 4 It is a schematic flowchart of an image rendering provided by an embodiment of the present application;

[0043] Figure 5 It is a schematic diagram of an image rendering resource package provided by an embodiment of the present application;

[0044] Figure 6 It is a schematic diagram of a display interface provided by an embodiment of the present application;

[0045] Figure 7 It is a schematic diagram of a rendered image provided by an embodiment of the present application;

[0046] Figure 8 It is a schematic diagram of the setting of a mirror model provided by an embodiment of the present application;

[0047] Figure 9 It is another schematic diagram of a rendered image provided by an embodiment of the present application;

[0048] Figure 10 It is another schematic diagram of a display interface provided by an embodiment of the present application;

[0049] Figure 11 It is a schematic structural diagram of an image rendering device provided by an embodiment of the present application;

[0050] Figure 12 It is a schematic structural diagram of a terminal device provided by an embodiment of the present application;

[0051] Figure 13 It is a schematic structural diagram of a server provided by an embodiment of the present application. Detailed implementation manners

[0052] To make the objectives, technical solutions and advantages of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the accompanying drawings.

[0053] In the field of computer technology, an object model is often used to describe information such as the texture, shape, size, and structure of the corresponding object. At least one object model can be used to construct a virtual scene, and a virtual camera can be used to perform virtual imaging on the virtual scene to obtain a rendered image.

[0054] In some cases, the object model in the virtual scene includes a mirror model for characterizing a mirror. Since a mirror has the physical property of mirror imaging, therefore, the rendered image needs to include not only the objects in the virtual scene, and when the object in the rendered image includes a mirror, the mirror also needs to include the mirror image of the object imaged in the mirror. Based on this, when there is a mirror model in the virtual scene, how to perform image rendering becomes a technical problem that urgently needs to be solved.

[0055] An embodiment of the present application provides an image rendering method, which can render a rendered image, and the rendered image includes an object, a mirror, and a mirror image of the object imaged in the mirror, improving the accuracy of the rendered image.

[0056] To facilitate the description of various alternative embodiments of the present application, the technical terms involved will be explained and described below.

[0057] Virtual scene: A virtual scene is a scene fictionalized using at least one object model. The object model is used to describe information such as the texture, shape, size, and structure of an object. For example, the object model can be a building model, a vehicle model, an electrical appliance model, a biological model, etc. In an embodiment of the present application, the object model includes a mirror model and an object model other than the mirror model. The mirror model is used to describe information such as the texture, shape, size, and structure of the mirror, and the object model is used to describe information such as the texture, shape, size, and structure of an object other than the mirror (such as a building, a vehicle, an electrical appliance, a biological, etc.). Optionally, the virtual scene further includes a light source. The embodiment of the present application does not limit the type of the light source. For example, the light source is a parallel light or a lamp, etc.

[0058] Main Camera: The main camera is a virtual camera used for virtual imaging of a virtual scene. In an embodiment of the present application, the virtual scene includes a mirror model and an object model. The main camera performs virtual imaging on at least one of the mirror model and the object model according to the position, orientation, etc. of the main camera. The main camera corresponds to the camera coordinate system of the main camera. In the following description, the camera coordinate system of the main camera can be abbreviated as the main camera coordinate system. In an exemplary implementation, the position of the main camera can be used as the origin of the main camera coordinate system, and the space with the position of the main camera as the coordinate origin is called the observation space of the main camera. The main camera corresponds to a viewing frustum. In the following description, the viewing frustum corresponding to the main camera can be abbreviated as the main camera viewing frustum. The main camera can perform virtual imaging on all or part of the virtual scene located within the main camera viewing frustum.

[0059] Mirror Mask: A mirror mask is a layout that includes an occluded area and an unoccluded area. Optionally, the mirror mask has the same size as the blank image. When the mirror mask covers the blank image, the image area covered by the occluded area in the blank image cannot be written, while the image area covered by the unoccluded area in the blank image can be written. Here, the writing includes at least one of depth value writing, texture value writing, color value writing, etc. In the embodiments of the present application, the mirror model represents the mirror, and the object model represents the object. After virtual imaging of the mirror model and the object model by the main camera, the obtained image includes the image area where the mirror is located and the image area where the object is located. In the following description, the image area where the mirror is located is referred to as the mirror area, and the image area where the object is located is referred to as the object area. The occluded area is used to occlude the object area in the blank image. Optionally, if the image also includes a remaining area other than the mirror area and the object area, the occluded area is also used to occlude the remaining area in the blank image to prevent these areas from being written during the virtual imaging process. By occluding the object area and the remaining area in the blank image with the occluded area, the blank image retains the mirror area corresponding to the unoccluded area, enabling the mirror area to be written during the virtual imaging process, thereby achieving rendering of the mirror image of the object imaged in the mirror.

[0060] Rendering Optimization: There are many performance bottlenecks in image rendering. Rendering optimization refers to finding the performance bottlenecks in image rendering and making targeted optimizations to improve the rendering efficiency. Common rendering optimization methods include model optimization, culling, multithreading, caching, etc.

[0061] Graphics Processing Unit (GPU): A dedicated chip used for graphics image processing in modern personal computers, servers, mobile devices, game consoles, etc.

[0062] Rendering Pipeline: The graphics rendering process running in the GPU, including rendering components such as vertex shaders, rasterization, and pixel shaders. Code can be written in the shaders to flexibly control the GPU to perform rendering based on the rendering components.

[0063] Depth Test: A step in the GPU rendering pipeline that can test the depth of a pixel point. If the test fails, the pixel point is not rendered; if the test passes, the pixel point is rendered.

[0064] Pixel Shader: A necessary step in the GPU rendering pipeline. The program performs coloring calculations on the rasterized pixels according to the code and outputs them to the frame buffer after passing the test, completing one rendering pipeline process.

[0065] Coordinate transformation: The process of transforming coordinates in one spatial coordinate system to another. The transformation relationship between coordinate systems is usually represented by a matrix. Multiplying the matrix on the left by the coordinate vector, the resulting value is the transformed coordinate.

[0066] Frame Buffer: A section of memory in the GPU used to store image space data, which can be specified as the drawing target by the rendering pipeline and data can be written into it through the rendering pipeline.

[0067] Render Pass: Includes a series of settings of rendering states, input of geometric data, execution of vertex and fragment shaders, and writing of rendering outputs. In complex rendering scenarios, multiple Render Passes are usually required to complete the entire rendering process.

[0068] Figure 1 It is a schematic diagram of the implementation environment of an image rendering method provided by an embodiment of the present application. As Figure 1 shown, this implementation environment includes a terminal device 101 and a server 102. Among them, the image rendering method in the embodiment of the present application can be executed by the terminal device 101, or by the server 102, or jointly by the terminal device 101 and the server 102.

[0069] The terminal device 101 can be a smart phone, game console, desktop computer, tablet computer, laptop portable computer, smart TV, smart vehicle-mounted device, smart voice interaction device, smart home appliance, etc. The server 102 can be a single server, or a server cluster composed of multiple servers, or any one of a cloud computing platform and a virtualization center. The embodiment of the present application does not limit this. The server 102 can be connected to the terminal device 101 through a communication network, and this communication network is a wired network or a wireless network. The server 102 can have functions such as data processing, data storage, and data sending and receiving, which are not limited in the embodiment of the present application. The number of the terminal device 101 and the server 102 is not limited and can be one or more.

[0070] As Figure 2 shown, Figure 2 An embodiment of the present application provides an image rendering method. This method can be applied to the above implementation environment, and can render a rendered image when there is a mirror model in a three-dimensional scene, and the rendered image includes an object, a mirror, and an image of the object imaged in the mirror. For ease of description, the terminal device 101 or the server 102 that executes the Figure 2 shown method is called an electronic device. That is to say, the method in the embodiment of the present application can be executed by the electronic device. As Figure 2 shown, this method includes the following steps.

[0071] Step 201: Determine a mirror mask based on the camera parameters of the main camera. The main camera is used to perform virtual imaging on a mirror model and an object model included in a virtual scene. The mirror mask is used to occlude other regions in a blank image except for the mirror region, and the mirror region is the image region where the mirror represented by the mirror model is located.

[0072] In the embodiment of the present application, if the mirror model is visible in the main camera's view, it indicates that the mirror region will be included in the image obtained after rendering based on the main camera. Since the mirror has the physical property of mirror imaging, there may or may not be an image of an object in the mirror region. Based on this, the electronic device needs to render the mirror region. By determining the mirror mask based on the camera parameters of the main camera and using the mirror mask to frame the mirror region, it is possible to avoid rendering beyond the mirror region and ensure the accuracy of the rendering result.

[0073] The following first introduces possible implementation manners of whether the mirror model is visible in the main camera's view, as shown in the following implementation manners A1 to A3.

[0074] In a possible implementation manner A1, since the main camera can image an object model located within the viewing frustum of the main camera, the electronic device needs to determine whether the mirror model is located within the viewing frustum of the main camera. If the mirror model is not located within the viewing frustum of the main camera, the mirror model is not visible in the main camera's view. If the mirror model is located within the viewing frustum of the main camera, the mirror model is visible in the main camera's view.

[0075] In a possible implementation manner A2, the mirror model represents a mirror. Based on the principle of light reflection, it can be known that the light emitted or reflected by an object enters the eye after being reflected by the mirror, and the eye can see the image of the object along the reverse extension line of the light, enabling the mirror to have the physical property of mirror imaging. That is to say, the mirror is a surface that can perform mirror imaging. Generally, the mirror model includes a front side and a back side. The front side can perform mirror imaging, and the back side cannot perform mirror imaging. Optionally, the electronic device needs to determine whether the front side of the mirror model faces the main camera. If the front side of the mirror model faces the main camera, it indicates that the mirror model is visible in the main camera's view. If the front side of the mirror model does not face the main camera, it indicates that the mirror model is not visible in the main camera's view.

[0076] In a possible implementation manner A3, Step 201 includes: If the angle between the mirror orientation of the mirror model and the orientation of the main camera pointing to the mirror model is an obtuse angle or a straight angle, and the mirror model is located within the viewing frustum of the main camera, determine the mirror mask based on the camera parameters of the main camera.

[0077] In the embodiments of the present application, the mirror surface orientation of the mirror surface model is perpendicular to the surface of the mirror surface represented by the mirror surface model. That is to say, the mirror surface orientation of the mirror surface model is the normal direction of the mirror surface. The pointing orientation of the main camera pointing to the mirror surface model is a line-of-sight direction of the main camera. If the angle between the mirror surface orientation and the pointing orientation is 0 or an acute angle, it indicates that the front of the mirror surface model is not facing the main camera. If the angle between the mirror surface orientation and the pointing orientation is an obtuse angle or a flat angle, it indicates that the front of the mirror surface model is facing the main camera.

[0078] The electronic device can obtain a first judgment result, and use the first judgment result to represent that the angle between the mirror surface orientation and the pointing orientation is 0 or an acute angle or an obtuse angle or a flat angle, that is, use the first judgment result to represent whether the angle between the mirror surface orientation of the mirror surface model and the pointing orientation of the main camera pointing to the mirror surface model is an obtuse angle or a flat angle, so as to use the first judgment result to represent whether the front of the mirror surface model is facing the main camera. The embodiments of the present application do not limit the manner of obtaining the first judgment result.

[0079] In an exemplary embodiment, before step 201, it further includes: obtaining the position data of the main camera, the position data of the mirror surface model, and a first vector representing the mirror surface orientation; determining a second vector representing the pointing orientation based on the position data of the main camera and the position data of the mirror surface model; determining that the angle between the mirror surface orientation and the pointing orientation is an obtuse angle or a flat angle based on the first vector and the second vector.

[0080] In the embodiments of the present application, the position data of the main camera is used to represent the position of the main camera in the world coordinate system, the position data of the mirror surface model is used to represent the position of the mirror surface model in the world coordinate system, and the world coordinate system is the real coordinates of the object model in the virtual scene in three-dimensional space.

[0081] Subtract the position data of the main camera from the position data of the mirror surface model to obtain a second vector representing the pointing orientation of the main camera pointing to the mirror surface model. Calculate the dot product between the first vector and the second vector to obtain the first judgment result. Optionally, if the position data of the main camera is D′, the position data of the mirror surface model is D, and the first vector representing the mirror surface orientation is F, then the first judgment result vis(D, D′, F) satisfies: vis(D, D′, F) = F · (D - D′).

[0082] It should be noted that the above determination method of the first judgment result is only illustrative, and in actual applications, there may be other determination methods. For example, the electronic device can calculate the cosine value between the first vector and the second vector, and use the cosine value as the first judgment result.

[0083] When the first judgment result is the dot product or cosine value between the first vector and the second vector, if the first judgment result is equal to zero, the first judgment result indicates that the mirror surface orientation is perpendicular to the pointing orientation, that is, the included angle between the mirror surface orientation and the pointing orientation is 90 degrees. If the first judgment result is less than zero, the first judgment result indicates that the included angle between the mirror surface orientation and the pointing orientation is an obtuse angle or a flat angle. If the first judgment result is greater than zero, the first judgment result indicates that the included angle between the mirror surface orientation and the pointing orientation is 0 or an acute angle.

[0084] In the embodiments of the present application, if the mirror model is located within the viewing frustum of the main camera, it means that the main camera will perform virtual imaging on the mirror model. In this case, it is necessary for the mirror camera to perform virtual imaging on the object model in the mirror area. If the mirror model is not located within the viewing frustum of the main camera, it means that the main camera will not perform virtual imaging on the mirror model, and naturally there is no need for the mirror camera to perform virtual imaging on the object model in the mirror area.

[0085] The electronic device can obtain the second judgment result, and the second judgment result indicates whether the mirror model is located within the viewing frustum of the main camera. The embodiments of the present application do not limit the manner of obtaining the second judgment result.

[0086] In an exemplary embodiment, before step 201, it further includes: obtaining the vertex position data of the mirror model; for any clipping plane on the viewing frustum of the main camera, based on the vertex position data and the plane parameters of any clipping plane, determining the judgment result of any clipping plane, and the judgment result of any clipping plane indicates whether the vertex corresponding to the vertex position data is located on the side of the viewing frustum of the main camera where any clipping plane is located; based on the judgment results of each clipping plane, determining that the mirror model is located within the viewing frustum of the main camera.

[0087] In the embodiments of the present application, the mirror model includes multiple vertices, and each vertex corresponds to a position data. Among them, the position data corresponding to the vertex is used to represent the position of the vertex in the world coordinate system, and the position data corresponding to the vertex can be simply referred to as vertex position data. Optionally, since the mirror surface represented by the mirror model is a plane, the vertices of the mirror model can be located on the mirror edge or within the mirror surface.

[0088] For example, if the mirror model represents a mirror in the shape of a quadrilateral, the mirror model includes four vertices of the quadrilateral, and these four vertices are sequentially denoted as vertex v(0), v(1), v(2), v(3). The vertex position data of a vertex is the three-dimensional coordinate of the vertex, and the three-dimensional coordinate includes the x-axis coordinate, the y-axis coordinate, and the z-axis coordinate. Exemplarily, in the form of {x, y, z}, the vertex position data of the four vertices can be expressed as:

[0089]

[0090] As can be seen from the vertex position data of the above four vertices, the above four vertices can form a square with an area of 1. Among them, the z-axis coordinate is 0.01, that is, the z-axis is offset by 0.01, so that the mirror surface can be attached to the plane surface of any model without z-axis conflict (Z-Frighting) with the vertices on the attachment part. Among them, the z-axis conflict belongs to the depth conflict.

[0091] That is to say, when the mirror surface model represents a plane and the mirror surface model needs to be attached to the plane surface of the object model, the target axis coordinates of the vertices in the plane surface of the object model can be offset by a certain offset amount to obtain the target axis coordinates of the vertices in the mirror surface model. Based on the target axis coordinates of the vertices in the mirror surface model, the vertex position data of the vertices in the mirror surface model can be determined. The target axis coordinate corresponds to the coordinate axis perpendicular to the plane surface of the object model. In this way, the depth conflict between the vertices on the plane surface of the object model and the vertices in the mirror surface model can be avoided.

[0092] Generally, the viewing frustum of the main camera is a three-dimensional space in the shape of a frustum of a pyramid, and each face of the frustum is a clipping plane of the viewing frustum. For any clipping plane, the electronic device can obtain the plane parameters of the clipping plane. The embodiments of the present application do not limit the plane parameters of the clipping plane. Exemplarily, the plane parameters of the clipping plane include at least one of the normal vector of the clipping plane, the distance between the clipping plane and the origin, the perimeter of the clipping plane, the area of the clipping plane, the plane equation of the clipping plane, etc.

[0093] Optionally, the electronic device can call the acquisition function of the clipping plane. For example, some development engines come with the GeometryUtility.CalculateFrustumPlanes(Camera) function, which is used to obtain the clipping plane and is an acquisition function of the clipping plane. The electronic device calculates the plane parameters of each clipping plane on the viewing frustum of the main camera based on the projection matrix of the main camera by calling the acquisition function of the clipping plane.

[0094] The judgment result of the clipping plane can be determined based on the vertex position data and the plane parameters of any clipping plane, and whether the vertex is located on the side of the viewing frustum of the main camera of the clipping plane can be characterized by the judgment result of the clipping plane. It can be understood that the determination method of the judgment result of the clipping plane is different when the plane parameters of the clipping plane are different. Hereinafter, taking the plane parameters of the clipping plane including the normal vector of the clipping plane and the distance between the clipping plane and the origin as an example for elaboration, and other implementation manners will not be elaborated here.

[0095] The dot product result can be obtained by dot multiplying the normal vector of the clipping plane with the vertex position data, and the addition result can be obtained by adding the dot product result to the distance between the clipping plane and the origin. If the addition result is positive, the determination result of the clipping plane indicates that the vertex is on the side where the viewing frustum of the main camera of the clipping plane is located. If the addition result is negative, the determination result of the clipping plane indicates that the vertex is on the side where the viewing frustum of the main camera of the clipping plane is not located. If the addition result is zero, the determination result of the clipping plane indicates that the vertex is on the clipping plane.

[0096] It can be understood that the vertices of the mirror model include at least one, and the determination result of any clipping plane includes the determination results between the clipping plane and each vertex. The determination result between the clipping plane and any vertex indicates whether the vertex is on the side where the viewing frustum of the main camera of the clipping plane is located.

[0097] For any clipping plane, if the determination results between the clipping plane and each vertex all indicate that the vertex is on the side where the viewing frustum of the main camera of the clipping plane is not located, it is determined that the determination result of the clipping plane indicates that all vertices of the mirror model are on the side where the viewing frustum of the main camera of the clipping plane is not located, which is equivalent to the mirror model being on the side where the viewing frustum of the main camera of the clipping plane is not located. If there is a determination result between the clipping plane and the vertex that indicates that the vertex is on the side where the viewing frustum of the main camera of the clipping plane is located, it is determined that the determination result of the clipping plane indicates that there are vertices of the mirror model on the side where the viewing frustum of the main camera of the clipping plane is not located, which is equivalent to all or part of the mirror model being on the side where the viewing frustum of the main camera of the clipping plane is not located.

[0098] After determining the determination results of each clipping plane, if it is determined based on the determination results of each clipping plane that there is no mirror model located on the side where the viewing frustum of the main camera of the clipping plane is not located, it is determined that the second determination result indicates that the mirror model is located within the viewing frustum of the main camera. If it is determined based on the determination results of each clipping plane that there is a mirror model located on the side where the viewing frustum of the main camera of the clipping plane is not located, it is determined that the second determination result indicates that the mirror model is not located within the viewing frustum of the main camera.

[0099] Optionally, the viewing frustum of the main camera includes 6 clipping planes, and the mirror model includes 4 vertices. Then the second determination result can be determined according to the following code.

[0100]

[0101] Optionally, the vertex position data of the mirror model mentioned above refers to the position data of the vertices in the world coordinate system. If the electronic device obtains the position data of the vertices in the local space, it can convert the position data of the vertices in the local space into vertex position data based on the conversion relationship between the local space and the world space. For example, if the conversion relationship between the local space and the world space is the conversion matrix M, and the electronic device obtains the position data of four vertices in the local space, then M is multiplied by the position data of the j-th vertex in the local space to obtain the vertex position data corners[j] of this vertex.

[0102] It can be understood that in the embodiments of the present application, the vertices of the mirror model and the clipping planes of the viewing frustum are used to determine whether the mirror model is located within the viewing frustum. Compared with general methods, for example, first determining the bounding box of the model in the world space and then performing viewing frustum culling on the bounding box to determine whether the model is located within the viewing frustum, the solution of the embodiments of the present application has a smaller computational amount, and the vertices used fully conform to the mirror model itself, which can improve the computational performance. Optionally, for the vertices located outside the viewing frustum, they can be directly culled, and the culling efficiency is higher.

[0103] In the embodiments of the present application, if the angle between the mirror orientation and the pointing orientation is an obtuse angle or a flat angle, and the mirror model is located within the viewing frustum of the main camera, it indicates that the mirror model is facing the main camera and the main camera needs to perform virtual imaging on the mirror model. In this case, the mirror model is visible in the view of the main camera, and a mirror camera needs to be used for rendering.

[0104] If the angle between the mirror orientation and the pointing orientation is 0 or an acute angle, and / or the mirror model is not located within the viewing frustum of the main camera, it indicates that the mirror model is not facing the main camera, and / or the main camera does not need to perform virtual imaging on the mirror model. In this case, the mirror model is not visible in the view of the main camera. If the mirror model is not visible in the view of the main camera, it means that the main camera does not need to perform virtual imaging on the mirror model, and naturally there is no need to perform rendering through the mirror camera. That is to say, the implementation principle of step 204 can be directly followed, and based on the camera parameters of the main camera, the object model is rendered in a blank image to obtain a rendered image, and the rendering method will not be elaborated here.

[0105] As mentioned above, when using the mirror camera for rendering, it is necessary to ensure that the rendering area of the mirror camera does not exceed the mirror area in the rendered image. Based on this, a mirror mask needs to be set to frame the mirror area through the mirror mask, so that the mirror camera will not exceed the mirror area during rendering. Generally, through the rendering of the main camera, the mirror area in the image is obtained. Therefore, it is necessary to determine the mirror mask based on the camera parameters of the main camera.

[0106] In an exemplary embodiment, the mirror mask includes an unoccluded region corresponding to the mirror region. "Determining the mirror mask based on the camera parameters of the main camera" in step 201 includes: determining the vertex position data of the unoccluded region based on the camera parameters of the main camera and the vertex position data of the mirror model; and determining the mirror mask based on the vertex position data of the unoccluded region.

[0107] In the embodiment of the present application, the electronic device can obtain pre-rendered shader resources and determine the mirror mask based on the pre-rendered shader resources. The pre-rendered shader resources are resources of a shader, which is an editable program that can replace the fixed rendering pipeline. By setting the shader, it is possible to determine the mirror mask based on the camera parameters of the main camera and perform mirror rendering based on the mirror mask.

[0108] Optionally, the shader corresponding to the pre-rendered shader resources includes a vertex shader, which is used to transform the vertex position data of the mirror model based on the camera parameters of the main camera. The transformation includes: first transforming from the world coordinate system to the main camera coordinate system, and then transforming from the main camera coordinate system to the screen coordinate system to obtain the vertex position data of the unoccluded region. The vertex position data of the mirror model corresponds to the world coordinate system, and the vertex position data of the unoccluded region corresponds to the screen coordinate system. It can be understood that the two-dimensional image corresponds to the screen coordinate system, that is, the blank image, the first image, the rendered image, etc. all correspond to the screen coordinate system, and the unoccluded region is used to frame the mirror region in the blank image.

[0109] Optionally, the camera parameters of the main camera include the projection matrix of the main camera, which is used to convert the position data within the viewing frustum of the main camera into position data on the screen coordinate system. Based on this, the vertex shader can use the projection matrix of the main camera to transform the vertex position data of the mirror model into the vertex position data of the unoccluded region, and the transformation process will not be elaborated here.

[0110] After determining the vertex position data of the unoccluded region, the unoccluded region can be framed in the layout. The other regions in the layout except the unoccluded region are occluded regions, and the mirror mask is obtained based on the occluded regions and the unoccluded region.

[0111] Optionally, the electronic device may obtain a plurality of initial masks. By combining different initial masks, different combined masks can be obtained. For example, the electronic device may obtain n initial masks, and each initial mask can be used or not used. Based on this, the n initial masks can be combined to obtain 2 to the nth power minus one combined mask, a total of 2 to the nth power of combined masks. In the embodiments of the present application, the mirror mask is a combined mask. After determining the vertex position data of the unoccluded area, the usage of each initial mask can be determined, and the usage of each initial mask indicates whether to use the initial mask. The electronic device combines the used initial masks to obtain the mirror mask.

[0112] The rendering pipeline can be set in the shader corresponding to the pre-rendered shader resource to implement determining the mirror mask and other information related to the mirror mask based on the rendering pipeline. Exemplarily, the setting of the rendering pipeline is shown in the following code.

[0113]

[0114] From the above code, it can be seen that for the rendering pipeline, mask testing can be set to be enabled, and whether to use the initial mask is represented by the reference value _ReflectStencil. Optionally, the i-th reference value is the first numerical value (for example, 0), indicating that the i-th initial mask is not used, and the i-th reference value is the second numerical value (for example, 1), indicating that the i-th initial mask is used. Enable mask testing. If the mask test passes, specify the placement position of the mirror mask, write the mirror mask at the placement position, and the mirror mask is represented by n reference values _ReflectStencil. In addition, enable depth testing and disable depth writing and color writing.

[0115] Step 202: Obtain the camera parameters of the mirror camera, where the mirror camera is used to perform virtual imaging on the object model based on the mirror mask.

[0116] In the embodiments of the present application, the mirror camera is a virtual image that is mirror-symmetric to the main camera based on the mirror model, as Figure 3 shown, the main camera and the mirror camera are symmetric based on the mirror model. If the main camera or the mirror model in the world space changes, the virtual image of the main camera will also change accordingly, that is, the mirror camera needs to be updated in real time.

[0117] Optionally, the electronic device may obtain the camera parameters of the mirror camera input by the user. Alternatively, the electronic device may determine the camera parameters of the mirror camera based on the camera parameters of the main camera and the relevant parameters of the mirror model.

[0118] In an exemplary embodiment, the camera parameters of the mirror camera include the viewing angle parameters and projection matrix of the mirror camera; step 202 includes steps 2021 to 2022 (not shown in the figure).

[0119] Step 2021, obtain the viewing angle parameters of the mirror camera, where the viewing angle parameters of the mirror camera characterize at least one of the position or orientation of the mirror camera.

[0120] In the embodiments of the present application, the viewing angle parameters of the mirror camera can characterize at least one of the position of the mirror camera in the world space and the orientation of the mirror camera in the world space. The manner of obtaining the viewing angle parameters of the mirror camera is not limited herein. Exemplarily, the electronic device can obtain the viewing angle parameters of the mirror camera input by the user, or the electronic device can extract the viewing angle parameters of the mirror camera from a configuration file, etc.

[0121] Optionally, the viewing angle parameters of the mirror camera include the position data of the mirror camera and the orientation data of the mirror camera. Step 2021 includes: determining the position data of the mirror camera based on the position data of the main camera, the position data of the mirror model, and the first vector characterizing the mirror orientation; determining the orientation data of the mirror camera based on the first vector and the orientation data of the main camera.

[0122] In the embodiments of the present application, the position data of the mirror camera is the three-dimensional coordinates of the mirror camera in the world coordinate system, which is used to characterize the position of the mirror camera in the world space. The orientation data of the mirror camera includes at least one of the front vector, right vector, and up vector of the mirror camera in the world coordinate. Among them, the intersection point of each line of sight of the mirror camera is the optical center of the mirror camera. The vector extending forward from the optical center of the mirror camera is the front vector of the mirror camera, which is used to characterize the front of the mirror camera. The vector extending upward from the optical center of the mirror camera is the up vector of the mirror camera, which is used to characterize the upper part of the mirror camera. The vector extending to the right from the optical center of the mirror camera is the front vector of the mirror camera, which is used to characterize the right side of the mirror camera. It can be understood that the front vector, right vector, and up vector of the mirror camera are perpendicular to each other, and the orientation of the mirror camera can be reflected by at least one of the three.

[0123] On the one hand, the electronic device can obtain the position data of the main camera. The position data of the main camera is the three-dimensional coordinates of the main camera in the world coordinate system, which is used to characterize the position of the main camera in the world space. The electronic device can also obtain the position data of the mirror model. The position data of the mirror model is the three-dimensional coordinates of the mirror model in the world coordinate system, which is used to characterize the position of the mirror model in the world space. In addition, the electronic device can also obtain the first vector, where the first vector is used to characterize the mirror orientation of the mirror model, and the mirror orientation is perpendicular to the mirror model and faces the side where the object to be imaged is located.

[0124] The electronic device can calculate the position data of the mirror camera based on the reflection algorithm in the three-dimensional space, the position data of the main camera, the position data of the mirror model, and the first vector. The position data of the mirror camera are the three-dimensional coordinates of the mirror camera in the world coordinate system, which are used to represent the position of the mirror camera in the world space.

[0125] Optionally, the position data of the main camera is D′, the position data of the mirror model is D, and the first vector of the mirror model is F. Then the position data D″ of the mirror camera satisfies: D″ = Reflect(D′ - D, -F) + D. Here, Reflect represents the reflection calculation function in the three-dimensional space. Assuming the two variables of the function are Dir and Normal respectively, the function is defined as: Reflect(Dir, Normal) = Dir - 2×(Normal·Dir)×Normal.

[0126] On the other hand, the electronic device can obtain the orientation data of the main camera. The orientation data of the main camera includes at least one of the front vector, the right vector, and the up vector of the main camera in the world coordinate. Among them, the intersection point of each line of sight of the main camera is the optical center of the main camera. The vector extending forward from the optical center of the main camera is the front vector of the main camera, which is used to represent the front of the main camera. The vector extending upward from the optical center of the main camera is the up vector of the main camera, which is used to represent the upper part of the main camera. The vector extending to the right from the optical center of the main camera is the front vector of the main camera, which is used to represent the right side of the main camera. It can be understood that the front vector, the right vector, and the up vector of the main camera are perpendicular to each other, and the orientation of the main camera can be reflected by at least one of the three.

[0127] The front vector of the mirror camera can be determined based on the first vector and the front vector of the main camera. Similarly, the right vector of the mirror camera can be determined based on the first vector and the right vector of the main camera. The up vector of the mirror camera can be determined based on the first vector and the up vector of the main camera.

[0128] Optionally, the first vector of the mirror model is F, the front vector of the main camera is Forward. Then the front vector Forward′ of the mirror camera satisfies: Forward′ = Reflect(Forward, -F). The up vector of the main camera is Up. Then the up vector Up′ of the mirror camera satisfies: Up′ = Reflect(Up, -F). Reflect is the reflection calculation function defined above.

[0129] In an exemplary embodiment, the development engine may provide a LookUp function, and the LookUp function corresponds to two variables. The value of one of the variables can be set to the forward vector of the mirror camera plus the position data of the mirror camera, that is, the value of one of the variables is Forward′ + D″. The value of the other variable can be set to the up vector of the mirror camera, that is, the value of the other variable is Up′. By setting the LookUp function and the values of the two variables, the position data of the mirror camera, the forward vector of the mirror camera, and the up vector of the mirror camera are set, and the space observed by the mirror camera in the world space is obtained, that is, the observation space of the mirror camera is obtained.

[0130] Step 2022, determine the projection matrix of the mirror camera based on the viewing parameters of the mirror camera.

[0131] In the embodiment of the present application, the projection matrix is a matrix used to convert points in the world coordinate system into points in the screen coordinate system. It can be understood that since the camera performs virtual imaging on the object model within the viewing frustum, the projection matrix is used to project points within the viewing frustum onto the image. Based on this, the electronic device can determine the projection matrix of the mirror camera based on the viewing parameters of the mirror camera, and project points within the viewing frustum of the mirror camera onto the image through the projection matrix of the mirror camera.

[0132] It can be understood that the viewing frustum of the main camera includes a near clipping plane and a far clipping plane, and both the near clipping plane and the far clipping plane are perpendicular to the forward vector of the main camera. Since the mirror camera is a mirror image of the main camera, the viewing frustum of the mirror camera and the viewing frustum of the main camera are mirror images. The viewing frustum of the mirror camera also includes a near clipping plane and a far clipping plane, and both the near clipping plane and the far clipping plane are perpendicular to the forward vector of the mirror camera.

[0133] Optionally, since the mirror camera is a mirror image of the main camera, and the viewing frustum of the mirror camera and the viewing frustum of the main camera are mirror images, and the projection matrix is used to project points within the viewing frustum onto the image, the mirror camera can have the same projection matrix as the main camera. That is to say, the electronic device can obtain the projection matrix of the main camera, and the projection matrix of the main camera is used to project points within the viewing frustum of the main camera onto the image, and the projection matrix of the main camera can be used as the projection matrix of the mirror camera.

[0134] In an exemplary embodiment, object models may be included on both sides of the mirror model, such as Figure 3In it, the left side of the mirror model includes a seat model 301, and the right side of the mirror model includes a tree model 302. Generally, when the main camera performs virtual imaging on the mirror model, the mirror model faces the main camera, and the mirror model images the object model on the side where the main camera is located, and will not image the object model on the side where the main camera is not located. That is to say, the object represented by the object model on the side where the main camera is located can be included in the mirror area in the image.

[0135] When the mirror camera performs virtual imaging, if there is an object model between the mirror camera and the mirror model, resulting in the mirror model performing virtual imaging on the object model, while in fact the object model between the mirror camera and the mirror model should not be imaged. For example Figure 3 In it, the tree model 302 should not be imaged by the mirror camera. Based on this, for the mirror frustum of the main camera's frustum, the near clipping plane of the mirror frustum can be set to the mirror model, and the far clipping plane of the mirror frustum remains unchanged to obtain the frustum of the mirror camera.

[0136] Since the projection matrix of the mirror camera is used to project the points within the frustum of the mirror camera onto the image, the electronic device can determine the projection matrix of the mirror camera based on the relevant parameters of the mirror model and the viewing angle parameters of the mirror camera.

[0137] Optionally, step 2022 includes: determining a normal vector representing the mirror orientation based on the viewing angle parameters of the mirror camera and a first vector representing the mirror orientation, the first vector is a vector based on the world coordinate system, and the normal vector is a vector based on the camera coordinate system of the mirror camera; determining the distance parameter between the mirror model and the origin based on the viewing angle parameters of the mirror camera, the normal vector, and the position data of the mirror model, the position data of the mirror model is data based on the world coordinate system, and the distance parameter is a parameter based on the camera coordinate system of the mirror camera; determining the projection matrix of the mirror camera based on the normal vector and the distance parameter.

[0138] In the embodiments of the present application, the relevant parameters of the mirror model include the first vector of the mirror model, and the first vector of the mirror model represents the orientation of the mirror model in the world space. The first vector of the mirror model can be converted into the normal vector of the mirror model based on the viewing angle parameters of the mirror camera, and the normal vector of the mirror model is used to represent the orientation of the mirror model in the viewing space of the mirror camera.

[0139] Optionally, the viewing angle parameters of the mirror camera are a matrix determined based on the position data and orientation data of the mirror camera. Multiply the inverse matrix of the matrix corresponding to the viewing angle parameters of the mirror camera by the first vector of the mirror model, and take the negative of the result as the normal vector of the mirror model. That is to say, if the viewing angle parameters of the mirror camera are V and the first vector of the mirror model is F, then the normal vector N of the mirror model vSatisfy: N v = -V - ×F.

[0140] Similarly, the relevant parameters of the mirror model include the position data of the mirror model. The position data of the mirror model represents the position of the mirror model in the world space. Based on the viewing parameters of the mirror camera and the normal vector of the mirror model, the position data of the mirror model can be converted into a distance parameter, and the distance parameter represents the position of the mirror model in the viewing space of the mirror camera, that is, the distance parameter represents the distance between the mirror model and the origin of the mirror camera coordinate system.

[0141] Optionally, multiply the inverse matrix of the matrix corresponding to the viewing parameters of the mirror camera by the position data of the mirror model to obtain a multiplication result. Dot-multiply the multiplication result by the normal vector of the mirror model, and take the negative of the obtained result as the distance parameter. That is to say, the viewing parameter of the mirror camera is V, the position data of the mirror model is D, and the normal vector of the mirror model is N v , then the distance parameter D of the mirror model v Satisfy: D v = -(V - ×D)·N v .

[0142] Generally, the development engine comes with a calculation function for the projection matrix. For example, the calculation function of this projection matrix is the CalculateObliqueMatrix function. The calculation function of the projection matrix is used to describe the mapping relationship between the near clipping plane and the projection matrix. Exemplarily, the near clipping plane of the mirror camera is the mirror model in the viewing space of the mirror camera, which can be characterized by the normal vector and distance parameter of the mirror model. Taking the normal vector and distance parameter of the mirror model as the two independent variables of the calculation function of the projection matrix, the near clipping plane located in the viewing space of the mirror camera is specified. After mapping through the calculation function, an initial projection matrix is obtained, and the calculation process of the calculation function will not be elaborated here.

[0143] Optionally, if the near clipping plane of the mirror camera is the mirror model located in the viewing space of the mirror camera, then the initial projection matrix is not perpendicular to the line of sight of the mirror camera in a constant form. In this case, the initial projection matrix can be called an oblique projection matrix.

[0144] After obtaining the oblique projection matrix, since the viewing angle of mirror rendering is reversed left and right, the first row of the initial projection matrix needs to be negated to obtain the projection matrix of the mirror camera.

[0145] Step 203: Render the object model in the blank image based on the camera parameters of the mirror camera and the mirror mask to obtain a first image. The mirror area in the first image includes the mirror image of the object represented by the object model.

[0146] In the embodiments of the present application, a shader can be used to implement the rendering process of the mirror camera. A shader is a programmable program that can be used to replace the fixed rendering pipeline and is used to implement image rendering. Mask test code for indicating rendering based on the mirror mask can be added to the shader, and the camera parameters of the mirror camera can be set in the shader, so that the shader can render the object model in the blank image based on the camera parameters of the mirror camera and the mirror mask to obtain the first image.

[0147] Optionally, the mask test code is as follows.

[0148]

[0149] From the above code, it can be seen that the shader will perform a mask test during the mirror rendering process, so that the rendering of the mirror camera does not exceed the unoccluded area of the mirror mask, that is, does not exceed the mirror area. Whether to use the initial mask is represented by the reference value _ReflectStencil, and the mirror mask is obtained by combining the initial masks and written into the mirror mask. The mask test parameter of the shader is _ReflectStencilComp, and the default value of the mask test parameter is a set value. For example, the set value is Disabled, and Disabled represents turning off the mask test. Before the shader performs mirror rendering, the value of _ReflectStencilComp is set to Equal, and Equal represents mirror rendering based on the mask test to implement the drawing of the virtual scene.

[0150] That is to say, the shader can use the mirror mask to occlude the blank image to obtain the unoccluded area in the blank image. In addition, based on the camera parameters of the mirror camera, the shader projects any point in the frustum of the mirror camera onto the blank image. If the point is located in the unoccluded area of the blank image, pixel information is written in the unoccluded area of the blank image. If the point is not located in the unoccluded area of the blank image, pixel information is not written. In this way, the object represented by the object model is written into the area where the mirror is located in the blank image to obtain the first image.

[0151] In an exemplary embodiment, step 203 includes: obtaining an inversion command, where the inversion command is used to mirror-invert the viewing space of the mirror camera; rendering the object model in the blank image based on the inversion command, the camera parameters of the mirror camera, and the mirror mask to obtain the first image.

[0152] Since the viewing spaces of the main camera and the mirror camera are mirror images of each other, the main camera coordinate system and the mirror camera coordinate system are left - hand and right - hand coordinate systems of each other. That is to say, if the main camera coordinate system is a left - hand coordinate system, the mirror camera coordinate system needs to be a right - hand coordinate system. Similarly, if the main camera coordinate system is a right - hand coordinate system, the mirror camera coordinate system needs to be a left - hand coordinate system. Based on this, the electronic device can obtain an inversion command, and through the inversion command, the viewing space of the mirror camera is mirror - inverted, so that the viewing space of the mirror camera presents a mirror image of the viewing space of the main camera.

[0153] The embodiments of the present application do not limit the way of obtaining the inversion command. Exemplarily, the electronic device can obtain the inversion command input by the user, or, before determining to perform mirror rendering, the electronic device obtains a pre - configured inversion command. Optionally, the electronic device generates a camera command queue, and the camera command queue is used to set the camera parameters of the main camera or the mirror camera used for rendering. During the process of passing in the camera parameters of the mirror camera in the camera command queue, an inversion command can also be passed in. For example, pass in InvertCulling, and through the inversion command, the viewing space of the mirror camera is inverted, so that the Graphics Processing Unit (GPU) inverts the determination result of the front and back.

[0154] Next, the shader uses the mirror mask to occlude the blank image to obtain the unoccluded area in the blank image. In addition, based on the camera parameters of the mirror camera and the inversion command, the shader projects any point in the frustum of the mirror camera onto the blank image. If the point is located in the unoccluded area of the blank image, pixel information is written in the unoccluded area of the blank image. If the point is not located in the unoccluded area of the blank image, no pixel information is written.

[0155] In the embodiments of the present application, the first image includes pixel information of each third point on the surface of the object model facing the mirror camera. That is to say, if any point on the surface of the object model faces the mirror camera, then this point is a third point. The mirror camera is used to perform virtual imaging on each third point and will not perform virtual imaging on other points on the surface of the object model except the third points.

[0156] Any pixel point on the mirror area of the blank image corresponds to a line of sight of the mirror camera. If the line of sight passes through a third point, the electronic device can write the pixel information of the third point at the pixel point in the mirror area of the blank image. If the line of sight does not pass through a third point, the electronic device can write the set pixel information at the pixel point in the mirror area of the blank image. Any pixel point on the mirror area is the second point mentioned below. Based on this, it can be known that the pixel information of the second point is the set pixel information or the pixel information of the third point.

[0157] Step 204: Based on the camera parameters of the main camera, render the mirror model and the object model in the first image to obtain a rendered image.

[0158] In the embodiments of the present application, the electronic device may project any point in the viewing frustum of the main camera onto the first image based on the camera parameters of the main camera, and write pixel information in the first image. In this way, the first image is rendered to obtain a rendered image.

[0159] In an exemplary embodiment, the rendered image includes pixel information of each first point located on the surface of the object model facing the main camera. That is to say, if any point on the surface of the object model faces the main camera, then this point is a first point. The main camera is used to perform virtual imaging on each first point, and will not perform virtual imaging on other points on the surface of the object model except the first points.

[0160] Among them, step 204 includes steps 2041 to 2043 (not shown in the figure).

[0161] Step 2041: Based on the camera parameters of the main camera and the vertex position data of the mirror model, determine the depth of each second point on the mirror model, where the depth of the second point represents the distance between the second point and the main camera.

[0162] In the embodiments of the present application, the camera parameters of the main camera include the position data of the main camera, and the position data is used to represent the position of the main camera in the world coordinate system. The mirror model includes multiple vertices, and the vertex position data of any vertex is used to represent the position of the vertex in the world coordinate system. The depth of the vertex can be determined based on the position data of the main camera and the vertex position data of the vertex, and the distance between the vertex and the main camera is represented by the depth of the vertex.

[0163] The mirror model includes multiple second points. The second point is a point on the mirror represented by the mirror model. The second point can be a vertex of the mirror model or a non-vertex on the mirror model. For example, when the mirror model is a quadrilateral, since the quadrilateral corresponds to four vertices, therefore, the mirror model includes four vertices, but the number of second points included in the mirror model is greater than 4. Any second point can be a vertex of the quadrilateral or a point on any side of the quadrilateral (such as the midpoint of the side, etc.) or a point inside the quadrilateral (such as the centroid of the quadrilateral, the intersection point of the diagonals, etc.).

[0164] For any non-vertex, the depth of the non-vertex can be obtained by interpolating the depths of at least one vertex. The interpolation method will not be elaborated here. For example, when the mirror model is a quadrilateral and any non-vertex is the midpoint of an edge, the depth of the non-vertex can be obtained by interpolating the depths of the two vertices at both ends of the edge. Another example is that if any non-vertex is the centroid of the quadrilateral, the depth of the non-vertex can be obtained by interpolating the depths of the four vertices.

[0165] Based on this, the depths of each vertex and each non-vertex can be obtained, and thus the depths of each second point can be obtained.

[0166] Step 2042, if the line of sight of the main camera passes through any first point and any second point, and the depth of any first point is not greater than the depth of any second point, write the pixel information of any first point in the first image. The depth of any first point represents the distance between any first point and the main camera.

[0167] In the embodiment of the present application, any pixel point on the first image corresponds to a line of sight of the main camera. If this line of sight passes through the first point and the second point, the electronic device can determine the depth of the first point based on the position data of the first point in the world coordinate system and the position data of the main camera in the world coordinate system, and represent the distance between the first point and the main camera through the depth of the first point.

[0168] If the depth of the first point is greater than the depth of the second point, it means that the distance between the first point and the main camera is greater than the distance between the second point and the main camera, that is, the line of sight of the main camera passes through the second point and the first point in sequence. In this case, the first point is blocked by the second point, that is to say, the first point is located behind the mirror model, and there is no need to write the pixel information of the first point in the first image.

[0169] Similarly, if the depth of the first point is less than the depth of the second point, it means that the distance between the first point and the main camera is less than the distance between the second point and the main camera, that is, the line of sight of the main camera passes through the first point and the second point in sequence. In this case, it is necessary to write the pixel information of the first point at the pixel point corresponding to the line of sight of the main camera in the first image. It can be understood that before writing the pixel information of the first point at the pixel point, the pixel information of the second point already corresponds to this pixel point. Based on this, it is necessary to delete the pixel information of the second point corresponding to this pixel point and rewrite the pixel information of the first point at the pixel point.

[0170] It can be understood that if the depth of the first point is equal to the depth of the second point, it means that the distance between the first point and the main camera is equal to the distance between the second point and the main camera, that is, the line of sight of the main camera passes through the second point and the first point at the same time. In this case, it is possible not to write the pixel information of the first point in the first image, or it is also possible to write the pixel information of the first point at the pixel point corresponding to the line of sight of the main camera in the first image.

[0171] Step 2043, if the line of sight of the main camera passes through any one of the first points and does not pass through any one of the second points, write the pixel information of any one of the first points in the first image.

[0172] In the embodiments of the present application, if the line of sight of the main camera passes through the first point and does not pass through the second point, the electronic device does not need to perform depth determination, and directly writes the pixel information of the first point at the pixel point corresponding to the line of sight of the main camera in the first image.

[0173] It can be understood that if the line of sight of the main camera does not pass through the first point and does not pass through the second point, the electronic device can write the set pixel information at the pixel point corresponding to the line of sight. If the line of sight of the main camera passes through the second point and does not pass through the first point, the electronic device does not need to write pixel information at the pixel point corresponding to the line of sight.

[0174] The embodiments of the present application do not limit the pixel information mentioned above. Exemplarily, the pixel information may be at least one of a color value, a depth value, a texture value, etc.

[0175] It can be understood that before rendering based on the camera parameters of the main camera, rendering is performed based on the camera parameters of the mirror camera. When using the mirror camera for rendering, an inversion command is used to mirror-invert the viewing space of the mirror camera and reverse the determination result of the GPU for the front and back. When using the main camera for rendering, there is no need to invert the viewing space of the main camera. That is to say, the inversion command can be directly cancelled to avoid the GPU reversing the determination result of the front and back. It is also possible to reverse the determination result of the GPU for the front and back again. That is, before step 204, it further includes: obtaining a reduction command, and instructing, through the reduction command, that the viewing space of the main camera is mirror-inverted with respect to the viewing space of the mirror camera to restore the determination result of the GPU for the front and back.

[0176] In addition, when using the mirror camera for rendering, a mirror mask is used for rendering to ensure that the area of the mirror rendering is located in the area where the mirror is located in the blank image. When using the main camera for rendering, there is no need to use the mirror mask. Based on this, before step 204, it further includes: deleting the mirror mask.

[0177] Optionally, set the mask test parameter _ReflectStencilComp of the shader to a set value. For example, the set value is Disabled, and it is indicated by the set value that the mask test is turned off, so that the shader does not use the mirror mask, but renders in the global area of the first image based on the camera parameters of the main camera.

[0178] Optionally, the mirror mask is obtained by combining multiple initial masks, and multiple reference values _ReflectStencil are used to indicate whether to use the initial masks. Each reference value can be reset to a first value so that each reference value indicates not to use the corresponding initial mask, thereby deleting the mirror mask.

[0179] In addition, when using the mirror camera for rendering, the shader can project any point in the frustum of the mirror camera onto a blank image based on the camera parameters of the mirror camera. That is to say, during the mirror rendering process, the electronic device can obtain the depth value corresponding to any point in the blank image, and this depth value represents the distance between the point corresponding to this point in the frustum of the mirror camera and the origin of the mirror camera coordinate system.

[0180] Generally, through the projection matrix of the mirror camera, each point in the virtual scene can be transformed into the clip space of the mirror camera. The frustum of the mirror camera is a part of the clip space of the mirror camera and is used to describe the space that the mirror camera can observe. The depth value of any point in the clip space of the mirror camera is the distance between this point and the origin of the mirror camera coordinate system, and the depth values of each point in the clip space of the mirror camera include the depth values corresponding to each point in the blank image. The electronic device can store the depth values of each point in the clip space of the mirror camera in a buffer area.

[0181] Before using the main camera for rendering, since the mirror camera has been rendered and the buffer area stores the various depth values involved in the mirror rendering, therefore, the various depth values in the buffer area can be deleted to empty the buffer area, so that the depth values involved in the main camera rendering can be stored in the buffer area.

[0182] An observation matrix of the main camera can be generated based on the position data, orientation data, etc. of the main camera, and the position and orientation of the main camera in the world space are represented by the observation matrix of the main camera. The observation matrix and projection matrix of the main camera can be passed into the shader so that the shader can render the mirror model and the object model in the first image based on the camera parameters of the main camera to obtain a rendered image.

[0183] Optionally, the settings of the pipeline parameters in the shader are as follows.

[0184]

[0185] It can be seen from the above code that for the rendering pipeline, the mask test can be set to be turned on, and by setting the value of the reference value _ReflectStencil to 0, it is realized that the reference value indicates whether to use the initial mask, thereby deleting the mirror mask. The depth test is turned on and the color writing is turned off.

[0186] In addition, for the vertex shader in the shader, the projection matrix of the main camera can be set for the vertex shader, so that the vertex shader can project any point in the viewing frustum of the main camera onto the first image based on the camera parameters of the main camera. Before the main camera renders, the pixel shader does not need to output, that is, the pixel shader does not need to write pixel information into the first image. During the rendering process of the main camera, the pixel shader is required to write pixel information into the first image to obtain the rendered image.

[0187] From the above, it can be seen that the mirror camera renders the blank image. The screen of the electronic device is used to display the image. Based on this, before the mirror camera enters the rendering process, the screen needs to be cleared. The main camera renders the image obtained by rendering the mirror model. Therefore, after the mirror camera exits the rendering process, it will enter the rendering process of the main camera, and at this time, the screen will no longer be cleared. The environment in which the mirror camera renders, including depth data, region clipping relationships, back face culling, etc., needs to be restored after the mirror camera renders.

[0188] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards in the relevant regions. For example, the mirror model, object model, etc. involved in this application are all obtained under full authorization.

[0189] In the above method, first determine the mirror mask based on the camera parameters of the main camera, and then render the object model in the blank image based on the camera parameters of the mirror camera and the mirror mask to obtain the first image, realizing the rendering of the object in the mirror area of the blank image, so that the first image includes the mirror and the mirror image of the object imaged in the mirror. In addition, the mirror camera only renders in the mirror area, reducing the rendering area and the occupation of rendering resources. Then, based on the camera parameters of the main camera, render the mirror model and the object model in the first image to obtain the rendered image, realizing the rendering of the object in the first image, so that the rendered image includes the mirror, the object and the virtual image of the object imaged in the mirror, improving the accuracy of the rendered image. In addition, the main camera renders in the first image instead of in an additional blank image, which helps to reduce the storage resources occupied by the image. By reducing the occupation of rendering resources in the rendering stage of the mirror camera and reducing the occupation of storage resources in the rendering stage of the main camera, even an electronic device with a relatively low configuration can have better rendering performance, improving the applicability.

[0190] The above describes the image rendering method of the embodiments of the present application from the perspective of method steps. Next, a systematic and comprehensive description will be given. As Figure 4 shown, Figure 4 is a schematic flowchart of image rendering provided by the embodiments of the present application.

[0191] In the embodiments of the present application, when starting to render a frame of an image, it is necessary to determine whether the mirror model is visible in the perspective of the main camera. If the mirror model is not visible in the perspective of the main camera, rendering is directly performed based on the camera parameters of the main camera. If the mirror model is visible in the perspective of the main camera, rendering is first performed based on the camera parameters of the mirror camera, and then rendering is performed based on the camera parameters of the main camera.

[0192] Optionally, the position data and orientation data of the mirror camera can be determined, and the projection matrix of the mirror camera can also be determined, so as to obtain the camera parameters of the mirror camera. Then, pre-rendering of the mirror model is performed based on the camera parameters of the main camera. Among them, the process of pre-rendering the mirror model is the process of determining the mirror mask based on the camera parameters of the main camera. Then, parameters related to mirror camera rendering are set, that is, mask test code is added in the shader, and an inversion command is passed in the camera command queue, etc. Then, rendering is performed based on the camera parameters of the mirror camera to write pixel information in the mirror area of the blank image to obtain a first image. After that, the parameters related to mirror camera rendering are restored, that is, the inversion command in the camera command queue is cancelled to restore the GPU's determination of the front and back, and the mirror mask is closed, etc. Then, post-rendering of the mirror model is performed based on the camera parameters of the main camera. The process of post-rendering the mirror model refers to the process of determining the depth of each second point on the mirror model. Then, rendering is performed based on the camera parameters of the main camera to write pixel information in the first image to obtain a rendered image.

[0193] Optionally, the script code of the rendering solution of the embodiments of the present application is as follows.

[0194]

[0195]

[0196]

[0197] In the above code, preCmd is the pre-rendering command of the mirror camera, which is used to configure the parameters used for pre-rendering the mirror camera. postCmd is the post-rendering command of the mirror camera, which is used to configure the parameters used for post-rendering the mirror camera.

[0198] As can be seen from the above code, on the one hand, the position and orientation of the mirror camera are determined to obtain the viewing angle parameters of the main camera. On the other hand, the projection matrix of the main camera is determined. Then, based on the viewing angle parameters and the projection matrix of the main camera, pre-rendering of the mirror model is performed to determine the mirror mask corresponding to the mirror area. Then, an inversion command is passed in and the mirror mask is opened to set the parameters related to the mirror camera rendering. After the mirror camera performs rendering, the inversion command is revoked and the mirror mask is closed to restore the parameters related to the mirror camera rendering. Then, post-rendering of the mirror model is performed based on the viewing angle parameters and the projection matrix of the main camera, and then the main camera performs rendering.

[0199] In an exemplary embodiment, the image rendering method in the embodiments of the present application can be implemented based on a development engine, so that for a scene with a mirror rendering requirement, the development engine can be used to quickly render a rendered image, which is briefly described below.

[0200] First, an image rendering resource package can be introduced into the development engine. By parsing the image rendering resource package, two shader resources and one script resource are obtained. As Figure 5 shown, Figure 5 (1) in shows the script resource, which is used to create a mirror model. Figure 5 (2) in shows the first shader resource, which is a pre-rendering shader resource and is used to perform pre-rendering of the mirror model based on the camera parameters of the main camera. Figure 5 (3) in shows the second shader resource, which is a post-rendering shader resource and is used to perform post-rendering of the mirror model based on the camera parameters of the main camera.

[0201] The development engine can provide an initial virtual scene, as Figure 6 shown, Figure 6 shows an initial virtual scene. The initial virtual scene includes a main camera (Main Camera), a directional light (Directional Light), and a group of object (Objects) models. Among them, the coordinate system where the main camera is located is the main camera coordinate system, and the main camera coordinate system includes three mutually perpendicular coordinate axes. The object models include a cylinder (Cylinder) model, a sphere (Sphere) model, and a plane (Plane) model, and the cylinder model and the sphere model are located on the plane model. In addition, the initial virtual scene also includes a cube (Cube) model, which is used to place the mirror model.

[0202] Figure 6 also shows the directory hierarchy, which is used to display each object included in the initial virtual scene. In addition, Figure 6A toolbar is also shown, which includes at least one tool. It can be understood that Figure 6 The tool identifiers, the number of tools, etc. shown in the toolbar are all illustrative, and the functions of each tool are also illustrative. For example, in the order from top to bottom, Figure 6 The functions of the 5 tools shown are in sequence: click function, move function, refresh function, zoom function, and crop function.

[0203] It can be understood that by using the camera parameters of the main camera to Figure 6 render the virtual scene shown, a rendered image as Figure 7 shown can be obtained. The rendered image includes a plane, a cylinder, a sphere, and a cube.

[0204] Next, add the following mask test code in the shader. The meaning of this code has been described above and will not be elaborated here.

[0205]

[0206] Then, create a Mirror model in the cube model. Optionally, add script resources in the development engine to be used for creating the mirror model, as Figure 8 shown, Figure 8 a setting page for a mirror model is shown. It can be set that the virtual scene includes a main camera and a mirror camera, and the parameters of the mirror camera are set. The parameters of the mirror camera include the position data of the mirror camera, the rotation data of the mirror camera, and the scaling factor of the mirror camera. In addition, the setting page also displays the resource files related to the mirror camera, including script resources, pre-rendered shader resources, and pre-rendered shader resources.

[0207] After completing the above configuration, based on the camera parameters of the main camera and the camera parameters of the mirror camera, render the above virtual scene, and a rendered image as Figure 9 shown can be obtained. As Figure 9 can be seen, the mirror is successfully rendered into the image, and the content in the mirror accurately reflects the virtual scene. The virtual scene includes a cylinder, a sphere, a plane, and a cube. That is to say, the rendered image includes the mirror, each object in the virtual scene, and the mirror images of each object imaged in the mirror. It can be understood that by moving the main camera or modifying the virtual scene including the mirror model, it can be seen that the mirror rendering content is updated in real time and accurately, meeting the real-time rendering requirements of the planar reflective mirror.

[0208] In addition, after rendering the virtual scene, within the hierarchy where the cube in the target hierarchy is located, there is a hierarchy where a mirror is located, as Figure 10As shown in the figure. The layer where the mirror surface is located includes the layer where the render target is located. Among them, there are no layers included within the layer where the render target is located. That is to say, the value of the render target is None. During the rendering process of the mirror camera, the rendering result is not rendered to an independent render target, and the render target of the mirror camera is the same as that of the main camera. By setting the rendering of the mirror camera and the main camera under the same render target, it is possible to reduce resources such as storage and video memory required for rendering, meeting the performance requirements of low overhead.

[0209] From the above content, it can be seen that the render target of the mirror camera and the main camera is the same, and the rendering of the mirror camera is earlier than that of the main camera. By adding a pre-processing command before the rendering of the mirror camera, a mirror mask is determined based on the camera parameters of the main camera, so that the rendering of the mirror camera does not exceed the unoccluded area of the mirror mask. In addition, the viewing space, projection space, face culling, rendering area, depth, etc. required for the rendering process of the mirror camera can also be set. By adding a post-processing command after the rendering of the mirror camera, the mirror mask is erased. In addition, the settings required for the rendering process of the mirror camera need to be restored. This enables the rendering of the mirror camera to be fully embedded before the rendering of the main camera, so that during the rendering stages of the main camera and the mirror camera, the same render target is rendered, saving a large amount of reading and writing of GPU render targets, saving the memory of the render target, that is, reducing the occupancy of GPU memory and GPU read-write bandwidth, and reducing the overhead of GPU hardware.

[0210] For electronic devices with poor performance, such as mobile terminals. Since the overhead brought by switching the render target is more obvious, for a solution that renders on different render targets, the rendering performance of mobile terminals is one of the bottlenecks of this solution. By optimizing the GPU memory and bandwidth in the embodiments of this application, electronic devices with poor performance can also perform rendering relatively smoothly, improving the rendering performance of the electronic devices.

[0211] In addition, all the content related to the solution of the embodiments of this application is aggregated in one script resource and two shader resources, enabling the script resource and shader resources to be conveniently deployed in the development engine, so that the solution of the embodiments of this application can be flexibly applied to various rendering engines and application scenarios, with stronger flexibility and controllability.

[0212] Figure 11 The following shows a schematic structural diagram of an image rendering device provided by an embodiment of this application, as Figure 11 shown, the device includes:

[0213] A determination module 1101 is configured to determine a mirror mask based on the camera parameters of a main camera. The main camera is used to perform virtual imaging on a mirror model and an object model included in a virtual scene. The mirror mask is used to occlude other regions in a blank image except for the mirror region, and the mirror region is the image region where the mirror represented by the mirror model is located.

[0214] An acquisition module 1102 is configured to acquire the camera parameters of a mirror camera. The mirror camera is used to perform virtual imaging on the object model based on the mirror mask.

[0215] A rendering module 1103 is configured to render the object model in the blank image based on the camera parameters of the mirror camera and the mirror mask, to obtain a first image. The mirror region in the first image includes the mirror image of the object represented by the object model.

[0216] The rendering module 1103 is further configured to render the mirror model and the object model in the first image based on the camera parameters of the main camera, to obtain a rendered image.

[0217] In a possible implementation, the determination module 1101 is configured to determine the mirror mask based on the camera parameters of the main camera if the angle between the mirror orientation of the mirror model and the pointing orientation of the main camera pointing to the mirror model is an obtuse angle or a straight angle, and the mirror model is located within the viewing frustum of the main camera.

[0218] In a possible implementation, the acquisition module 1102 is further configured to acquire the position data of the main camera, the position data of the mirror model, and a first vector representing the mirror orientation.

[0219] The determination module 1101 is further configured to determine a second vector representing the pointing orientation based on the position data of the main camera and the position data of the mirror model.

[0220] The determination module 1101 is further configured to determine that the angle between the mirror orientation and the pointing orientation is an obtuse angle or a straight angle based on the first vector and the second vector.

[0221] In a possible implementation, the acquisition module 1102 is further configured to acquire the vertex position data of the mirror model.

[0222] The determination module 1101 is further configured to determine, for any clipping plane on the viewing frustum of the main camera, a judgment result of any clipping plane based on the vertex position data and the plane parameters of any clipping plane. The judgment result of any clipping plane represents whether the vertex corresponding to the vertex position data is located on the side of the viewing frustum of the main camera where any clipping plane is located.

[0223] The determination module 1101 is further configured to determine that the mirror model is located within the viewing frustum of the main camera based on the judgment results of the respective clipping planes.

[0224] In a possible implementation, the mirror mask includes an unoccluded area, and the unoccluded area corresponds to the mirror area;

[0225] A determination module 1101, configured to determine vertex position data of the unoccluded area based on camera parameters of the main camera and vertex position data of the mirror model; and determine the mirror mask based on the vertex position data of the unoccluded area.

[0226] In a possible implementation, the camera parameters of the mirror camera include the viewing angle parameters and the projection matrix of the mirror camera;

[0227] An acquisition module 1102, configured to acquire the viewing angle parameters of the mirror camera, where the viewing angle parameters of the mirror camera characterize at least one of the position or orientation of the mirror camera; and determine the projection matrix of the mirror camera based on the viewing angle parameters of the mirror camera.

[0228] In a possible implementation, the viewing angle parameters of the mirror camera include the position data and the orientation data of the mirror camera;

[0229] An acquisition module 1102, configured to determine the position data of the mirror camera based on the position data of the main camera, the position data of the mirror model, and a first vector characterizing the mirror orientation; and determine the orientation data of the mirror camera based on the first vector and the orientation data of the main camera.

[0230] In a possible implementation, an acquisition module 1102 is configured to determine a normal vector characterizing the mirror orientation based on the viewing angle parameters of the mirror camera and a first vector characterizing the mirror orientation, where the first vector is a vector based on the world coordinate system and the normal vector is a vector based on the camera coordinate system of the mirror camera; determine a distance parameter between the mirror model and the origin based on the viewing angle parameters of the mirror camera, the normal vector, and the position data of the mirror model, where the position data of the mirror model is data based on the world coordinate system and the distance parameter is a parameter based on the camera coordinate system of the mirror camera; and determine the projection matrix of the mirror camera based on the normal vector and the distance parameter.

[0231] In a possible implementation, a rendering module 1103 is configured to acquire an inversion command, where the inversion command is used to mirror-invert the viewing space of the mirror camera; and render an object model in a blank image based on the inversion command, the camera parameters of the mirror camera, and the mirror mask to obtain a first image.

[0232] In a possible implementation, the rendered image includes pixel information of each first point on the surface of the object model facing the main camera;

[0233] A rendering module 1103 is configured to determine the depth of each second point on the mirror model based on the camera parameters of the main camera and the vertex position data of the mirror model. The depth of the second point represents the distance between the second point and the main camera. If the line of sight of the main camera passes through any first point and any second point, and the depth of any first point is not greater than the depth of any second point, the pixel information of any first point is written into the first image. The depth of any first point represents the distance between any first point and the main camera. If the line of sight of the main camera passes through any first point and does not pass through any second point, the pixel information of any first point is written into the first image.

[0234] In the above device, the mirror mask is first determined based on the camera parameters of the main camera, and then the object model is rendered in the blank image based on the camera parameters of the mirror camera and the mirror mask to obtain the first image, realizing the rendering of the object in the mirror area of the blank image, so that the first image includes the mirror and the mirror image of the object imaged in the mirror. In addition, the mirror camera only renders in the mirror area, reducing the rendering area and the occupation of rendering resources. Then, based on the camera parameters of the main camera, the mirror model and the object model are rendered in the first image to obtain the rendered image, realizing the rendering of the object in the first image, so that the rendered image includes the mirror, the object, and the virtual image of the object imaged in the mirror, improving the accuracy of the rendered image. In addition, the main camera renders in the first image instead of in an additional blank image, which is beneficial to reducing the storage resources occupied by the image. By reducing the occupation of rendering resources in the rendering stage of the mirror camera and reducing the occupation of storage resources in the rendering stage of the main camera, even an electronic device with a lower configuration can have better rendering performance, improving the applicability.

[0235] It should be understood that when the above Figure 11 When the provided device realizes its functions, only the above division of each functional module is used for illustration. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device provided in the above embodiment and the method embodiment belong to the same concept, and the specific implementation process can be seen in the method embodiment, which will not be elaborated here.

[0236] Figure 12 The structural block diagram of a terminal device 1200 provided by an exemplary embodiment of the present application is shown. The terminal device 1200 includes a processor 1201 and a memory 1202.

[0237] The processor 1201 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. The processor 1201 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 1201 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1201 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1201 may further include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.

[0238] The memory 1202 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 1202 may further include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1202 is used to store at least one computer program, and the at least one computer program is used to be executed by the processor 1201 to implement the image rendering method provided in the method embodiments of the present application.

[0239] In some embodiments, the terminal device 1200 may further optionally include: a peripheral device interface 1203 and at least one peripheral device. The processor 1201, the memory 1202, and the peripheral device interface 1203 may be connected through a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 1203 through a bus, signal lines, or a circuit board. Specifically, the peripheral devices include at least one of a radio frequency circuit 1204, a display screen 1205, a camera assembly 1206, an audio circuit 1207, and a power supply 1208.

[0240] The peripheral device interface 1203 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 1201 and the memory 1202. In some embodiments, the processor 1201, the memory 1202, and the peripheral device interface 1203 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1201, the memory 1202, and the peripheral device interface 1203 can be implemented on a separate chip or circuit board, and this embodiment does not limit this.

[0241] The radio frequency circuit 1204 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1204 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1204 converts an electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1204 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and so on. The radio frequency circuit 1204 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G, and 5G), a wireless local area network, and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1204 may further include a circuit related to NFC (Near Field Communication), and this application does not limit this.

[0242] The display screen 1205 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1205 is a touch display screen, the display screen 1205 also has the ability to collect touch signals on or above the surface of the display screen 1205. The touch signals can be input to the processor 1201 as control signals for processing. At this time, the display screen 1205 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, there may be one display screen 1205, which is provided on the front panel of the terminal device 1200; in other embodiments, there may be at least two display screens 1205, which are respectively provided on different surfaces of the terminal device 1200 or are in a foldable design; in other embodiments, the display screen 1205 may be a flexible display screen, which is provided on the curved surface or the folding surface of the terminal device 1200. Even further, the display screen 1205 can also be set to an irregular non-rectangular shape, that is, a special-shaped screen. The display screen 1205 can be prepared using materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0243] The camera module 1206 is used to collect images or videos. Optionally, the camera module 1206 includes a front camera and a rear camera. Generally, the front camera is provided on the front panel of the terminal, and the rear camera is provided on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera respectively, to achieve functions such as background blurring by fusing the main camera and the depth-of-field camera, panoramic shooting by fusing the main camera and the wide-angle camera, and VR (Virtual Reality) shooting functions or other fused shooting functions. In some embodiments, the camera module 1206 may further include a flash. The flash can be a single-color-temperature flash or a two-color-temperature flash. A two-color-temperature flash refers to a combination of a warm-light flash and a cold-light flash, which can be used for light compensation under different color temperatures.

[0244] The audio circuit 1207 may include a microphone and a speaker. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into electrical signals for input to the processor 1201 for processing, or input to the radio frequency circuit 1204 to achieve voice communication. For the purpose of stereo collection or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal device 1200. The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert the electrical signal from the processor 1201 or the radio frequency circuit 1204 into sound waves. The speaker may be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for uses such as ranging. In some embodiments, the audio circuit 1207 may further include a headphone jack.

[0245] The power supply 1208 is used to supply power to each component in the terminal device 1200. The power supply 1208 may be alternating current, direct current, a disposable battery or a rechargeable battery. When the power supply 1208 includes a rechargeable battery, the rechargeable battery may be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery may also be used to support fast charging technology.

[0246] In some embodiments, the terminal device 1200 further includes one or more sensors 1209. The one or more sensors 1209 include but are not limited to: an acceleration sensor 1211, a gyroscope sensor 1212, a pressure sensor 1213, an optical sensor 1214, and a proximity sensor 1215.

[0247] The acceleration sensor 1211 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established with the terminal device 1200. For example, the acceleration sensor 1211 can be used to detect the components of the gravitational acceleration on the three coordinate axes. The processor 1201 can control the display screen 1205 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 1211. The acceleration sensor 1211 can also be used for the collection of game or user's motion data.

[0248] The gyroscope sensor 1212 can detect the body direction and rotation angle of the terminal device 1200. The gyroscope sensor 1212 can cooperate with the acceleration sensor 1211 to collect the 3D actions of the user on the terminal device 1200. According to the data collected by the gyroscope sensor 1212, the processor 1201 can achieve the following functions: motion sensing (such as changing the UI according to the user's tilting operation), image stabilization during shooting, game control, and inertial navigation.

[0249] The pressure sensor 1213 can be disposed on the side frame of the terminal device 1200 and / or the lower layer of the display screen 1205. When the pressure sensor 1213 is disposed on the side frame of the terminal device 1200, it can detect the holding signal of the user on the terminal device 1200, and the processor 1201 can perform left / right hand recognition or quick operation according to the holding signal collected by the pressure sensor 1213. When the pressure sensor 1213 is disposed on the lower layer of the display screen 1205, the processor 1201 can control the operable controls on the UI interface according to the pressure operation of the user on the display screen 1205. The operable controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0250] The optical sensor 1214 is used to collect the ambient light intensity. In one embodiment, the processor 1201 can control the display brightness of the display screen 1205 according to the ambient light intensity collected by the optical sensor 1214. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1205 is increased; when the ambient light intensity is low, the display brightness of the display screen 1205 is decreased. In another embodiment, the processor 1201 can also dynamically adjust the shooting parameters of the camera module 1206 according to the ambient light intensity collected by the optical sensor 1214.

[0251] The proximity sensor 1215, also known as a distance sensor, is usually disposed on the front panel of the terminal device 1200. The proximity sensor 1215 is used to collect the distance between the user and the front of the terminal device 1200. In one embodiment, when the proximity sensor 1215 detects that the distance between the user and the front of the terminal device 1200 is gradually decreasing, the processor 1201 controls the display screen 1205 to switch from the lit state to the off state; when the proximity sensor 1215 detects that the distance between the user and the front of the terminal device 1200 is gradually increasing, the processor 1201 controls the display screen 1205 to switch from the off state to the lit state.

[0252] Those skilled in the art can understand that Figure 12 the structure shown in does not limit the terminal device 1200, and it may include more or fewer components than shown in the figure, or combine some components, or adopt different component arrangements.

[0253] Figure 13It is a schematic structural diagram of the server provided by the embodiment of the present application. The server 1300 may vary greatly due to different configurations or performances, and may include one or more processors 1301 and one or more memories 1302. Among them, at least one computer program is stored in the one or more memories 1302, and the at least one computer program is loaded and executed by the one or more processors 1301 to implement the image rendering method provided by each of the above method embodiments. Exemplarily, the processor 1301 is a CPU. Of course, the server 1300 may also have components such as a wired or wireless network interface, a keyboard, and an input / output interface for input and output. The server 1300 may also include other components for implementing the functions of the device, which will not be elaborated here.

[0254] In an exemplary embodiment, a computer-readable storage medium is also provided. At least one computer program is stored in the storage medium, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any one of the above image rendering methods.

[0255] Optionally, the above computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0256] In an exemplary embodiment, a computer program is also provided. The computer program is at least one, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any one of the above image rendering methods.

[0257] In an exemplary embodiment, a computer program product is also provided. At least one computer program is stored in the computer program product, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any one of the above image rendering methods.

[0258] It should be understood that "a plurality of" mentioned herein refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0259] The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0260] The above are only exemplary embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present application shall be included within the protection scope of the present application.

Claims

1. An image rendering method, characterized in that, The method includes: Determining a mirror mask based on the camera parameters of the main camera, where the main camera is used for virtual imaging of a mirror model and an object model included in a virtual scene, and the mirror mask is used to occlude other areas in a blank image except for the mirror area, and the mirror area is the image area where the mirror represented by the mirror model is located; Obtaining the camera parameters of a mirror camera, where the mirror camera is used for virtual imaging of the object model based on the mirror mask; Rendering the object model in the blank image based on the camera parameters of the mirror camera and the mirror mask to obtain a first image, where the mirror area in the first image includes the mirror image of the object represented by the object model; Rendering the mirror model and the object model in the first image based on the camera parameters of the main camera to obtain a rendered image.

2. The method according to claim 1, wherein The determining the mirror mask based on the camera parameters of the main camera includes: If the angle between the mirror orientation of the mirror model and the pointing orientation of the main camera pointing to the mirror model is an obtuse angle or a straight angle, and the mirror model is located within the frustum of the main camera, determining the mirror mask based on the camera parameters of the main camera.

3. The method according to claim 2, wherein Before the determining the mirror mask based on the camera parameters of the main camera, it further includes: Obtaining the position data of the main camera, the position data of the mirror model, and a first vector representing the mirror orientation; Determining a second vector representing the pointing orientation based on the position data of the main camera and the position data of the mirror model; Determining that the angle between the mirror orientation and the pointing orientation is an obtuse angle or a straight angle based on the first vector and the second vector.

4. The method according to claim 2, characterized in that, Before the determining the mirror mask based on the camera parameters of the main camera, it further includes: Obtaining the vertex position data of the mirror model; For any clipping plane on the frustum of the main camera, determining the judgment result of the any clipping plane based on the vertex position data and the plane parameters of the any clipping plane, where the judgment result of the any clipping plane represents whether the vertex corresponding to the vertex position data is located on the side of the frustum of the main camera where the any clipping plane is located; Determining that the mirror model is located within the frustum of the main camera based on the judgment results of each clipping plane.

5. The method according to claim 1, wherein The mirror mask includes an unoccluded area, and the unoccluded area corresponds to the mirror area; The determining the mirror mask based on the camera parameters of the main camera includes: Determining the vertex position data of the unoccluded area based on the camera parameters of the main camera and the vertex position data of the mirror model; Determining the mirror mask based on the vertex position data of the unoccluded area.

6. The method according to claim 1, wherein The camera parameters of the mirror camera include the viewing angle parameters and the projection matrix of the mirror camera; the obtaining the camera parameters of the mirror camera includes: Obtaining the viewing angle parameters of the mirror camera, where the viewing angle parameters of the mirror camera represent at least one of the position or orientation of the mirror camera; Determining the projection matrix of the mirror camera based on the viewing angle parameters of the mirror camera.

7. The method according to claim 6, wherein The viewing angle parameters of the mirror camera include the position data of the mirror camera and the orientation data of the mirror camera; obtaining the viewing angle parameters of the mirror camera includes: Determining the position data of the mirror camera based on the position data of the main camera, the position data of the mirror model, and a first vector representing the mirror orientation; Determining the orientation data of the mirror camera based on the first vector and the orientation data of the main camera.

8. The method according to claim 6, characterized in that, The determining the projection matrix of the mirror camera based on the viewing angle parameters of the mirror camera includes: Determining a normal vector representing the mirror orientation based on the viewing angle parameters of the mirror camera and the first vector representing the mirror orientation, where the first vector is a vector based on the world coordinate system and the normal vector is a vector based on the camera coordinate system of the mirror camera; Determining a distance parameter between the mirror model and the origin based on the viewing angle parameters of the mirror camera, the normal vector, and the position data of the mirror model, where the position data of the mirror model is data based on the world coordinate system and the distance parameter is a parameter based on the camera coordinate system of the mirror camera; Determining the projection matrix of the mirror camera based on the normal vector and the distance parameter.

9. The method according to claim 1, wherein The rendering the object model in the blank image based on the camera parameters of the mirror camera and the mirror mask to obtain a first image includes: Obtaining an inversion command for mirror-inverting the viewing space of the mirror camera; Rendering the object model in the blank image based on the inversion command, the camera parameters of the mirror camera, and the mirror mask to obtain a first image.

10. The method according to claim 1, wherein The rendered image includes pixel information of each first point on the surface of the object model facing the main camera; the rendering the mirror model and the object model in the first image based on the camera parameters of the main camera to obtain a rendered image includes: Determining the depth of each second point on the mirror model based on the camera parameters of the main camera and the vertex position data of the mirror model, where the depth of the second point represents the distance between the second point and the main camera; If the line of sight of the main camera passes through any one of the first points and any one of the second points, and the depth of any one of the first points is not greater than the depth of any one of the second points, writing the pixel information of any one of the first points in the first image, where the depth of any one of the first points represents the distance between any one of the first points and the main camera; If the line of sight of the main camera passes through any one of the first points and does not pass through any one of the second points, writing the pixel information of any one of the first points in the first image.

11. An image rendering device, characterized in that, The apparatus includes: A determining module, configured to determine a mirror mask based on the camera parameters of a main camera, where the main camera is used for virtual imaging of a mirror model and an object model included in a virtual scene, and the mirror mask is used for occluding other regions except the mirror region in a blank image, and the mirror region is the image region where the mirror represented by the mirror model is located; An acquisition module, configured to acquire camera parameters of a mirror camera, where the mirror camera is used to perform virtual imaging on the object model based on the mirror mask; A rendering module, configured to render the object model in the blank image based on the camera parameters of the mirror camera and the mirror mask to obtain a first image, where the mirror area in the first image includes a mirror image of the object represented by the object model; The rendering module is further configured to render the mirror model and the object model in the first image based on the camera parameters of the main camera to obtain a rendered image.

12. An electronic device, characterized in that, The electronic device includes a processor and a memory, and at least one computer program is stored in the memory. The at least one computer program is loaded and executed by the processor so that the electronic device implements the image rendering method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, At least one computer program is stored in the computer-readable storage medium. The at least one computer program is loaded and executed by a processor so that an electronic device implements the image rendering method according to any one of claims 1 to 10.

14. A computer program product, characterized in that, At least one computer program is stored in the computer program product. The at least one computer program is loaded and executed by a processor so that an electronic device implements the image rendering method according to any one of claims 1 to 10.

Images