Image processing method, electronic device, storage medium, program product and chip

Abstract

The application relates to the technical field of terminals, in particular to an image processing method, an electronic device, a storage medium, a program product and a chip. In the method, when a first image is rendered, the electronic device can determine first light map data of each vertex pixel in the first image according to the coordinates of each vertex pixel in the first image in a light map coordinate system corresponding to the first image, wherein the light map coordinate system is a coordinate system corresponding to a first light map corresponding to the first image. The electronic device can determine light map data of non-vertex pixels according to the first light map data of each vertex pixel. The electronic device can render the first image according to the first light map data of the vertex pixels and the non-vertex pixels. In this way, the dependence on memory bandwidth can be reduced, thereby improving the quality of image rendering.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, electronic device, storage medium, program product, and chip. Background Technology

[0002] In some image rendering methods, the graphics processor (GPU) in an electronic device can pre-calculate the lighting of static objects with stable light sources in the image and store the calculation results in memory as lightmaps. Then, during image rendering, the GPU can read the lightmap data from memory and use it to render the image. However, when the memory bandwidth of an electronic device is limited, the GPU's reading of lightmap data from memory can be slow, leading to quality issues such as loss of detail and blurring in certain areas of the rendered image. Summary of the Invention

[0003] To address the issue of poor image rendering quality, embodiments of this application provide an image processing method, an electronic device, a storage medium, a program product, and a chip.

[0004] In a first aspect, embodiments of this application provide an image processing method for processing a first image in an electronic device. The first image includes multiple polygonal patches, comprising: acquiring first lighting map data of each vertex pixel in the multiple polygonal patches in a first lighting map corresponding to the first image; determining second lighting map data of non-vertex pixels in the multiple polygonal patches based on the first lighting map data; and rendering the first image based on the first lighting map data and the second lighting map data.

[0005] Based on the above scheme, by determining the lightmap data of each pixel based on the first lightmap data of the vertex pixels, instead of directly reading the lightmap data of each pixel from the first lightmap, the dependence on memory bandwidth can be reduced, thereby improving the speed of obtaining lightmap data and improving the quality of the first image rendering.

[0006] It is understandable that, in some implementations, electronic devices can acquire the first lightmap data of each vertex pixel in a first lightmap corresponding to a first image in a multi-polygon patch.

[0007] Furthermore, compared to some image processing methods that rely on cloud resources for lightmap reconstruction and rendering, this method reduces reliance on other hardware. Moreover, the image processing method mentioned in this application can be applied not only to Web3D scenes where lightmap reconstruction and rendering are performed using cloud resources, but also to scenes where static global illumination effects are rendered on images to be rendered on mobile devices, and to scenes suitable for rendering volumetric lighting.

[0008] Furthermore, the image processing method mentioned in this application embodiment does not require additional calculation of light map data using spherical harmonic parameters, which can reduce the impact on game performance in game scenes, reduce the loading time of game scenes, and accelerate the iteration speed of game scenes.

[0009] In some implementations of the first aspect, the first image includes a first polygonal patch, the first polygonal patch includes a first vertex pixel, and obtaining the first lightmap data of each vertex pixel in the first lightmap corresponding to the first image in the plurality of polygonal patches includes: obtaining the first coordinate of the first vertex pixel in the image coordinate system corresponding to the first image; determining the second coordinate of the first vertex pixel in the lightmap coordinate system corresponding to the first lightmap based on the first coordinate; and determining the first lightmap data corresponding to the first vertex pixel based on the correspondence between the coordinates in the lightmap coordinate system corresponding to the first lightmap and the lightmap data, and the second coordinate.

[0010] It can be understood that the image coordinate system corresponding to the first image can be a coordinate system established with the lower left corner of the first image as the origin, the first direction x (parallel to the right of the paper) as the first direction, and the second direction y (parallel to the top of the paper) as the second direction. Here, the first direction x and the second direction y are perpendicular to each other, and the unit length of both the first direction x and the second direction y is a pixel. In this way, the electronic device can determine which row and column of the vertex pixel in each triangular facet of the static mesh in the first image, and thus determine the coordinates (x, y) of the vertex pixel in each triangular facet of the static mesh in the image coordinate system corresponding to the first image. i y i ).

[0011] It can be understood that the coordinate system corresponding to the lightmap can be a coordinate system established with the lower left corner of the lightmap as the origin, the third direction u (parallel to the right of the paper) as the first direction, and the fourth direction v (parallel to the top of the paper) as the last direction. The third direction u and the fourth direction v are perpendicular to each other, and the unit length of both the third direction u and the fourth direction v is a pixel. In this way, the electronic device can use the lightmap information corresponding to the static mesh, and the coordinates (x, y, y) of the vertex pixels in each triangle of the static mesh in the image coordinate system corresponding to the first image. i y i ), calculate the coordinates (u) of the vertex pixel in the lightmap coordinate system corresponding to the lightmap. i v i ).

[0012] In some methods of determining the second coordinates, an electronic device can determine the second coordinates of the first vertex pixel in the lightmap coordinate system corresponding to the first lightmap based on the lightmap information of the lightmap corresponding to the first lightmap and the first coordinates. The lightmap information may include information such as texture resolution, scaling, and offset. For example, the electronic device can determine the second coordinates using the following formula:

[0013] (u i v i )=[(x i y i [(scale + offset) * resolution]

[0014] Among them, (u i v i (x) can represent the second coordinate, that is, the row and column number of the vertex pixel in the lightmap of each triangle in the static mesh. i y i ) can represent the first coordinate, scale can represent the scaling amount, offset can represent the offset amount, and resolution can represent the resolution.

[0015] In this embodiment, the first lightmap data corresponding to the first vertex pixel is determined by the correspondence between the coordinates in the lightmap coordinate system corresponding to the first lightmap and the lightmap data, as well as the second coordinates. This eliminates the need to read the lightmap data of each pixel from the first lightmap, reducing the dependence on memory bandwidth and thus improving the speed of obtaining lightmap data and enhancing the quality of the first image rendering.

[0016] In some implementations of the first aspect, determining the first lighting map data corresponding to the first vertex pixel based on the correspondence between the coordinates in the lighting map coordinate system corresponding to the first lighting map and the lighting map data, as well as the second coordinates, includes: inputting the second coordinates into a linear regression model, and outputting the first lighting map data corresponding to the first vertex pixel based on the linear regression model; wherein, the linear regression model includes the correspondence between the coordinates in the lighting map coordinate system corresponding to the first lighting map and the lighting map data.

[0017] In this embodiment of the application, by using a linear regression model to determine the first lighting map data corresponding to the first vertex pixel, the amount of data processing of the electronic device can be reduced, thereby increasing the speed of acquiring lighting map data and further improving the quality of the first image rendering.

[0018] In some implementations of the first aspect, determining the second lighting map data of non-vertex pixels in multiple polygonal patches based on the first lighting map data includes: performing interpolation processing on the first lighting map data of each vertex pixel in the multiple polygonal patches to determine the second lighting map data of non-vertex pixels in the multiple polygonal patches.

[0019] In this embodiment, by interpolating the first lightmap data of vertex pixels, the second lightmap data of non-vertex pixels in multiple polygon patches is determined. This eliminates the need to read the lightmap data of each pixel from the first lightmap, reducing reliance on memory bandwidth and thus improving the speed of acquiring lightmap data and enhancing the quality of the first image rendering.

[0020] In some implementations of the first aspect, the electronic device includes a rendering service and a graphics processor. The rendering service acquires light map data of each vertex pixel in a plurality of polygonal patches in a first light map corresponding to a first image. The graphics processor determines second light map data of non-vertex pixels in the plurality of polygonal patches based on the first light map data. The graphics processor renders the first image based on the first light map data and the second light map data.

[0021] In some implementations of the first aspect, the rendering service includes a scene data extraction module and a lighting data fitting module. The scene data extraction module obtains the first coordinates of the first vertex pixel in the image coordinate system corresponding to the first image. Based on the first coordinates, the scene data extraction module determines the second coordinates of the first vertex pixel in the lighting map coordinate system corresponding to the first lighting map. Based on the correspondence between the coordinates in the lighting map coordinate system corresponding to the first lighting map and the lighting map data, as well as the second coordinates, the lighting data fitting module determines the first lighting map data corresponding to the first vertex pixel.

[0022] In some implementations of the first aspect, the rendering service also includes a fitting data writing module, and the method further includes: the fitting data writing module determines the color data of the vertex pixels in the multiple polygonal patches based on the first light map data of the vertex pixels in the multiple polygonal patches.

[0023] In a second aspect, embodiments of this application provide an electronic device, including: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, for executing any of the image processing methods described in the first aspect or the various implementations provided in the first aspect.

[0024] Thirdly, embodiments of this application provide a readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the image processing method described in the first aspect or any of the various implementations provided in the first aspect.

[0025] Fourthly, embodiments of this application provide a computer program product, which includes computer instructions. When executed by an electronic device, the electronic device performs the image processing method described in the first aspect or any of the various implementations provided in the first aspect.

[0026] Fifthly, embodiments of this application provide a chip including a processor coupled to a memory for executing computer programs or instructions stored in the memory, such that the chip implements the image processing method described in the first aspect or any of the various implementations provided in the first aspect.

[0027] It is understood that the specific implementation methods and beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions of the data filling methods in the first aspect or any of the various implementation methods provided in the first aspect, and will not be repeated here. Attached Figure Description

[0028] Figure 1 According to an embodiment of this application, a schematic diagram of a first image is shown;

[0029] Figure 2 According to an embodiment of this application, a light map representing a first image is shown;

[0030] Figure 3 According to an embodiment of this application, a schematic diagram of the interaction between an electronic device and an image processing system is shown;

[0031] Figure 4 According to an embodiment of this application, a flowchart illustrating an implementation method for scene data extraction is shown;

[0032] Figure 5 According to an embodiment of this application, a schematic diagram is shown where the position of a pixel on a static object in a light map is different from the position of the pixel corresponding to that point in a first image.

[0033] Figure 6 According to an embodiment of this application, a schematic diagram of an image coordinate system corresponding to a first image is shown;

[0034] Figure 7 According to an embodiment of this application, a schematic diagram of a lightmap coordinate system corresponding to a lightmap is shown;

[0035] Figure 8According to an embodiment of this application, a flowchart illustrating an implementation method for illumination data fitting is shown;

[0036] Figure 9 According to an embodiment of this application, a schematic diagram is shown for determining the associated light map data corresponding to a vertex pixel;

[0037] Figure 10 According to an embodiment of this application, a flowchart illustrating an implementation method for fitting data writing is shown;

[0038] Figure 11 According to an embodiment of this application, a schematic flowchart of an image processing method is shown;

[0039] Figure 12A According to an embodiment of this application, a schematic diagram is shown of an implementation method that replaces the original light map data with the color data corresponding to the vertex pixels.

[0040] Figure 12B According to an embodiment of this application, a comparative schematic diagram is shown showing the rendering effects of an image processing method using existing technology and an image processing method mentioned in the embodiment of this application;

[0041] Figure 13 According to an embodiment of this application, a schematic diagram of the hardware structure of an electronic device is shown. Detailed Implementation

[0042] The embodiments of this application include, but are not limited to, an image processing method, an electronic device, a storage medium, a program product, and a chip.

[0043] To facilitate understanding of the solutions mentioned in the embodiments of this application, the terminology involved in the embodiments of this application will be introduced first.

[0044] (1) Static mesh: A geometric shape composed of polygons, capable of representing various three-dimensional objects. For example, Figure 1 A schematic diagram of a first image is shown, the first image A including a static grid corresponding to a roof, the static grid being composed of multiple triangles.

[0045] (2) Polygonal facets: Components of a static mesh. Continuing as... Figure 1 As shown, the static grid corresponding to the roof in the first image A can be composed of multiple triangular facets a.

[0046] (3) Lightmap: An image that represents the lighting data of static objects in an image. For example, Figure 2 A schematic diagram is shown representing a lightmap corresponding to a first image. In this lightmap B corresponding to the first image A, the lighting data corresponding to pixels in region b is the same as the lighting data corresponding to pixels in triangular facet a.

[0047] It is understood that the image processing methods mentioned in the embodiments of this application can be applied to electronic devices. In some implementations, electronic devices may be referred to as terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. For example, electronic devices may be mobile phones, tablets, computers with wireless transceiver capabilities, smart TVs, wearable devices, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless charging in smart cities, wireless terminals in smart homes, and other similar devices.

[0048] The image processing method mentioned in the embodiments of this application can be applied to rendering static objects such as scene buildings, terrain features, and props in game scenes.

[0049] To address the aforementioned issue of poor image rendering quality, this application provides an image processing method. In this method, when rendering a first image, the electronic device can determine the first lighting map data of each vertex pixel based on the coordinates of each vertex pixel in the lighting map coordinate system corresponding to the first image, where the lighting map coordinate system is the coordinate system corresponding to the first lighting map of the first image. The electronic device can then determine the second lighting map data of non-vertex pixels based on the first lighting map data of each vertex pixel. Furthermore, the electronic device can render the first image based on the first lighting map data of the vertex pixels and the second lighting map data of the non-vertex pixels. Thus, by determining the lighting map data of each pixel based on the first lighting map data of the vertex pixels, instead of directly reading the lighting map data of each pixel from the first lighting map corresponding to the first image, the dependence on memory bandwidth can be reduced, the speed of acquiring lighting map data can be improved, and thereby the quality of image rendering can be enhanced.

[0050] As can be understood, as mentioned above, the image processing method mentioned in the embodiments of this application can be applied to scenes that render static objects such as scene buildings, terrain features and props in game scenes.

[0051] like Figure 3 As shown, during the game development phase (i.e., game development state), game developers can set up a game engine editor 310 and a database 320 on an electronic device. The game engine editor 310 may include a plugin 330, which may include an image processing system 340. The image processing system 340 can be used to determine the first lightmap data of vertex pixels during the game packaging phase. In some implementations, the image processing system 340 may include a scene data extraction module 301, a lighting data fitting module 302, and a fitted data writing module 303.

[0052] The scene data extraction module 301 can be used to traverse the static mesh on all static objects in the first image and extract the coordinates of the vertex pixels of each triangular facet in the static mesh in the light map coordinate system corresponding to the first light map.

[0053] The lighting data fitting module 302 can be used to determine the first lighting map data of the vertex pixels in the triangular facet based on the coordinates of the vertex pixels in the first lighting map coordinate system.

[0054] The fitting data writing module 303 can be used to determine the color data of the vertex pixels in the triangular facet based on the first lightmap data of the vertex pixels in the triangular facet. During the rendering of the first image, the graphics processor can determine the color data of the non-vertex pixels in the triangular facet based on the color data of the vertex pixels. The color data of the vertex pixels and the color data of the non-vertex pixels in the triangular facet can be used to render the first image. For example, a static global illumination effect can be rendered on the first image to change the brightness of static objects in the first image.

[0055] In some other implementations, the scene data extraction module 301 can also be used to traverse the static meshes on all static objects in the first image and extract the light map data of each pixel in each triangular facet of the static mesh from the first light map corresponding to the first image.

[0056] The lighting data fitting module 302 can also be used to train a linear regression model based on the lighting map data of each pixel in the triangular facet. This linear regression model can represent the correspondence between the coordinates in the lighting map coordinate system of the first lighting map and the lighting map data.

[0057] Database 320 can store the color data of vertex pixels in the static mesh of all static objects in the game scene (this vertex pixel color data is also called static mesh data or the mesh vertex color data of the static mesh).

[0058] The following is combined with Figure 4 ,right Figure 3 The scene data extraction module 301 will be described in detail. Figure 4 The diagram illustrates a flowchart of a scene data extraction implementation method, which can be executed by the scene data extraction module 301. For example, the scene data extraction implementation method may include:

[0059] S401: Traverse the static mesh on all static objects in the first image.

[0060] In some implementations, the first image can be an image from the game scene, which may include multiple static objects, such as scene buildings, terrain features, and props.

[0061] S402: Extract the lightmap information corresponding to the static mesh.

[0062] It is understandable that in some implementations, the first image may have a corresponding lightmap, which may include the lightmap data of each pixel in each triangle facet of the static mesh on all static objects in the first image. In other implementations, the first image may have multiple corresponding lightmaps, each of which may include the lightmap data of each pixel in each triangle facet of the static mesh on some static objects in the first image.

[0063] Furthermore, in some implementations, the resolution of the first image and the resolution of the corresponding first lightmap can be different. Therefore, the pixel position of a point on a static object in the first lightmap is different from the pixel position of that point in the first image. For example, ... Figure 5 As shown, the pixel position corresponding to a point on the roof in the first image is... Figure 5 The position shown as s in (a) corresponds to the pixel position in the first lightmap. Figure 5 The position is shown as s' in (b). Thus, the scene data extraction module 301 can extract the lightmap information corresponding to the static mesh on each static object. This lightmap information may include information such as texture index, resolution, scaling, and offset.

[0064] S403: Extract the coordinates of the vertex pixels of each triangular facet in the static mesh in the image coordinate system corresponding to the first image.

[0065] It can be understood that the coordinates of the vertex pixels in each triangular facet within the image coordinate system corresponding to the first image can be an example of the first coordinate system. For example... Figure 6 As shown, the image coordinate system corresponding to the first image can be a coordinate system established with the lower left corner of the first image as the origin, the first direction x (parallel to the right of the paper) as the first direction x, and the second direction y (parallel to the top of the paper) as the second direction y. The first direction x and the second direction y are perpendicular to each other, and the unit length of both the first direction x and the second direction y is a pixel. Thus, the scene data extraction module 301 can determine which row and column of the vertex pixel in each triangular facet of the static mesh in the first image, thereby determining the coordinates (x, y) of the vertex pixel in each triangular facet of the static mesh in the image coordinate system corresponding to the first image. i y i ).

[0066] S404: Calculate the coordinates of the vertex pixel in the lightmap coordinate system corresponding to the first lightmap.

[0067] In some implementations, the coordinates of a vertex pixel in the lightmap coordinate system corresponding to the first lightmap can be an example of the second coordinate system. For example... Figure 7 As shown, the coordinate system corresponding to the first lightmap can be a coordinate system established with the lower left corner of the first lightmap as the origin, the third direction u (parallel to the right of the paper) as the right direction, and the fourth direction v (parallel to the top of the paper) as the left direction. The third direction u and the fourth direction v are perpendicular to each other, and the unit length of both the third direction u and the fourth direction v is pixels. Thus, the scene data extraction module 301 can use the lightmap information corresponding to the static mesh, and the coordinates (x, y, y) of the vertex pixels in each triangle of the static mesh in the image coordinate system corresponding to the first image, to... i y i ), calculate the coordinates (u) of the vertex pixel in the lightmap coordinate system corresponding to the first lightmap. i v i ).

[0068] For example, the scene data extraction module 301 can calculate the coordinates of a vertex pixel in the lightmap coordinate system corresponding to the first lightmap based on information such as texture resolution, scaling, and offset in the lightmap information, as well as the coordinates of the vertex pixel in the image coordinate system corresponding to the first image. For example, the scene data extraction module 301 can determine the coordinates of the vertex pixel in the lightmap coordinate system corresponding to the first lightmap using the following formula:

[0069] (u i v i )=[(x i y i[(scale + offset) * resolution]

[0070] Among them, (u i v i (x) can represent the coordinates of a vertex pixel in each triangle of a static mesh within the coordinate system of the first lightmap, i.e., the row and column number of the vertex pixel in the first lightmap of each triangle of the static mesh. i y i ) can represent the coordinates of the vertex pixels of each triangle in the static mesh in the image coordinate system corresponding to the first image, scale can represent the scaling amount, offset can represent the offset amount, and resolution can represent the resolution.

[0071] S405: Using each triangular facet as a unit, scan line by line according to the coordinates of the three vertex pixels in the first lightmap coordinate system to obtain the lightmap data of each pixel in each triangular facet.

[0072] It can be understood that the pixels in each triangular facet can include vertex pixels and non-vertex pixels. Both vertex and non-vertex pixels possess lightmap data. That is, the lightmap data of a pixel can include both vertex and non-vertex pixel lightmap data.

[0073] In some implementations, the scene data extraction module 301 can extract the light map data of all pixels in the triangular facets included in the first image by taking each triangular facet as a unit and obtaining the coordinates of the three vertex pixels in the first light map's corresponding light map coordinate system.

[0074] It is understood that S401 to S405 shown above are merely illustrative of an embodiment of this application. In actual applications, more or fewer steps may be included. For example, the scene data extraction module 301 may execute S401 to S404, and the scene data extraction module 301 may also execute S405. This embodiment of the application does not impose specific limitations. Furthermore, the execution order of S401 to S405 shown above is also illustrative of an embodiment of this application. In actual applications, other execution orders may be followed. For example, S401 to S404 may be executed sequentially, and S401 and S405 may be executed in parallel.

[0075] The following is combined with Figure 8 ,right Figure 3 The illumination data fitting module 302 in the module will be introduced. For example... Figure 8The diagram illustrates a flowchart of an implementation method for illumination data fitting, which can be executed by the illumination data fitting module 302. For example, the implementation method for illumination data fitting may include:

[0076] S801: Obtain the light map data of all pixels within the triangular facet.

[0077] It can be understood that after the scene data extraction module 301 obtains the lighting map data of all pixels in the triangular facets included in the first image, the scene data extraction module 301 can output the lighting map data of all pixels in the triangular facets to the lighting data fitting module 302. Here, the pixels in the triangular facets include vertex pixels and non-vertex pixels. During the training of the linear regression model to be trained, the lighting map data of the vertex pixels can be referred to as the labeled lighting map data of the vertex pixels.

[0078] S802: Classify the lightmap data.

[0079] In some implementations, the lighting data fitting module 302 can reclassify the lighting map data of all pixels in all triangles based on the vertex pixels in a triangle. For example, the lighting data fitting module 302 can determine at least four triangles connected to a vertex pixel in a triangle based on the position of the triangle in the first image, and determine the lighting map data of all pixels in at least four triangles as the associated lighting map data of that vertex pixel.

[0080] It is understood that the associated lightmap data of a vertex pixel may include the lightmap data of that vertex pixel, the lightmap data of all non-vertex pixels in the triangle, and the lightmap data of all non-vertex pixels in at least four triangles connected to that vertex pixel.

[0081] For example, such as Figure 9 As shown, for triangle a1, since the vertex pixel p in triangle a1 is also a vertex pixel in triangle a2, triangle a3, triangle a4, triangle a5, and triangle a6, the lighting data fitting module 302 can determine the lighting map data of the non-vertex pixels in triangle a2, triangle a3, triangle a4, triangle a5, and triangle a6 as the associated lighting map data of vertex pixel p.

[0082] The embodiments of this application extract the associated lighting map data of vertex pixels on a triangular facet basis, rather than extracting the associated lighting map data of vertex pixels according to the content represented by the lighting map data in some classification schemes. This not only improves the accuracy of the linear regression model, but also does not need to distinguish the type of the first image, making it more versatile.

[0083] S803: Train the linear regression model to be trained based on the classified light map data.

[0084] In some implementations, the lighting data fitting module 302 can train a linear regression model based on the coordinates of pixels in the lighting map coordinate system corresponding to the first lighting map and the pixel's lighting map data in the classified lighting map data. The linear regression model can represent the correspondence between the coordinates in the lighting map coordinate system of the first lighting map and the lighting map data.

[0085] For example, the lighting data fitting module 302 can determine the linear regression model to be trained based on the coordinates of non-vertex pixels in the lighting map coordinate system corresponding to the first lighting map and the lighting map data of non-vertex pixels. Then, the lighting data fitting module 302 can determine the predicted lighting map data of vertex pixels using the linear regression model to be trained and the coordinates of vertex pixels in the lighting map coordinate system corresponding to the first lighting map. Furthermore, the lighting data fitting module 302 can adjust the parameters of the linear regression model to be trained based on the labeled lighting map data and the predicted lighting map data of vertex pixels until the adjusted linear regression model meets the training stopping condition, thus obtaining the trained linear regression module. The training stopping condition can be that the difference between the labeled lighting map data and the predicted lighting map data of vertex pixels is within a preset difference range, or it can be that the number of training iterations exceeds a training iteration threshold.

[0086] S804: Obtain the fitted lightmap data of the vertex pixels in the triangular facet.

[0087] In some implementations, after obtaining the trained linear regression model, the lighting data fitting module 302 can determine the fitted lighting map data (i.e., the first lighting map data of the vertex pixels) of the triangular facet based on the coordinates of the pixels in the first lighting map coordinate system. Then, based on the fitted lighting map data of the vertex pixels and the lighting map data of the non-vertex pixels, the lighting map is reconstructed to visualize the fitting results, thereby improving the accuracy of the fitting results. Furthermore, the lighting data fitting module 302 can evaluate the fitting results between the reconstructed lighting map and the original lighting map (i.e., the first lighting map), for example, by using the mean squared error (MSE) fitting evaluation method.

[0088] It is understandable that after determining the fitted light map data of the vertex pixels, the light data fitting module 302 can generate the index number of each vertex pixel, and then obtain the first light map data of the corresponding vertex pixel based on the index number of the vertex pixel.

[0089] S805: Pack and fit the lightmap data and return it to the engine.

[0090] In some implementations, if the fitting result meets the fitting conditions, such as the mean square error of the same pixel's light map data in the restored light map and the original light map being within a preset mean square error range, the light data fitting module 302 can return the fitted light map data of the pixels in the triangular facet to the game engine.

[0091] Thus, by employing machine learning methods, taking vertex pixels as the object, and based on the associated lightmap data of vertex pixels, feature data that can fit the static global illumination effect is learned.

[0092] It is understood that S801 to S805 shown above are only one illustration of the embodiments of this application. In actual application, more or fewer steps may be included. For example, the illumination data fitting module 302 may execute S801 to S803, or it may execute S801, S802, S803 and S805, or it may execute S801 to S804. The embodiments of this application are not specifically limited.

[0093] The following is combined with Figure 10 ,right Figure 3 The fitting data writing module 303 will be introduced here. For example... Figure 10The diagram illustrates a flowchart of a method for writing fitting data, which can be executed by the fitting data writing module 303. For example, the implementation of this illumination data fitting method may include:

[0094] S1001: Traverse the static mesh on all static objects in the first image.

[0095] It is understandable that S1001 is essentially the same as S401. The specific implementation of S1001 can be referred to S401, and will not be elaborated here.

[0096] S1002: Obtain the index number of each vertex pixel in the static mesh.

[0097] As can be understood, as mentioned above, after determining the light map data of the pixels, the light data fitting module 303 can generate the index number of each vertex pixel.

[0098] S1003: Obtain the first light map data of each vertex pixel from the lighting data fitting module based on the index number of each vertex pixel.

[0099] It is understandable that the fitting data writing module 303 can obtain the first lighting map data of the corresponding vertex pixel based on the vertex pixel index number.

[0100] S1004: Determine the color data of the vertex pixel based on the first light map data of the vertex pixel.

[0101] In some implementations, the fitting data writing module 303 can determine the first lightmap data of each vertex pixel as the vertex pixel's color data, thereby writing the lightmap data into the vertex pixel's color channel. The vertex pixel's color data can be used to determine the color data of non-vertex pixels. Thus, when rendering the first image subsequently, only the vertex pixel's color data, i.e., the static mesh data or the mesh vertex color data of the static mesh volume, needs to be used. By using the mesh vertex color data to store the vertex pixel's lightmap data, the amount of data processing required during subsequent rendering of the first image can be reduced.

[0102] The following is combined with Figure 11 The image processing methods mentioned in the embodiments of this application will be described in detail. For example... Figure 11 The diagram illustrates a flowchart of an image processing method. This image processing method can be used to process a first image in an electronic device. The first image may include multiple polygonal patches, which can be distinguished by using terms such as first polygonal patch, second polygonal patch, etc.

[0103] In some implementations, the image processing method may include:

[0104] S1101: Obtain the first lighting map data of each vertex pixel in the first image corresponding to the first lighting map in the first polygon patch.

[0105] In some implementations, the electronic device can acquire first lightmap data of each vertex pixel in a first lightmap corresponding to a first image in a multi-polygon patch.

[0106] It is understood that electronic devices may include rendering services, such as Figure 3 The game engine editor 310 mentioned in the text. Therefore, in some other implementations, the rendering service can obtain the first lightmap data of each vertex pixel in the first image corresponding to the first lightmap in the first polygon patch.

[0107] Understandable, continue as Figure 3 As shown, the image processing system 340 may include a scene data extraction module 301 and a lighting data fitting module 302. Therefore, in another implementation, the scene data extraction module 301 can acquire the first lighting map data of each vertex pixel in a plurality of polygon patches in the first lighting map corresponding to the first image.

[0108] It is understandable that a polygonal facet can include multiple vertex pixels; for example, a first polygonal facet can include a first vertex pixel. The following section uses the first vertex pixel as an example to introduce how to obtain the first lightmap data of the vertex pixel in the first lightmap.

[0109] The scene data extraction module 301 can obtain the first vertex pixel in, for example Figure 6 The first coordinate in the image coordinate system corresponding to the first image shown is used to determine the first vertex pixel's position in the image coordinate system. Figure 7 The second coordinate in the lightmap coordinate system corresponding to the first lightmap shown is then used. Furthermore, the lighting data fitting module 302 can determine the first lightmap data of the first vertex pixel based on the correspondence between the coordinates in the first lightmap's lightmap coordinate system and the lightmap data, as well as the second coordinate. The method for determining the first lightmap of the first vertex pixel can be referred to above; to avoid repetition, it will not be elaborated here.

[0110] In some methods for determining the first lightmap data of the first vertex pixel, the lighting data fitting module 302 can input the second coordinates into a linear regression model and output the first lightmap data of the first vertex pixel based on the linear regression model. The linear regression model includes the correspondence between the coordinates in the lightmap coordinate system of the first lightmap and the lightmap data. For specific training methods of the linear regression model, please refer to... Figure 8To avoid repetition, I will not go into details here.

[0111] S1102: Based on the first lighting map data, determine the second lighting map data for non-vertex pixels in multiple polygonal patches.

[0112] In some implementations, the electronic device can determine the second lighting map data of non-vertex pixels in a plurality of polygonal patches based on the first lighting map data of each vertex pixel in the plurality of polygonal patches. For example, the electronic device can perform interpolation processing on the first lighting map data of each vertex pixel in the plurality of polygonal patches to determine the second lighting map data of non-vertex pixels in the plurality of polygonal patches.

[0113] It is understood that electronic devices may include graphics processors. Therefore, in some implementations, the graphics processor can determine second lighting map data for non-vertex pixels in a plurality of polygonal patches based on first lighting map data for each vertex pixel in the polygonal patches. For example, the graphics processor can interpolate the first lighting map data for each vertex pixel in the plurality of polygonal patches to determine the second lighting map data for non-vertex pixels in the polygonal patches.

[0114] In other implementations, the electronic device may include a central processing unit (CPU), which may also determine second lightmap data for non-vertex pixels in the multiple polygonal patches based on first lightmap data for each vertex pixel in the multiple polygonal patches.

[0115] S1103: Render the first image based on the first light map data and the second light map data.

[0116] In some implementations, the electronic device can render the first image based on the first lightmap data of each vertex pixel in multiple polygon patches and the second lightmap data of non-vertex pixels.

[0117] As previously mentioned, electronic devices may include a graphics processor. Therefore, in some implementations, the graphics processor may render a first image based on first lightmap data of individual vertex pixels and second lightmap data of non-vertex pixels in a plurality of polygon patches.

[0118] Understandable, continue as Figure 3As shown, the image processing system 340 in the game engine editor 310 may further include a fitting data writing module 303. In some implementations, the fitting data writing module 303 can determine the color data of the vertex pixels within the multiple polygonal patches based on the first lightmap data of the vertex pixels within the multiple polygonal patches, thereby writing the first lightmap data into the color channels of the vertex pixels. In other implementations, after the fitting data writing module 303 writes the first lightmap data into the color channels of the vertex pixels, the graphics processor can interpolate the color data of the non-vertex pixels based on the color data of each vertex pixel, and render the first image based on the color data of each vertex pixel and the color data of the non-vertex pixels. For example, in the code for rendering a static global illumination effect, the code for sampling the lightmap data is canceled, and the original lightmap data is replaced with the color data corresponding to the vertex pixels.

[0119] For example, such as Figure 12A The diagram illustrates a method for replacing the original lightmap data with the color data corresponding to vertex pixels. This implementation may include:

[0120] S1201: Responds to user actions that modify the lightmap sampling code and to user actions that enable plugins in the game engine editor interface.

[0121] It is understandable that electronic devices can respond to user actions that modify lightmap sampling code according to instructions, and electronic devices can respond to user actions that open plugins in the game engine editor interface.

[0122] S1202: Responds to the user's action of clicking the game packaging button in the game engine editor interface.

[0123] It is understandable that electronic devices can respond to the user's action of clicking the game packaging button on the game engine editor interface and enter the game packaging stage.

[0124] S1203: When the game is launched and packaged, the static global illumination fitting capability is triggered.

[0125] It is understandable that, when entering the game packaging stage, the first lightmap data of the vertex pixels of the triangular facets in the static mesh of all static objects in the first image can be obtained and stored in the database, thereby triggering the static global illumination fitting capability.

[0126] S1204; Write the fitted lightmap data into the mesh vertex data of the static mesh, and continue to complete the remaining packaging process.

[0127] It is understandable that in the code that renders static global illumination effects, the electronic device can respond to the user's code that cancels the original sampling of lightmap data, and replace the original lightmap data with the color data corresponding to the vertex pixels, so as to write the fitted lightmap data into the mesh vertex data of the static mesh and continue to complete the remaining packaging process.

[0128] In this embodiment, the lightmap data of each pixel is determined based on the first lightmap data of the vertex pixels, instead of directly reading the lightmap data of each pixel from the first lightmap. This reduces the dependence on memory bandwidth, thereby improving the quality of image rendering. Furthermore, it saves the instructions required for sampling the lightmap data in the lightmap, such as the instructions for the graphics processor to interact with memory, thus improving game performance in game scenes.

[0129] Furthermore, compared to some image processing methods that rely on cloud resources for lightmap reconstruction and rendering, this method reduces reliance on other hardware. Moreover, the image processing method mentioned in this application can be applied not only to Web3D scenes where lightmap reconstruction and rendering are performed using cloud resources, but also to scenes where static global illumination effects are rendered on images to be rendered on mobile devices, and to scenes suitable for rendering volumetric lighting.

[0130] Furthermore, the image processing method mentioned in this application embodiment does not require additional calculation of light map data using spherical harmonic parameters, which can reduce the impact on game performance in game scenes, reduce the loading time of game scenes, and accelerate the iteration speed of game scenes.

[0131] The following is combined with Figure 12B The effects of rendering a first image using existing image processing methods and using the image processing methods mentioned in the embodiments of this application are compared and explained.

[0132] like Figure 12B As shown, Figure 12B Image (a) is the first image without static global illumination effects rendered. Figure 12B (b) is a first image rendered using existing technology, such as a first image rendered from the lightmap data of each pixel in a first lightmap corresponding to the first image. Figure 12B (c) is the first image rendered using the image processing method mentioned in the embodiments of this application.

[0133] contrast Figure 12B (a) Figure 12B (b) and Figure 12BAs can be seen from (c), the brightness of static objects (e.g., rooftops) in the first image without rendering static global illumination effects is relatively dark. The brightness of the first image rendered using existing technology is slightly different from the brightness of static objects in the first image rendered using the image processing method mentioned in the embodiments of this application, but both are brighter than the brightness of static objects in the first image without rendering static global illumination effects.

[0134] It is understood that the image processing method mentioned in the embodiments of this application can be applied to electronic devices. The hardware structure of the electronic device will be described below.

[0135] Figure 13 A schematic diagram of the hardware structure of an electronic device is shown. It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of both.

[0136] like Figure 13 As shown, the electronic device may include a processor 1310, an external memory interface 1313, an internal memory 1321, a universal serial bus (USB) interface 1330, a charging management module 1340, a power management module 1341, a battery 1342, an antenna 1, an antenna 2, a mobile communication module 1350, a wireless communication module 1360, an audio module 1370, a speaker 1370A, a receiver 1370B, a microphone 1370C, a headphone jack 1370D, a sensor module 1380, buttons 1390, a motor 1391, an indicator 1392, a camera 1393, a display screen 1394, and a subscriber identification module (SIM) card interface 1395, etc. The sensor module 1380 may include a pressure sensor 1380A, a gyroscope sensor 1380B, a barometric pressure sensor 1380C, a magnetic sensor 1380D, an accelerometer sensor 1380E, a distance sensor 1380F, a proximity sensor 1380G, a fingerprint sensor 1380H, a temperature sensor 1380J, a touch sensor 1380K, an ambient light sensor 1380L, a bone conduction sensor 1380M, etc.

[0137] In some embodiments, the processor 1310 may include one or more interfaces. Interfaces may include integrated circuit interfaces, integrated circuit built-in audio interfaces, pulse code modulation (PCM) interfaces, universal asynchronous receiver / transmitter (UART) interfaces, mobile industry processor interfaces (MIPI), general-purpose input / output (GPIO) interfaces, subscriber identity module (SIM) interfaces, and / or universal serial bus (USB) interfaces, etc.

[0138] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a limitation on the structure of the electronic device. In other embodiments of this application, the electronic device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0139] The charging management module 1340 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 1340 receives charging input from the wired charger via a USB interface 1330. In some wireless charging embodiments, the charging management module 1340 receives wireless charging input via the wireless charging coil of the electronic device. While charging the battery 1342, the charging management module 1340 can also supply power to the electronic device via the power management module 1341.

[0140] The power management module 1341 connects the battery 1342, the charging management module 1340, and the processor 1310. The power management module 1341 receives input from the battery 1342 and / or the charging management module 1340, providing power to the processor 1310, internal memory 1321, display screen 1394, camera 1393, and wireless communication module 1360, etc. The power management module 1341 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 1341 may also be located within the processor 1310. In other embodiments, the power management module 1341 and the charging management module 1340 may be located in the same device.

[0141] In some implementations, the power management module 1341 can supply power to the audio module 1370 so that the audio module 1370 can switch from a power-off stopped state to a power-on working state.

[0142] Electronic devices implement display functions through a GPU, a display screen 1394, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 1394 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. The processor 1310 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0143] The display screen 1394 is used to display images, videos, etc. The display screen 1394 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include one or N displays 1394, where N is a positive integer greater than 1.

[0144] Electronic devices can achieve shooting functions through ISP, camera 1393, video codec, GPU, display 1394 and application processor.

[0145] Camera 1393 is used to capture still objects or video. An optical image of the object is generated through the lens and projected onto a photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP (Internet Service Provider) for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP (Digital Signal Processor) for processing. The DSP converts the digital image signal into image signals in standard formats such as RGB and YUV. In some embodiments, the electronic device may include one or N cameras 1393, where N is a positive integer greater than 1.

[0146] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when an electronic device is selecting a frequency, a DSP is used to perform Fourier transforms on the frequency energy.

[0147] Video codecs are used to compress or decompress digital video. Electronic devices can support one or more video codecs. This allows the electronic device to play or record video in various encoded formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0148] This application provides a computer program product that, when run on a device, causes the device to execute the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar to those of the related embodiments described above, and will not be repeated here.

[0149] This application provides a readable storage medium containing instructions that, when executed on a device, cause the device to perform the technical solutions described in the above embodiments. The implementation principle and technical effects are similar and will not be repeated here.

[0150] This application provides a chip for executing instructions. When the chip is running, it executes the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0151] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the aforementioned element.

[0152] Although this application has been illustrated and described with reference to certain embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the scope of this application.

Claims

1. An image processing method, characterized in that, For processing a first image in an electronic device, the first image comprising a plurality of polygonal patches, including: Obtain the first light map data of each vertex pixel in the plurality of polygonal patches in the first light map corresponding to the first image; Based on the first light map data, the second light map data of the non-vertex pixels in the plurality of polygonal patches is determined; The first image is rendered based on the first light map data and the second light map data.

2. The method according to claim 1, characterized in that, The first image includes a first polygonal patch, and the first polygonal patch includes a first vertex pixel. The step of obtaining the first lighting map data of each vertex pixel in the plurality of polygon patches in the first lighting map corresponding to the first image includes: Obtain the first coordinates of the first vertex pixel in the image coordinate system corresponding to the first image; Based on the first coordinates, the second coordinates of the first vertex pixel in the lightmap coordinate system corresponding to the first lightmap are determined; Based on the correspondence between the coordinates in the lightmap coordinate system and the lightmap data in the first lightmap, and the second coordinate, the first lightmap data corresponding to the first vertex pixel is determined.

3. The method according to claim 2, characterized in that, The step of determining the first lightmap data corresponding to the first vertex pixel based on the correspondence between the coordinates in the lightmap coordinate system and the lightmap data in the first lightmap, and the second coordinates, includes: The second coordinate is input into a linear regression model, and the first light map data corresponding to the first vertex pixel is output based on the linear regression model; wherein, the linear regression model includes the correspondence between the coordinates in the light map coordinate system of the first light map and the light map data.

4. The method according to any one of claims 1 to 3, characterized in that, The step of determining the second lighting map data of non-vertex pixels in the plurality of polygon patches based on the first lighting map data includes: Interpolation processing is performed on the first light map data of each vertex pixel in the plurality of polygonal patches to determine the second light map data of the non-vertex pixels in the plurality of polygonal patches.

5. The method according to claim 2 or 3, characterized in that, The electronic device includes rendering services and a graphics processor. The rendering service obtains the light map data of each vertex pixel in the plurality of polygon patches in the first light map corresponding to the first image; The graphics processor determines the second lighting map data of the non-vertex pixels in the plurality of polygon patches based on the first lighting map data; The graphics processor renders the first image based on the first light map data and the second light map data.

6. The method according to claim 5, characterized in that, The rendering service includes a scene data extraction module and a lighting data fitting module. The scene data extraction module obtains the first coordinates of the first vertex pixel in the image coordinate system corresponding to the first image; The scene data extraction module determines the second coordinates of the first vertex pixel in the lightmap coordinate system corresponding to the first lightmap based on the first coordinates; The lighting data fitting module determines the first lighting map data corresponding to the first vertex pixel based on the correspondence between the coordinates in the lighting map coordinate system corresponding to the first lighting map and the lighting map data, as well as the second coordinates.

7. The method according to claim 6, characterized in that, The rendering service also includes a fitting data writing module, and the method further includes: The fitting data writing module determines the color data of the vertex pixels within the plurality of polygonal facets based on the first lighting map data of the vertex pixels within the plurality of polygonal facets.

8. An electronic device, characterized in that, Includes: a memory for storing instructions executed by one or more processors of the electronic device. And a processor, which is one or more processors of the electronic device, for performing the image processing method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that, The readable storage medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the image processing method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by an electronic device, enable the electronic device to perform the image processing method as described in any one of claims 1 to 7.

11. A chip, characterized in that, The chip includes a processor coupled to a memory for executing a computer program or instructions stored in the memory, such that the chip implements the image processing method according to any one of claims 1 to 7.

Images