Image processing method and device, electronic equipment and readable storage medium

Abstract

The invention discloses an image processing method and device, electronic equipment and a readable storage medium, and belongs to the technical field of electronic equipment. The method is applied to an XR device and comprises the steps that first coordinate information of a first vertex is acquired, and the first vertex is a vertex of a second vertex of a three-dimensional object projected to a screen of the XR device; based on the first coordinate information, sampling is carried out in an anti-distortion texture to obtain second coordinate information, the anti-distortion texture is used for carrying out anti-distortion on the texture of the three-dimensional object, and at least one piece of coordinate information or at least one piece of radius information is stored in the anti-distortion texture; and based on the second coordinate information, sampling and drawing the texture of the three-dimensional object to obtain a rendered image corresponding to the three-dimensional object.

Description

Technical Field

[0001] The present application belongs to the technical field of electronic equipment, and specifically relates to an image processing method, device, electronic equipment and readable storage medium. Background Art

[0002] In Extended Reality (XR) technology, in order to achieve a wider field of view, the image on the XR device screen will be magnified by a lens before entering the human eye. The screen image will cause pincushion distortion after passing through the lens, so the rendered image will be dedistorted into a barrel shape before being displayed on the screen, so that the image after passing through the lens is corrected to a normal shape.

[0003] At present, the above anti-distortion operations usually include grid-based anti-distortion and polynomial-based anti-distortion. In some scenes, the position of the vertices of an object in the rendered image projected onto the two-dimensional screen does not necessarily fall on the vertices of the pre-processed distorted mesh, making it impossible to use the pre-processed distorted mesh to perform anti-distortion processing on the object. Therefore, grid-based anti-distortion cannot be used, and only polynomial-based anti-distortion can be used.

[0004] However, polynomial-based anti-distortion requires anti-distortion processing for each mesh vertex of a 3D object. When there are many vertices of a 3D object, it will affect the rendering performance. Moreover, the anti-distortion polynomial used in polynomial-based anti-distortion can often reach 7 or even higher, resulting in a large amount of calculation, which leads to poor anti-distortion performance. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an image processing method, device, electronic device and readable storage medium, which can improve the performance of anti-distortion.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising: obtaining first coordinate information of a first vertex, the first vertex being the vertex of a second vertex of a three-dimensional object projected onto a screen of an XR device; based on the first coordinate information, sampling in an anti-distortion texture to obtain second coordinate information, the anti-distortion texture being used to anti-distort the texture of the three-dimensional object, the anti-distortion texture storing at least one coordinate information or at least one radius information; based on the second coordinate information, sampling and drawing the texture of the three-dimensional object to obtain a rendered image corresponding to the three-dimensional object.

[0007] In the second aspect, an embodiment of the present application provides an image processing device, which includes an acquisition module, a sampling module and a drawing module; the acquisition module is used to acquire first coordinate information of a first vertex, where the first vertex is a vertex where the second vertex of the three-dimensional object is projected onto the screen of the XR device; the sampling module is used to sample in an anti-distortion texture based on the first coordinate information acquired by the acquisition module to obtain second coordinate information, where the anti-distortion texture is used to anti-distort the texture of the three-dimensional object, and at least one coordinate information or at least one radius information is stored in the anti-distortion texture; the drawing module is used to sample and draw the texture of the three-dimensional object based on the second coordinate information sampled by the sampling module to obtain a rendered image corresponding to the three-dimensional object.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program / program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0012] In an embodiment of the present application, the first coordinate information of the first vertex is obtained, and the first vertex is the vertex of the second vertex of the three-dimensional object projected onto the screen of the XR device; based on the first coordinate information, the anti-distortion texture is sampled to obtain the second coordinate information, and the anti-distortion texture is used to anti-distort the texture of the three-dimensional object, and the anti-distortion texture stores at least one coordinate information or at least one radius information; based on the second coordinate information, the texture of the three-dimensional object is sampled and drawn to obtain a rendered image corresponding to the three-dimensional object. Through this scheme, when the three-dimensional object is projected onto the screen of the XR device, the coordinate information of the point projected onto the screen by the vertex of the three-dimensional object can be directly sampled in the anti-distortion texture to obtain the coordinate information of the point after anti-distortion, and then the image is drawn and rendered based on the coordinate information after anti-distortion, without using polynomial-based anti-distortion to perform anti-distortion processing on the vertices of the three-dimensional object, so that real-time calculation by high-order polynomials can be avoided, thereby greatly reducing the amount of calculation and improving the performance of anti-distortion. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a schematic diagram of a non-antidistortion grid in the related art;

[0014] Figure 2 is a schematic diagram of an anti-distortion preprocessing grid in the related art;

[0015] Figure 3 It is a schematic diagram of the related art that cannot use the anti-distortion grid for anti-distortion processing;

[0016] Figure 4 It is one of the flow charts of the image processing method provided in the embodiment of the present application;

[0017] Figure 5 This is the second flowchart of the image processing method provided in the embodiment of the present application;

[0018] Figure 6 This is the third flowchart of the image processing method provided in the embodiment of the present application;

[0019] Figure 7 This is the fourth flowchart of the image processing method provided in the embodiment of the present application;

[0020] Figure 8 is a schematic diagram of the placement relationship between the first lens and the screen in the image processing method provided in an embodiment of the present application;

[0021] Fig. 9 It is a schematic diagram of storing at least one coordinate information of the anti-distortion texture in the image processing method provided by an embodiment of the present application;

[0022] Fig.10It is a schematic diagram of storing at least one radius information in the anti-distortion texture in the image processing method provided in an embodiment of the present application;

[0023] Fig.11 is a schematic diagram of an image processing device provided in an embodiment of the present application;

[0024] Fig.12 is a schematic diagram of an electronic device provided in an embodiment of the present application;

[0025] Fig.13 It is a hardware schematic diagram of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0027] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0028] The term "indication" in this application can be a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, operations to be performed, or request results in the sent indication; an indirect indication can be understood as the receiver determining the corresponding information according to the indication sent by the sender, or making a judgment and determining the operation to be performed or the request result according to the judgment result.

[0029] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0030] The image processing method, device, electronic device and readable storage medium provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0031] In Extended Reality (XR) technology, in order to achieve a wider field of view, the image on the XR device screen will be magnified by a lens before entering the human eye. The screen image will cause pincushion distortion after passing through the lens, so the rendered image will be dedistorted into a barrel shape before being displayed on the screen, so that the image after passing through the lens is corrected to a normal shape.

[0032] Currently, the above anti-distortion operation generally includes mesh-based anti-distortion and polynomial-based anti-distortion.

[0033] 1. Grid-based anti-distortion and its characteristics

[0034] Grid-based anti-distortion can pre-generate a grid distortion grid through the parameters of lens optical calibration. The vertex positions of each row and column evenly distributed on the grid have been reversed, but the UV (texture coordinates) remain the original evenly distributed values. Then the entire XR 3D scene is rendered off-screen, which is equivalent to mapping the 3D scene into a 2D image. After that, this 2D image is drawn on the pre-generated grid distortion grid to complete the anti-distortion processing.

[0035] 2. Polynomial-based anti-distortion and its characteristics

[0036] The anti-distortion principle based on polynomials is to fit the discrete distortion vertices calibrated by the optical lens into a curve, and the curve is expressed by a polynomial. The polynomial has two forms: forward and reverse. The forward form can input the coordinates of the target point entering the human eye through the optical machine and output the corresponding coordinates on the screen; the reverse form can input the corresponding coordinates on the screen and output the coordinates of the target point entering the human eye through the optical machine.

[0037] However, there are some specific cases where it is not possible to use pre-distorted meshes.

[0038] For example, for some objects in a 3D scene that need to have their definition improved (such as a 3D user interface, or a plane, curved surface, cylinder, cube, or sphere for playing a video, etc.), in order to improve the definition, you cannot first render the entire scene into an image by the first rendering, and then render the image a second time to display it, because the rendering quality will be reduced by two renderings. This situation deals with a single object in the scene, not a 2D image that covers the entire screen. The position of the object's vertices projected onto the 2D screen is arbitrary and does not necessarily fall on the vertices of the pre-processed mesh distortion mesh. Therefore, the pre-processed mesh distortion mesh cannot be used to perform anti-distortion processing on the object.

[0039] For example, some requirements require that mesh vertices be moved before dedistortion, and these movements are arbitrary, and then dedistortion can be performed. In this case, the pre-dedistorted mesh cannot be used, and the mesh vertices are irregular after arbitrary movement, and the pre-processed mesh dedistortion mesh cannot be used for dedistortion processing.

[0040] In the above cases, only polynomial-based anti-distortion can be used. The polynomial is used to perform real-time calculations on the vertices projected onto the two-dimensional screen to calculate the position to which the anti-distortion should be applied. Then, the pixel shader performs sampling and finally the object is displayed on the screen to complete the anti-distortion processing. The following formula (1) is a typical anti-distortion polynomial formula:

[0041] d'=f(r)=p1×d^7+p2×d^6+p3×d^5+p4×d^4+p5×d^3+p6×d^2+p7×d+p8; (1)

[0042] Among them, d is the distance from the original position of the projected vertex of the object projected onto the two-dimensional screen without anti-distortion processing to the center of distortion, d' is the distance from the new position of the projected vertex of the object projected onto the two-dimensional screen after anti-distortion processing to the center of distortion, and p1~p8, q1~q8 are some constant coefficients. It can be seen that the distortion is radial distortion, that is, the points on the same circle from the center of distortion are also on the same concentric circle after distortion, so the input and output of the above polynomial are both distance values. After calculating the new distance through the polynomial formula, combined with the position before distortion, the two-dimensional coordinate value of the vertex after distortion can be calculated.

[0043] As can be seen from the above, for grid-based anti-distortion, since the grid is sparse, the UV of the screen pixel is automatically interpolated and generated by the rendering pipeline during the rasterization stage, so the entire distortion process is relatively simple. However, this method uses a regular grid with a certain spacing, which is generally used in the secondary rendering stage in XR, that is, it is aimed at the rendered image after the three-dimensional scene rendering has been completed. The image will be rendered on a grid that is full and parallel to the screen. It is a secondary processing anti-distortion of the entire scene rendering result, so this method has certain limitations.

[0044] The main problem with using mesh-based anti-distortion is that the vertex positions for anti-distortion are arbitrary and unknown. The pre-processed regular distribution grid anti-distortion grid is regular and discrete. For example, the non-anti-distortion grid is Figure 1 As shown in the figure, the mesh after anti-distortion preprocessing is as follows Figure 2 The processed vertices are not necessarily consistent with the vertices of the mesh anti-distortion mesh, but may fall on positions within the mesh, for example, Figure 3 As shown in the figure, the vertices A, B, and C of the triangle object mesh all fall in the middle of the anti-distortion mesh, and the anti-distortion information of the anti-distortion mesh vertices cannot be obtained. Therefore, the new position of the vertex after distortion cannot be obtained by preprocessing the regularly distributed mesh, and a polynomial must be used to calculate the distortion position in real time.

[0045] However, polynomial-based anti-distortion requires anti-distortion processing for each mesh vertex of a 3D object. When the number of vertices of a 3D object is large, it will affect the rendering performance. For example, for a rectangular plane object, although its mesh can be formed by 4 vertices, the distorted image obtained by performing polynomial calculations on only 4 vertices has low distortion granularity, low distortion accuracy and poor effect; therefore, it is necessary to increase the number of vertices of the rectangular plane to make the mesh as dense as possible, but this will greatly increase the amount of polynomial calculations, and the number of anti-distortion polynomials can often reach 7 or even higher. This results in poor anti-distortion performance.

[0046] In order to solve the above problems, the embodiments of the present application provide an image processing method, an apparatus, an electronic device and a readable storage medium. The image processing method provided in the embodiments of the present application can be applied to the anti-distortion scenario of XR devices.

[0047] Exemplarily, the coordinate information 1 of the vertex a of the three-dimensional object projected onto the screen of the XR device can be obtained first; then based on the coordinate information 1, sampling is performed in the anti-distortion texture to obtain coordinate information 2, the anti-distortion texture is used to anti-distort the texture of the three-dimensional object, and the anti-distortion texture stores at least one coordinate information or at least one radius information; finally, based on the coordinate information 2, the texture of the three-dimensional object is sampled and drawn to obtain a rendered image corresponding to the three-dimensional object. Through this scheme, when the three-dimensional object is projected onto the screen of the XR device, the coordinate information of the point projected onto the screen by the vertex of the three-dimensional object can be directly sampled in the anti-distortion texture to obtain the coordinate information of the point after anti-distortion, and then the image drawing and rendering is performed based on the anti-distortion coordinate information, without using polynomial-based anti-distortion to perform anti-distortion processing on the vertices of the three-dimensional object, so that real-time calculation by high-order polynomials can be avoided, thereby greatly reducing the amount of calculation and improving the performance of anti-distortion.

[0048] It should be noted that the image processing method provided in the embodiments of the present application may be executed by an image processing device, an electronic device, or a functional module in an electronic device, etc. In some embodiments of the present application, an XR device executing the image processing method is taken as an example to illustrate the image processing method provided in the embodiments of the present application.

[0049] Figure 4 FIG. 1 is a flow chart showing an image processing method provided by an embodiment of the present application. Figure 4 As shown, the image processing method provided in the embodiment of the present application may include the following steps 101 to 103.

[0050] Step 101: The XR device obtains first coordinate information of a first vertex.

[0051] The first vertex is the vertex of the second vertex of the three-dimensional object projected onto the screen of the XR device.

[0052] Optionally, in an embodiment of the present application, the above-mentioned three-dimensional object can be any object in a three-dimensional scene captured by the XR device, for example, any object such as a tree, a person, a car or a building.

[0053] Optionally, in an embodiment of the present application, the second vertex may be a mesh vertex of the three-dimensional object.

[0054] Optionally, in an embodiment of the present application, the second vertex may be projected onto the screen after undergoing a series of spatial transformations such as model transformation, view transformation, and projection transformation.

[0055] Optionally, in an embodiment of the present application, the XR device may first project the second vertex onto the screen in a vertex shader of a rendering pipeline of the XR device, and then obtain the first coordinate information of the first vertex projected onto the screen by the second vertex.

[0056] Optionally, in the embodiment of the present application, the above-mentioned screen may be a two-dimensional screen.

[0057] Optionally, in an embodiment of the present application, the first coordinate information may be coordinate information in a clipping space coordinate system, for example, the first coordinate information is (x, y).

[0058] It should be noted that the clipping space coordinate system, also known as the clipping coordinate system, is an important concept in computer graphics. It means that when drawing a graphic, the graphic is reduced or enlarged at a certain ratio and confined to a specific area so that it can be displayed on the screen.

[0059] Step 102: The XR device samples the anti-distortion texture based on the first coordinate information to obtain second coordinate information.

[0060] The anti-distortion texture is used to anti-distort the texture of a three-dimensional object, and the anti-distortion texture stores at least one coordinate information or at least one radius information.

[0061] It can be understood that the second coordinate information is the coordinate information of the first vertex after the three-dimensional object is dedistorted.

[0062] Optionally, in an embodiment of the present application, the anti-distortion texture may be created by the XR device and stored in the XR device, so that the anti-distortion texture may be directly used when an anti-distortion operation needs to be performed.

[0063] The specific method for creating the above-mentioned anti-distortion texture for the XR device will be described in detail in the following embodiments. In order to avoid repetition, it will not be described here.

[0064] Optionally, in the embodiment of the present application, the above sampling process may include interpolation.

[0065] Optionally, in the embodiment of the present application, sampling in the anti-distortion texture may specifically include: sampling according to the at least one coordinate information or the at least one radius information stored in the anti-distortion texture.

[0066] For example, taking the at least one coordinate information stored in the anti-distortion texture as an example, if the at least one coordinate information contains coordinate information corresponding to the first vertex, the coordinate information can be determined as the second coordinate information; if the at least one coordinate information does not contain coordinate information corresponding to the first vertex, sampling can be performed based on the at least one coordinate information to obtain the second coordinate information corresponding to the first vertex.

[0067] Optionally, in the embodiment of the present application, the anti-distortion texture stores the at least one coordinate information. Figure 4 ,like Figure 5 As shown, the above step 102 can be specifically implemented through the following steps 102a and 102b.

[0068] Step 102a: The XR device converts the first coordinate information into third coordinate information in a texture coordinate system.

[0069] Optionally, in an embodiment of the present application, the texture coordinate system may be a UV coordinate system, which has the lower left corner as the origin and increases to the right and upward, respectively, with a value range of 0 to 1.0.

[0070] It should be noted that UV coordinates mean that all image files are a two-dimensional plane, in which the horizontal direction is U and the vertical direction is V; through this two-dimensional UV coordinate system, any pixel on the image can be located.

[0071] For the specific method of converting the coordinate information in the clipping space coordinate system into the coordinate information in the texture coordinate system, reference may be made to the relevant description in the related art, and in order to avoid repetition, it will not be described here.

[0072] For example, assuming that the first coordinate information is (x, y), then the third coordinate information may be (U, V)=((x+1) / 2, (y+1) / 2)).

[0073] Step 102b: The XR device samples the anti-distortion texture according to the third coordinate information to obtain second coordinate information.

[0074] Optionally, in an embodiment of the present application, the XR device may interpolate the at least one coordinate information according to the third coordinate information to generate new coordinate information, and determine the second coordinate information corresponding to the third coordinate information from the generated new coordinate information.

[0075] In the embodiment of the present application, since the XR device can convert the above-mentioned first coordinate information into the above-mentioned third coordinate information in the texture coordinate system, and then sample the above-mentioned anti-distortion texture according to the third coordinate information to obtain the above-mentioned second coordinate information, the second coordinate information can be directly sampled from the coordinate information stored in the anti-distortion texture, thereby simplifying the process of obtaining the second coordinate information.

[0076] Optionally, in the embodiment of the present application, the anti-distortion texture stores the at least one radius information. Figure 4 ,like Figure 6 As shown, the above step 102 can be specifically implemented through the following steps 102c to 102f.

[0077] Step 102c: The XR device determines first radius information from the first vertex to the distortion center of the first lens in the XR device according to the first coordinate information.

[0078] Optionally, in an embodiment of the present application, the first lens may be any one of the left and right lenses of the XR device.

[0079] It should be noted that for each of the left and right lenses of the XR device, the XR device needs to execute the image processing method provided in the embodiment of the present application to obtain a rendered image of a three-dimensional object. For the sake of ease of explanation, only one of the lenses is illustrated in the embodiment of the present application.

[0080] For example, assuming that the first coordinate information is (x, y), then the first radius information is

[0081] Step 102d: The XR device converts the first radius information into second radius information in the texture coordinate system.

[0082] Optionally, in an embodiment of the present application, the texture coordinate system may be a UV coordinate system, and the second radius information may be information of a U coordinate in the UV coordinate.

[0083] For example, assuming that the first radius information is r, then the second radius information U=r / R; wherein R=1.0.

[0084] Step 102e: The XR device samples the anti-distortion texture according to the second radius information to obtain third radius information.

[0085] For a specific description of sampling in the anti-distortion texture, reference may be made to the relevant description in the above embodiment, and in order to avoid repetition, it will not be described again here.

[0086] Step 102f: The XR device determines the second coordinate information according to the first coordinate information, the first radius information, and the third radius information.

[0087] Optionally, in an embodiment of the present application, assuming that the first coordinate information is (x, y), the first radius information is r, and the third radius information is r', the XR device can determine the second coordinate information as (x', y') based on the first coordinate information, the first radius information and the third radius information, where (x', y') = (x×r' / r, y×r' / r).

[0088] In an embodiment of the present application, since the XR device can sample the anti-distortion texture based on the first radius information from the first vertex to the distortion center of the first lens in the XR device to obtain the second coordinate information, the second coordinate information can be determined when the anti-distortion texture stores the radius information, thereby streamlining the storage information of the anti-distortion texture and saving storage space of the XR device.

[0089] Step 103: The XR device samples and draws the texture of the three-dimensional object based on the second coordinate information to obtain a rendered image corresponding to the three-dimensional object.

[0090] Optionally, in an embodiment of the present application, the above-mentioned rendered image is the displayed image on the above-mentioned screen.

[0091] It should be noted that image rendering is the process of converting three-dimensional light energy transfer processing into a two-dimensional image. The scenes and entities are represented in three-dimensional form, which is closer to the real world and easier to manipulate and transform, while the display devices of graphics are mostly two-dimensional rasterized displays and dot matrix printers. The representation of N-dimensional raster and dot matrix from the representation of three-dimensional entity scenes is image rendering - that is, rasterization. The raster display can be regarded as a pixel matrix. Any graphic displayed on the raster display is actually a collection of pixels with one or more colors and grayscales.

[0092] The specific process of sampling and drawing the texture of a three-dimensional object can refer to the relevant description in the relevant technology, and in order to avoid repetition, it will not be repeated here.

[0093] Optionally, in the embodiment of the present application, combined with Figure 4 ,like Figure 7 As shown, the above step 103 can be specifically implemented through the following step 103a.

[0094] Step 103a: The XR device replaces the coordinate information of the first vertex with the second coordinate information, and samples and draws the texture of the three-dimensional object through the fragment shader of the XR device to obtain a rendered image.

[0095] Optionally, in an embodiment of the present application, the XR device can replace the coordinate information of the first vertex with the second coordinate information, enter the fragment shader of the three-dimensional object drawing, perform sampling and drawing of the object texture, and the final rendered image is the final anti-distorted image.

[0096] It should be noted that the above-mentioned fragment shader is an important component of the graphics rendering pipeline, which is responsible for performing color calculations on each pixel fragment, thereby ultimately presenting a graphics rendering effect.

[0097] In an embodiment of the present application, since the XR device can replace the coordinate information of the above-mentioned first vertex with the above-mentioned second coordinate information, and sample and draw the texture of the three-dimensional object through the fragment shader of the XR device to obtain the above-mentioned rendered image, it is possible to achieve anti-distortion processing of the three-dimensional object by replacing the coordinate information, thereby simplifying the anti-distortion processing process.

[0098] In the image processing method provided in the embodiment of the present application, when a three-dimensional object is projected onto the screen of the XR device, it is possible to directly sample the anti-distortion texture based on the coordinate information of the point on the screen projected from the vertex of the three-dimensional object to obtain the anti-distorted coordinate information of the point, and then perform image drawing and rendering based on the anti-distorted coordinate information, without the need to use polynomial-based anti-distortion to perform anti-distortion processing on the vertices of the three-dimensional object. Therefore, real-time calculations using high-order polynomials can be avoided, thereby greatly reducing the amount of calculations and improving the anti-distortion performance.

[0099] The specific method for the XR device to create the above anti-distortion texture is described in detail below.

[0100] In the rendering pipeline, the resolution of the texture may be smaller than the screen resolution, which means that the screen pixels may not have completely matching texels in the texture. However, when sampling in the texture according to the coordinates of the screen pixels, the rendering pipeline will automatically perform interpolation sampling, and this sampling is performed in the graphics processing unit (GPU), which can fully perform hardware acceleration and is highly efficient. Therefore, we can learn from the idea of ​​texture sampling in the rendering pipeline, store the regularly distributed distortion information in a texture, and then use the coordinates of the object vertices projected onto the two-dimensional screen to sample this texture in the vertex shading engine stage of the rendering pipeline. Although the distortion information in the texture is regularly distributed, the GPU will automatically perform sampling interpolation to obtain the distortion information after interpolation, and then move the projected vertex accordingly to complete the anti-distortion processing.

[0101] Optionally, in an embodiment of the present application, based on the above ideas, the XR device may create the above anti-distortion texture through the following method 1 or method 2.

[0102] Method 1

[0103] Optionally, the image processing method provided in the embodiment of the present application may further include the following steps 104 to 108.

[0104] Step 104: The XR device creates two-channel anti-distortion textures.

[0105] Optionally, in an embodiment of the present application, the two channels may be any two channels, for example, the two channels may be an R channel and a G channel.

[0106] Optionally, in an embodiment of the present application, the resolution of the above-mentioned anti-distortion texture can be any resolution; wherein the size of the resolution can directly affect the accuracy of anti-distortion, the larger the resolution, the more accurate the anti-distortion result, and the smaller the resolution, the coarser the anti-distortion result.

[0107] Optionally, in the embodiment of the present application, the resolution of the anti-distortion texture may be preset by the system, or may be arbitrarily set according to actual usage requirements.

[0108] Optionally, in an embodiment of the present application, the texture format of the anti-distortion texture takes the embedded system OpenGL as an example, and the texture format may be GL_RG, GL_RG8, GL_RG16 or GL_RG32F, etc.

[0109] Step 105: The XR device determines fourth radius information from the first pixel to the distortion center of the first lens in the XR device according to the fourth coordinate information of the first pixel in the anti-distortion texture in the clipping space coordinate system.

[0110] The fourth coordinate information is determined according to the row number and column number of the first pixel and the texture size of the anti-distortion texture.

[0111] Optionally, in the application embodiment itself, the XR device may first determine the fourth coordinate information of the first pixel in the clipping space coordinate system; wherein, different placement relationships between the first lens and the screen in the XR device may determine the fourth coordinate information by different methods.

[0112] For example, to facilitate calculation, Figure 8 As shown, the rectangle represents the screen, and the inscribed circle of the rectangle represents the first lens. Since the coordinate system used by the screen is a two-dimensional clipping space coordinate system, the value range of the screen coordinates is between -1.0 and 1.0, the origin of the coordinate system is at the center of the screen, the x-axis extends to the right, and the y-axis extends upward; assuming that the distortion center of the screen and the first lens is O, and the coordinates are (0, 0), then Figure 8It can be seen that the radius R of the inscribed distortion circle is 1.0. For the coordinates (x, y) of a certain point on the entire above-mentioned screen, the value range is -1.0 to 1.0. The number of vertices calculated by the anti-distortion polynomial is consistent with the resolution of the above-mentioned anti-distortion texture. Therefore, for the pixel in the rth row and cth column of the anti-distortion texture (it is agreed to start from the lower left corner of the texture, the number of rows increases upward, and the number of columns increases to the right), the corresponding coordinate information in the clipping space coordinate system is (x, y) = (2c / W-1.0, 2r / W-1.0), where the resolution of the anti-distortion texture (ie, the texture size) is W×W. In this way, the XR device can determine the fourth coordinate information mentioned above.

[0113] It should be noted that, if the distortion center of the first lens and the center of the screen do not coincide with each other, the fourth coordinate information may be determined based on an offset value between the distortion center and the center of the screen.

[0114] Optionally, in the embodiment of the present application, if the fourth coordinate information is (a, b), then the fourth radius information

[0115] Step 106: The XR device calculates fifth radius information according to the fourth radius information using an anti-distortion polynomial formula.

[0116] Optionally, in an embodiment of the present application, the XR device can use the above-mentioned fourth radius information as an input parameter, and use the above-mentioned anti-distortion polynomial formula fitted by the above-mentioned first lens calibration (the coordinate system during fitting is consistent with the cropping coordinate system used by the above-mentioned screen, and the origin is the above-mentioned distortion center) to calculate the above-mentioned fifth radius information of the new position to which it should be distorted is away from the distortion center.

[0117] Optionally, in an embodiment of the present application, the anti-distortion polynomial formula may be the above formula (1).

[0118] Step 107: The XR device determines fifth coordinate information according to the fourth coordinate information, the fourth radius information, and the fifth radius information.

[0119] Optionally, in an embodiment of the present application, assuming that the fourth coordinate information is (a, b), the fourth radius information is r1, and the fifth radius information is r1', the XR device can determine the second coordinate information as (x', y') based on the fourth coordinate information, the fourth radius information and the fifth radius information, where (x', y') = (a×r1' / r1, b×r1' / r1).

[0120] Step 108 : The XR device stores fifth coordinate information in two channels of the first pixel.

[0121] Among them, the at least one coordinate information includes the fifth coordinate information.

[0122] Optionally, in an embodiment of the present application, assuming that the second coordinate information is (x', y'), and the two channels are the R channel and the G channel, the XR device may store x' in the R channel and y' in the G channel; or, may store x' in the G channel and y' in the R channel.

[0123] Optionally, in an embodiment of the present application, the above-mentioned first pixel can be all pixels in the above-mentioned anti-distortion texture, that is, after executing the above-mentioned steps 104 to 108 for each pixel in the anti-distortion texture, the above-mentioned at least one coordinate information can be stored in the anti-distortion texture, that is, the initialization of the anti-distortion texture is completed.

[0124] In the embodiment of the present application, since the anti-distortion texture of the above two channels can be created and the anti-distortion coordinate information can be stored in the anti-distortion texture, it is convenient to directly sample the coordinate information in the anti-distortion texture during the anti-distortion processing, thereby simplifying the sampling process.

[0125] In the embodiment of the present application, since the anti-distortion texture created is a two-channel two-dimensional texture, it can store coordinate information and avoid wasting storage space. The row and column numbers of the texture pixels represent the original positions of the vertices after conversion. Fig. 9 As shown, the two channels of the anti-distortion texture store the coordinate information of the target position after distortion respectively. The coordinates of the distorted position can be calculated in advance using the anti-distortion polynomial formula, and the anti-distortion information is stored in the two-dimensional texture. After the vertices of the three-dimensional object mesh are projected onto the above screen, their coordinate information is converted and sampled in the two-dimensional texture to obtain the target position coordinate information, and then the screen coordinate information is updated. Then, after passing through the fragment shader, the anti-distortion processing of the three-dimensional object is completed by displaying it on the screen.

[0126] Method 2

[0127] Optionally, the image processing method provided in the embodiment of the present application may further include the following steps 109 to 112.

[0128] Step 109: The XR device creates an anti-distortion texture for one channel.

[0129] Optionally, in the embodiment of the present application, the one channel may be any channel, for example, the one channel may be an R channel.

[0130] Step 110: The XR device determines sixth radius information from the second pixel to the distortion center of the first lens in the XR device according to the column number of the second pixel in the anti-distortion texture and the texture size of the anti-distortion texture.

[0131] Optionally, in an embodiment of the present application, since the distortion of the lens is a symmetrical radial distortion, that is, as long as the original vertex is on a concentric circle, its distorted position is on another concentric circle, and is on the radius in the same direction of the circle as the original position and the distortion center, the column number in the one-dimensional texture can be used to represent the distance from the original projection point to the distortion center, and the texture R channel stores the distance from the distorted point to the distortion center.

[0132] For example, in combination with the above Figure 8 and Figure 8 In the corresponding example, the XR device can take a radius of 45° for preprocessing calculation. For the pixel in the cth column (starting from the left side of the texture and increasing to the right) of the one-dimensional texture, the corresponding distance to the center of distortion is r = R × c / W. Through this formula, the sixth radius information can be determined.

[0133] Step 111: The XR device calculates the seventh radius information according to the sixth radius information using an anti-distortion polynomial formula.

[0134] Optionally, in an embodiment of the present application, the electronic device takes the above-mentioned sixth radius information as an input parameter, and uses the anti-distortion polynomial formula fitted by optical lens calibration (the coordinate system during fitting is consistent with the cropping coordinate system used by the above-mentioned screen, and the origin is the distortion center) to calculate the above-mentioned seventh radius information of the new position to which it should be distorted from the distortion center.

[0135] Step 112 : The XR device stores seventh radius information in a channel of the second pixel.

[0136] Among them, the at least one radius information includes the seventh radius information.

[0137] Optionally, in an embodiment of the present application, the second pixel may be all pixels in the anti-distortion texture, that is, after executing steps 109 to 112 for each pixel in the anti-distortion texture, the at least one radius information may be stored in the anti-distortion texture, thereby completing the initialization of the anti-distortion texture.

[0138] For other descriptions in the embodiments of the present application, please refer to the relevant descriptions in the above-mentioned method 1. In order to avoid repetition, they will not be repeated here.

[0139] In the embodiment of the present application, since the anti-distortion texture of the above-mentioned one channel can be created and the anti-distortion radius information can be stored in the anti-distortion texture, it is convenient to directly sample the radius information in the anti-distortion texture during the anti-distortion processing, thereby simplifying the sampling process. The anti-distortion texture stores one-dimensional anti-distortion information, thereby further saving device storage space.

[0140] In the embodiment of the present application, the anti-distortion information can be compressed into a one-dimensional texture for storage, which can further save storage space. The distortion of the lens is a symmetrical radial distortion, so the original position and the distorted position of any vertex after projection can be described as the distance to the center of distortion, which is what the anti-distortion polynomial formula above means. The one-dimensional texture has only one row, and the column number represents the distance from the position of the vertex projected onto the screen to the center of distortion. Fig.10 As shown, the channel of the above anti-distortion texture stores the distance from the distorted position to the distortion center, which can be calculated by the anti-distortion polynomial formula, and the anti-distortion information is stored in the one-dimensional texture. After the object mesh vertex is projected onto the screen, the distance from the object to the distortion center is calculated based on its coordinate information. This distance is converted and sampled in the one-dimensional texture to obtain the distance from the distorted target position to the distortion center. Then the coordinate information of the target position on the two-dimensional screen can be calculated, and then the screen coordinate information is updated. Then, after the pixel shader is applied, the anti-distortion processing of the three-dimensional object is completed by displaying it on the screen.

[0141] By using the image processing method provided in the embodiment of the present application, a preprocessing of converting an anti-distortion polynomial into a one-dimensional or two-dimensional texture is implemented, and the anti-distortion information can be stored in the preprocessed texture. In the scenario where a polynomial-based anti-distortion solution must be used, an arbitrary vertex anti-distortion method based on a one-dimensional or two-dimensional texture is implemented, which can make full use of the natural hardware acceleration capability of the GPU, avoid the defect of the large amount of calculation of the real-time polynomial anti-distortion solution affecting the rendering performance, and greatly improve the anti-distortion performance.

[0142] The above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or, under the premise that there is no contradiction, can also be executed in combination with each other. The specific implementation can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0143] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0144] like Fig.11 As shown, an embodiment of the present application provides an image processing device 110 , which may include an acquisition module 111 , a sampling module 112 , and a drawing module 113 .

[0145] Among them, the acquisition module 111 can be used to obtain the first coordinate information of the first vertex, and the first vertex is the vertex of the second vertex of the three-dimensional object projected onto the screen of the XR device. The sampling module 112 can be used to sample the anti-distortion texture based on the first coordinate information obtained by the acquisition module 111 to obtain the second coordinate information. The anti-distortion texture is used to anti-distort the texture of the three-dimensional object, and the anti-distortion texture stores at least one coordinate information or at least one radius information. The drawing module 113 can be used to sample and draw the texture of the three-dimensional object based on the second coordinate information sampled by the sampling module 112 to obtain a rendered image corresponding to the three-dimensional object.

[0146] In a possible implementation, the anti-distortion texture stores the at least one coordinate information. The sampling module 112 may be specifically configured to convert the first coordinate information into third coordinate information in a texture coordinate system; and perform sampling in the anti-distortion texture according to the third coordinate information to obtain the second coordinate information.

[0147] In a possible implementation, the anti-distortion texture stores the at least one radius information. The sampling module 112 may be specifically configured to determine the first radius information from the first vertex to the distortion center of the first lens in the XR device according to the first coordinate information; and convert the first radius information into the second radius information in the texture coordinate system; and sample the anti-distortion texture according to the second radius information to obtain the third radius information; and determine the second coordinate information according to the first coordinate information, the first radius information and the third radius information.

[0148] In a possible implementation, the drawing module 113 may be specifically configured to replace the coordinate information of the first vertex with the second coordinate information, and sample and draw the texture of the three-dimensional object through a fragment shader of the XR device to obtain the rendered image.

[0149] In a possible implementation, the image processing device 110 may further include a processing module. The processing module may be used to create the above-mentioned anti-distortion textures of two channels; and according to the fourth coordinate information of the first pixel in the anti-distortion texture in the clipping space coordinate system, determine the fourth radius information of the distortion center of the first lens in the XR device from the first pixel, the fourth coordinate information is determined according to the row number and column number of the first pixel, and the texture size of the anti-distortion texture; and according to the fourth radius information, calculate the fifth radius information by the anti-distortion polynomial formula; and according to the fourth coordinate information, determine the fifth coordinate information, the fourth radius information and the fifth radius information; and store the fifth coordinate information in the two channels of the first pixel. Wherein, the above-mentioned at least one coordinate information includes the fifth coordinate information.

[0150] In a possible implementation, the image processing device 110 may further include a processing module. The processing module may be used to create the above-mentioned anti-distortion texture of a channel; and determine the sixth radius information from the second pixel to the distortion center of the first lens in the XR device according to the column number of the second pixel in the anti-distortion texture and the texture size of the anti-distortion texture; and calculate the seventh radius information according to the sixth radius information by the anti-distortion polynomial formula; and store the seventh radius information in a channel of the second pixel. The above-mentioned at least one radius information includes the seventh radius information.

[0151] In the image processing device provided in the embodiment of the present application, when a three-dimensional object is projected onto the screen of the XR device, the image processing device can directly sample the anti-distortion texture based on the coordinate information of the point on the screen projected from the vertex of the three-dimensional object, obtain the coordinate information of the point after anti-distortion, and then perform image drawing and rendering based on the anti-distortion coordinate information, without using polynomial-based anti-distortion to perform anti-distortion processing on the vertices of the three-dimensional object. Therefore, real-time calculations through high-order polynomials can be avoided, thereby greatly reducing the amount of calculations and improving the anti-distortion performance.

[0152] The image processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR), a virtual reality (virtual reality, VR) device, an XR device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0153] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0154] The image processing device provided in the embodiment of the present application can implement each process implemented in the above method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0155] like Fig.12 As shown, an embodiment of the present application further provides an electronic device 100, including a processor 101 and a memory 102, wherein the memory 102 stores programs or instructions that can be executed on the processor 101, and when the program or instructions are executed by the processor 101, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0156] It should be noted that the electronic devices in the embodiments of the present application include mobile electronic devices and non-mobile electronic devices.

[0157] Fig.13 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0158] like Fig.13 As shown, the electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.

[0159] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.13 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0160] Among them, the processor 1010 can be used to obtain first coordinate information of a first vertex, where the first vertex is the vertex of the second vertex of the three-dimensional object projected onto the screen of the XR device; and based on the obtained first coordinate information, sampling is performed in the anti-distortion texture to obtain second coordinate information, and the anti-distortion texture is used to anti-distort the texture of the three-dimensional object, and the anti-distortion texture stores at least one coordinate information or at least one radius information; and based on the second coordinate information obtained by sampling, the texture of the three-dimensional object is sampled and drawn to obtain a rendered image corresponding to the three-dimensional object.

[0161] In a possible implementation, the anti-distortion texture stores the at least one coordinate information. The processor 1010 may be specifically configured to convert the first coordinate information into third coordinate information in a texture coordinate system; and perform sampling in the anti-distortion texture according to the third coordinate information to obtain the second coordinate information.

[0162] In a possible implementation, the anti-distortion texture stores the at least one radius information. The processor 1010 may be specifically configured to determine, based on the first coordinate information, the first radius information from the first vertex to the distortion center of the first lens in the XR device; and convert the first radius information into second radius information in a texture coordinate system; and based on the second radius information, sample the anti-distortion texture to obtain third radius information; and determine the second coordinate information based on the first coordinate information, the first radius information, and the third radius information.

[0163] In a possible implementation, the processor 1010 can be specifically used to replace the coordinate information of the first vertex with the second coordinate information, and sample and draw the texture of the three-dimensional object through the fragment shader of the XR device to obtain the rendered image.

[0164] In a possible implementation, the processor 1010 may also be used to create the above-mentioned anti-distortion texture of two channels; and determine the fourth radius information of the first pixel to the distortion center of the first lens in the XR device according to the fourth coordinate information of the first pixel in the anti-distortion texture in the clipping space coordinate system, the fourth coordinate information is determined according to the row number and column number of the first pixel, and the texture size of the anti-distortion texture; and calculate the fifth radius information according to the fourth radius information by the anti-distortion polynomial formula; and determine the fifth coordinate information according to the fourth coordinate information, the fourth radius information and the fifth radius information; and store the fifth coordinate information in the two channels of the first pixel. Wherein, the above-mentioned at least one coordinate information includes the fifth coordinate information.

[0165] In a possible implementation, the processor 1010 may also be used to create the above-mentioned anti-distortion texture of a channel; and determine the sixth radius information from the second pixel to the distortion center of the first lens in the XR device according to the column number of the second pixel in the anti-distortion texture and the texture size of the anti-distortion texture; and calculate the seventh radius information according to the sixth radius information by using the anti-distortion polynomial formula; and store the seventh radius information in a channel of the second pixel. The at least one radius information includes the seventh radius information.

[0166] In the electronic device provided in the embodiment of the present application, when a three-dimensional object is projected onto the screen of the XR device, the electronic device can directly sample the anti-distortion texture based on the coordinate information of the point on the screen projected from the vertex of the three-dimensional object, obtain the coordinate information of the point after anti-distortion, and then perform image drawing and rendering based on the anti-distortion coordinate information, without using polynomial-based anti-distortion to perform anti-distortion processing on the vertices of the three-dimensional object. Therefore, real-time calculations through high-order polynomials can be avoided, thereby greatly reducing the amount of calculations and improving the anti-distortion performance.

[0167] It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0168] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0169] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.

[0170] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0171] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0172] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0173] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0174] An embodiment of the present application provides a computer program / program product, which is stored in a storage medium. The program / program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0175] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0176] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0177] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, applied to an extended reality XR device, characterized in that: The method comprises: Acquire first coordinate information of a first vertex, where the first vertex is a vertex on which a second vertex of the three-dimensional object is projected onto a screen of the XR device; Based on the first coordinate information, sampling is performed in an anti-distortion texture to obtain second coordinate information, where the anti-distortion texture is used to anti-distort the texture of the three-dimensional object, and the anti-distortion texture stores at least one coordinate information or at least one radius information; Based on the second coordinate information, the texture of the three-dimensional object is sampled and drawn to obtain a rendering image corresponding to the three-dimensional object.

2. The method according to claim 1, characterized in that The at least one coordinate information is stored in the anti-distortion texture; The step of sampling the anti-distortion texture based on the first coordinate information to obtain the second coordinate information includes: Converting the first coordinate information into third coordinate information in a texture coordinate system; According to the third coordinate information, sampling is performed in the anti-distortion texture to obtain the second coordinate information.

3. The method according to claim 1, characterized in that The at least one radius information is stored in the anti-distortion texture; The step of sampling the anti-distortion texture based on the first coordinate information to obtain the second coordinate information includes: Determine, according to the first coordinate information, first radius information from the first vertex to the distortion center of a first lens in the XR device; Converting the first radius information into second radius information in a texture coordinate system; According to the second radius information, sampling is performed in the anti-distortion texture to obtain third radius information; The second coordinate information is determined according to the first coordinate information, the first radius information and the third radius information.

4. The method according to claim 1, characterized in that The step of sampling and drawing the texture of the three-dimensional object based on the second coordinate information to obtain a rendered image corresponding to the three-dimensional object includes: The coordinate information of the first vertex is replaced with the second coordinate information, and the texture of the three-dimensional object is sampled and drawn through the fragment shader of the XR device to obtain the rendered image.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Creating two channels of the anti-distortion texture; Determine, according to fourth coordinate information of a first pixel in the anti-distortion texture in a clipping space coordinate system, fourth radius information from the first pixel to a distortion center of a first lens in the XR device, wherein the fourth coordinate information is determined according to a row number and a column number of the first pixel and a texture size of the anti-distortion texture; According to the fourth radius information, fifth radius information is calculated by using an anti-distortion polynomial formula; determining fifth coordinate information according to the fourth coordinate information, the fourth radius information and the fifth radius information; storing the fifth coordinate information in two channels of the first pixel; The at least one coordinate information includes the fifth coordinate information.

6. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Creating a channel of the anti-distortion texture; Determine, according to a column number of a second pixel in the anti-distortion texture and a texture size of the anti-distortion texture, sixth radius information from the second pixel to a distortion center of a first lens in the XR device; According to the sixth radius information, seventh radius information is calculated by using an anti-distortion polynomial formula; storing the seventh radius information in a channel of the second pixel; Among them, the at least one radius information includes the seventh radius information.

7. An image processing device, characterized in that: The device comprises an acquisition module, a sampling module and a drawing module; The acquisition module is used to acquire first coordinate information of a first vertex, where the first vertex is a vertex of a second vertex of the three-dimensional object projected onto a screen of the XR device; The sampling module is used to sample the anti-distortion texture based on the first coordinate information obtained by the obtaining module to obtain second coordinate information, wherein the anti-distortion texture is used to anti-distort the texture of the three-dimensional object, and the anti-distortion texture stores at least one coordinate information or at least one radius information; The drawing module is used to sample and draw the texture of the three-dimensional object based on the second coordinate information sampled by the sampling module to obtain a rendering image corresponding to the three-dimensional object.

8. The device according to claim 7, characterized in that The at least one coordinate information is stored in the anti-distortion texture; The sampling module is specifically used to convert the first coordinate information into third coordinate information in a texture coordinate system; and perform sampling in the anti-distortion texture according to the third coordinate information to obtain the second coordinate information.

9. The device according to claim 7, characterized in that The at least one radius information is stored in the anti-distortion texture; The sampling module is specifically used to determine the first radius information from the first vertex to the distortion center of the first lens in the XR device according to the first coordinate information; and convert the first radius information into second radius information in the texture coordinate system; and according to the second radius information, sample in the anti-distortion texture to obtain third radius information; and determine the second coordinate information according to the first coordinate information, the first radius information and the third radius information.

10. The device according to claim 7, characterized in that The drawing module is specifically used to replace the coordinate information of the first vertex with the second coordinate information, and sample and draw the texture of the three-dimensional object through the fragment shader of the XR device to obtain the rendered image.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the image processing method according to any one of claims 1 to 6 are implemented.

12. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image processing method according to any one of claims 1 to 6 are implemented.

Images