Image processing method, device, electronic device and storage medium

Abstract

The present application discloses an image processing method, device, electronic device and storage medium, belonging to the field of image processing technology. The method includes: obtaining a first rendered image and a plane grid; for each pixel in the plane grid, creating a sight vector between the pixel and the image acquisition device; based on the sight vector, dividing the depth value of the plane grid into N layers, N is a positive integer; according to the intersection of the sight vector and the N depth layers projected onto the plane grid, determining the target UV coordinate of the pixel; according to the target UV coordinate, sampling the target color value of the pixel in the first rendered image; according to the target color value corresponding to each pixel, obtaining a second rendered image, the second rendered image is the rendered image at the second moment.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an image processing method, device, electronic device and storage medium. Background Art

[0002] When rendering an image, since rendering is time-consuming, the rendering engine often renders the image based on the camera posture a few millimeters before sending it to the display device. However, during these few millimeters, the camera posture will be updated, so the rendered image seen by the user actually has a certain delay.

[0003] Currently, the problem of delay in the rendered image seen by users can be solved by using Positional Timewarp (PTW). However, this method has a large amount of calculation, which affects the computing performance, occupies a large amount of server memory, and is prone to forming holes due to missing pixels. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image processing method, device, electronic device and storage medium, so as to reduce the amount of rendering calculations during the image rendering process, save server memory usage, and avoid generating holes caused by missing pixels.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0006] Acquire a first rendered image and a plane mesh, wherein the first rendered image is determined based on an image acquired by an image acquisition device with a first pose information at a first moment, the plane mesh is a mesh created at a second moment and parallel to a near clipping plane under the first pose information, and the pose information of the image acquisition device at the second moment is the second pose information;

[0007] For each pixel point in the plane grid, create a sight line vector between the pixel point and the image acquisition device;

[0008] Dividing the depth value of the plane grid into N layers, where N is a positive integer, and the depth value of each depth layer in the N depth layers is different;

[0009] Determine the target UV coordinate of the pixel point according to the UV coordinate projected onto the plane grid by the target sight line vector and the intersection of the N depth layers, wherein the target sight line vector is the sight line vector after the sight line vector is converted into the tangent space;

[0010] Sampling the first rendered image according to the target UV coordinates to obtain a target color value of the pixel;

[0011] A second rendered image is obtained according to the target color value corresponding to each pixel, where the second rendered image is the rendered image at the second moment.

[0012] In a second aspect, an embodiment of the present application provides an image processing device, the device comprising:

[0013] A first acquisition module is used to acquire a first rendered image and a plane mesh, wherein the first rendered image is determined based on an image acquired by an image acquisition device with a first pose information at a first moment, the plane mesh is a mesh created at a second moment and parallel to a near clipping plane under the first pose information, and the pose information of the image acquisition device at the second moment is second pose information;

[0014] A creation module, used for creating a sight line vector between each pixel point in the plane grid and the image acquisition device;

[0015] A division module, used for dividing the depth value of the plane grid into N layers, where N is a positive integer, and the depth value of each depth layer in the N depth layers is different;

[0016] A first determination module is used to determine the target UV coordinates of the pixel point according to the UV coordinates projected onto the plane grid by the target sight line vector and the intersection points of the N depth layers, wherein the target sight line vector is the sight line vector after the sight line vector is converted into the tangent space;

[0017] A sampling module, used for sampling the first rendered image according to the target UV coordinates to obtain the target color value of the pixel point;

[0018] The second determining module is used to obtain a second rendering image according to the target color value corresponding to each pixel point, where the second rendering image is the rendering image at the second moment.

[0019] In a third aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0020] In a fourth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0021] In a fifth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0022] In an embodiment of the present application, the depth values ​​of the plane grid corresponding to the first rendered image are layered, and for each pixel point of the plane grid, a sight vector between the pixel point and the image acquisition device is created, and then the sight vector is converted into the tangent space to obtain a target sight vector. In this way, the target UV coordinates of the pixel point can be determined based on the intersection of the target sight vector and the N depth layers projected onto the UV coordinates on the plane grid, thereby realizing pixel reprojection. Compared with the prior art in which reprojection calculation is performed on each pixel point in the first rendered image that is rendered for the image captured at the first moment, the scheme of the embodiment of the present application has a smaller amount of calculation. Compared with the prior art in which the image acquisition device is converted from the first pose information at the first moment to the second pose information at the second moment, an image buffer area needs to be added to the central processing unit to perform movement calculation of each pixel point when performing pixel reprojection calculation. The scheme of the embodiment of the present application does not need to perform movement calculation of each pixel point, nor does it need to increase the image buffer area, thereby saving memory usage. Furthermore, the solution of the embodiment of the present application is to determine the target UV coordinates of the pixel points based on the UV coordinates projected onto the plane grid based on the intersection of the sight vector and the N depth layers, and then add the target color value corresponding to each pixel point to the plane grid to obtain a second rendered image. That is, the solution of the embodiment of the present application is to directly start from the pixel points seen by the sight vector at the second moment, and sample from the first rendered image to achieve the purpose of reprojection, so the problem of holes formed by missing pixels will not occur, which solves the problem of holes formed due to missing pixels in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flowchart of an image processing method provided by some embodiments of the present application;

[0024] Figure 2 is a schematic diagram of determining a plane grid provided by some embodiments of the present application;

[0025] Figure 3 is a schematic diagram of a first rendered image and a first depth image obtained by rendering an image captured by an image acquisition device at a first moment with a first pose information, provided in some embodiments of the present application;

[0026] Figure 4 is a schematic diagram of a plane grid image and a first depth image observed by an image acquisition device with first pose information at a second moment provided by some embodiments of the present application;

[0027] Figure 5 is a schematic diagram of a plane grid image and a first depth image observed by an image acquisition device with first pose information at a second moment provided by some embodiments of the present application;

[0028] Figure 6Some embodiments of the present application provide Figure 5 A schematic diagram of an image after simplifying an image;

[0029] Figure 7 is a flowchart of an image processing method provided by some embodiments of the present application;

[0030] Figure 8 is a schematic diagram of the structure of an image processing device shown in some embodiments of the present application;

[0031] Fig. 9 is a schematic diagram of the structure of an electronic device shown in some embodiments of the present application;

[0032] Fig.10 It is a schematic diagram of the hardware structure of an electronic device shown in some embodiments of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0034] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or N. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0035] Before introducing the technical solution of the embodiment of the present application, the background technology of the embodiment of the present application is first introduced:

[0036] The rendered images that users see actually have a certain delay, which is difficult for users to detect in ordinary 3D application scenarios. However, on Extended Reality (XR) devices, since the camera posture is obtained through the head display's sensor, it simulates the user's real head movement. The actual perspective of the user is different from the perspective that the user's brain perceives. Although this difference is only a few milliseconds, it can cause dizziness in the user and greatly reduce the user experience.

[0037] Currently, the PTW method can be used to solve the problem of delay in the rendering image seen by the user. Specifically, the depth data of the previous frame of rendered image is used to restore the three-dimensional spatial coordinates corresponding to each pixel point in the previous frame of rendered image, and then the latest head posture information is used to re-project it onto the two-dimensional screen. Then, the pixel points of the previous frame of rendered image are moved to the position of the two-dimensional screen, and the previous frame of rendered image can be redrawn.

[0038] The above solution of using PTW to solve the problem of delay in rendering images seen by users has the following defects:

[0039] (1) Each reprojection requires a lot of computation. In XR scenarios, the rendering resolution of a single eye often reaches 2k, and it is necessary to draw images of both eyes and perform this reprojection calculation for each pixel, resulting in poor overall algorithm performance.

[0040] (2) The above solution requires pixel movement in the graphics processing unit (GPU), but this is very difficult. Often, pixel movement can only be performed through image processing to generate a new image. However, on the one hand, this image processing method can generally only be performed in the central processing unit (CPU) and cannot be implemented in the GPU, and the processing time is relatively high. On the other hand, an image buffer needs to be added for pixel movement, which increases memory usage;

[0041] (3) The pixels in the previous frame of the rendered image will move to a new position in the next frame of the image, but there may not be a pixel that can be moved to the position of the pixel. In this case, the pixel at the position will be missing, forming a hole, which will greatly affect the user's visual experience. The additional repair of the hole will add new processing links and increase new time consumption, which is not conducive to the overall performance of the system. Another method is to use a plane grid with the same number of vertices as the number of pixels in the vertex shader stage, use the depth data to calculate the three-dimensional coordinates of each grid vertex, and then move each vertex of the plane grid to the three-dimensional coordinates. When viewing the grid from a new perspective, since the grid is no longer flat, the correct rendering result can be obtained from this perspective. The disadvantage of this method is that the grid that could have been composed of 6 vertices is turned into millions of vertices. Such a large amount of data will greatly reduce the rendering performance, which is not conducive to the final good user experience.

[0042] In order to solve the above problems, the embodiments of the present application provide an image processing method, device, electronic device and storage medium, which layer the depth values ​​of the plane grid corresponding to the first rendered image, and for each pixel point of the plane grid, create a line of sight vector between the pixel point and the image acquisition device, and then project the intersection of the line of sight vector and N depth layers to the UV coordinates on the plane grid to determine the target UV coordinate of the pixel point, thereby realizing pixel reprojection. Compared with the prior art in which reprojection calculation is performed on each pixel point in the first rendered image captured at the first moment, the scheme of the embodiments of the present application has a smaller amount of calculation. Compared with the prior art in which the image acquisition device converts from the first pose information at the first moment to the second pose information at the second moment, it is necessary to add an image buffer area in the central processing unit to perform the movement calculation of each pixel point when performing the reprojection calculation of the pixel point. The scheme of the embodiments of the present application does not need to perform the movement calculation of each pixel point, nor does it need to increase the image buffer area, thereby saving memory usage. Furthermore, the solution of the embodiment of the present application is to determine the target UV coordinates of the pixel points based on the UV coordinates projected onto the plane grid based on the intersection of the sight vector and the N depth layers, and then add the target color value corresponding to each pixel point to the plane grid to obtain a second rendered image. That is, the solution of the embodiment of the present application is to directly start from the pixel points seen by the sight vector at the second moment, and sample from the first rendered image to achieve the purpose of reprojection, so the problem of holes formed by missing pixels will not occur, which solves the problem of holes formed due to missing pixels in the prior art.

[0043] The technical solution of the embodiment of the present application can be applied to the scenario where the posture of the image acquisition device changes and the image is rendered.

[0044] The image processing method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0045] Figure 1 It is a flow chart of an image processing method provided in an embodiment of the present application. The executor of the image processing method may be an electronic device or an XR device. The electronic device may be, but is not limited to, a personal computer (PC), a smart phone, a tablet computer or a personal digital assistant (PDA), etc. The XR device here may be, but is not limited to, a virtual reality (VR) device or an augmented reality (AR) device, etc.

[0046] like Figure 1 As shown, the image processing method provided in the embodiment of the present application may include steps 110 to 160.

[0047] Step 110: Obtain a first rendered image and a plane mesh.

[0048] The first rendered image may be determined based on an image captured by an image acquisition device with a first pose information at a first moment. The first moment here may be a certain moment, and the image acquisition device may be a device for image acquisition, such as a camera or a virtual camera in a graphics rendering pipeline. The first pose information may be pose information of the image acquisition device at the first moment.

[0049] It should be noted that, when the image acquisition device is a camera, the first rendered image is an image acquired by the camera at the first moment with the first pose information. When the image acquisition device is a virtual camera in a graphics rendering pipeline, the first rendered image is an image obtained by rendering the image acquired by the virtual camera in the graphics rendering pipeline at the first moment with the first pose information.

[0050] The image acquisition devices in the following embodiments are described by taking a virtual camera in a graphics rendering pipeline as an example.

[0051] The plane grid can be a grid created at the second moment and parallel to the near clipping plane under the first pose information. Specifically, a plane grid can be created at the second moment, which is parallel to the near clipping plane under the first pose information. The plane grid covers the entire screen and can be called a texture surface.

[0052] The second moment here may be a moment later than the first moment. For example, if the first moment is 10:00 a.m. on September 30, 2024, the second moment may be 10:05 msec. on September 30, 2024.

[0053] It should be noted that in the embodiment of the present application, the difference between the second moment and the first moment is at the millisecond level. If the difference between the second moment and the first moment reaches the second level, then the image change caused by the posture movement between the first moment and the second moment is already greater than the image range of the first moment, then the image at the second moment has no valid content compared to the image at the first moment.

[0054] It should be noted that the above-mentioned near clipping plane can be understood as follows: under the first pose information, the field of view of the image acquisition device is a cone, such as Figure 2 As shown, the cone is cut into layers, which can be considered as cutting layers closer to the image acquisition device. The cut surface closest to the image acquisition device is the near clipping surface, and the cut surface farthest from the image acquisition device (such as Figure 2 The bottom circle in the figure is the far clipping plane.

[0055] It should be noted that the plane grid may not be created at the second moment, but may be created at the first moment. The creation time of the plane grid here can be set according to user needs and is not limited in the embodiment of the present application.

[0056] In some embodiments of the present application, at the second moment, the posture information of the image acquisition device may be second posture information, which is different from the first posture information.

[0057] Step 120: For each pixel point in the plane grid, create a sight vector between the pixel point and the image acquisition device.

[0058] For each pixel point in the plane grid, the sight vector may be a vector between the pixel point and the image acquisition device.

[0059] refer to Figure 3 , taking the image acquisition device as a virtual camera in the graphics rendering pipeline as an example, the virtual camera at the first moment with the first pose information (i.e. Figure 3 The rendered image of the Nth frame image is obtained by rendering the image captured by the camera pose 1), that is, Figure 3 The image 31 in is the first rendered image, and the depth information corresponding to the image 31 is as follows: Figure 3 As shown by the reference numeral 32 in the figure, that is, the reference numeral 32 is a set of depth values ​​in the depth map of the image 31. Figure 3 middle, is the sight vector of the virtual camera with the first pose information.

[0060] refer to Figure 4 , Figure 4 The virtual camera at the second moment takes the second pose information (i.e. Figure 4 Schematic diagram of the camera pose 2) when observing, Figure 4 The image 41 in the middle can be regarded as the plane mesh created. When the virtual camera observes with the second posture information, it sees a point P0 on the plane mesh. The sight vector between the created pixel point P0 and the camera.

[0061] Step 130: Divide the depth values ​​of the plane grid into N layers.

[0062] Wherein, N is a positive integer.

[0063] In some embodiments of the present application, the depth value of the plane grid may be divided into N layers, wherein the depth value of each depth layer in the N depth layers is different.

[0064] Continue to refer to the above Figure 4When the camera observes with the second pose information, it sees a point P0 on the plane grid. Normally, the UV coordinates at P0 can be used to sample in image 31, and then the N+1th frame image can be colored. However, the texture surface is only the projection plane of image 31, and the actual 3D scene is a stereoscopic space, that is, what you actually see in the 3D scene should be Figure 4 middle The intersection point of the extended line and the object in the three-dimensional scene is P', and its projection point on the texture surface is P. Therefore, we should use the UV coordinate at P to sample and color in image 31 as the color value at P0.

[0065] From the above analysis, we can see that the key point now is to find the UV coordinates corresponding to the projection point P'. Since the plane grid and image 31 are used to render the N+1 frame, it is impossible to directly The intersection judgment and intersection point are performed with all objects in the three-dimensional scene, because the amount of calculation is very huge. However, due to the depth information corresponding to image 31 (i.e. Figure 3 32 in the figure), and the depth information can be used for stereoscopic expression, so the vector Convert to tangent space so that The x-axis vector and the y-axis vector will be aligned with the tangent vector and the bitangent vector of the texture surface respectively, so that you can use The vector in the x-axis direction and the vector in the y-axis direction offset the texture coordinates at P0.

[0066] like Figure 5 As shown in the figure, the maximum z value in the tangent space of point P0 is 1.0, and then it is equally divided into N layers parallel to the surface texture within the range of 1.0. Since the depth value range is 0 to 1.0, it is equivalent to dividing the depth value under the surface texture into N layers, and the depth value of each depth layer is different, such as Figure 5 As shown, the depth value under the surface texture is divided into 10 layers.

[0067] It should be noted that after obtaining the sight line vector, the sight line vector can be normalized first. Perform normalization to obtain the normalized sight vector Will Divide by The Z-axis coordinate of The Z-axis coordinate is 1, so the maximum z value in the tangent space at point P0 is 1.0.

[0068] After the normalized sight vector back, The coordinates V'_nor.xyz represent Figure 5middle The intersection point P with the Nth depth layer in the tangent space N 'coordinates, The coordinate V'_nor.xy in the xy plane represents P N 'Project point P on the plane grid N Texture coordinate offset relative to P0.

[0069] It should be noted that step 130 may be performed after the plane mesh is created, and the process only needs to be performed once, and does not need to be performed every time the image is reprojected.

[0070] Step 140 : Determine the target UV coordinates of the pixel point according to the UV coordinates of the intersection of the target sight line vector and the N depth layers projected onto the plane grid.

[0071] The target sight line vector may be a sight line vector obtained by converting the sight line vector created in step 120 into the tangent space.

[0072] The target UV coordinates may be the UV coordinates of the pixel points.

[0073] In some embodiments of the present application, the target UV coordinates of the pixel point may be determined according to the UV coordinates projected onto the plane grid by the intersection of the target sight line vector and the N depth layers.

[0074] In some embodiments of the present application, in order to accurately determine the target UV coordinates of the pixel point, step 140 may specifically include:

[0075] Calculate the intersection point of the target sight line vector and the j-th depth layer among the N depth layers to obtain the first intersection point;

[0076] Calculate the projection point of the first intersection point on the plane grid to obtain the first projection point;

[0077] Obtaining a first depth value of the first projection point in the first depth image according to a UV coordinate of the first projection point on the plane grid;

[0078] When a depth difference between the first depth value and the second depth value of the j-th depth layer is less than or equal to 0, determining a target UV coordinate of the pixel point according to a UV coordinate of the first projection point on the plane grid;

[0079] When the depth difference between the first depth value and the second depth value of the j-th depth layer is greater than 0, j is updated to j+1, and the calculation of the intersection of the target sight line vector and the j-th depth layer in the N depth layers is returned to calculate the projection point of the first intersection on the plane grid to obtain the first projection point; when the depth difference between the first depth value and the second depth value of the j-th depth layer is less than or equal to 0, the target UV coordinates of the pixel point are determined according to the UV coordinates of the first projection point on the plane grid until j=N.

[0080] The first intersection point may be an intersection point between the target sight line vector and the j-th depth layer among the N depth layers, where the initial value of j is 1 and j is a positive integer.

[0081] The first projection point may be a projection point of the first intersection point on the plane grid.

[0082] like Figure 5 As shown, N is 10, j = 5 as an example, The intersection points with the fifth depth layer among the 10 depth layers are P5', that is, P5' is the first intersection point. The projection point of P5' on the plane grid is P5, that is, P5 is the first projection point.

[0083] The first depth image may be a depth image corresponding to the first rendered image.

[0084] The first depth value may be a depth value of the first projection point in the first depth image, that is, Figure 3 The label 32 in can reflect the first depth value.

[0085] The second depth value may be the depth value of the j-th depth layer, that is, Figure 5 Mean vector The location of the intersection with each depth layer, i.e. Figure 5 The position is shown by the hollow circle.

[0086] In some embodiments of the present application, the intersection of the target sight line vector and each of the N depth layers can be calculated layer by layer, and the projection point of the intersection on the plane grid can be calculated. Then, based on the UV coordinate of the projection point on the plane grid, the first depth value of the projection point in the first depth image can be obtained, and then it is determined whether the depth value of the projection point meets the requirements. If so, the target UV coordinate of the pixel point is determined according to the UV coordinate of the projection point on the plane grid. If not, the intersection of the target sight line vector and the next depth layer is calculated, and the projection point of the intersection on the plane grid is calculated. Then, it is determined whether the depth value of the projection point meets the requirements until the intersection of the target sight line vector and the last depth layer of the N depth layers is calculated.

[0087] Whether the depth value of the above-mentioned projection point meets the requirements, specifically, whether the depth difference between the first depth value of the projection point in the first depth image and the second depth value of the j-th depth layer is less than or equal to 0. When the depth difference between the first depth value and the second depth value is less than or equal to 0, the target UV coordinates of the pixel point are determined according to the UV coordinates of the projection point on the plane grid. When the depth difference between the first depth value and the second depth value is greater than 0, the intersection of the target line of sight vector and the next depth layer is continued to be calculated until the intersection of the target line of sight vector and the last depth layer of the N depth layers is calculated.

[0088] That is to say, when calculating the intersection of the target line of sight vector and the depth layer, there is no need to calculate all the intersections with each of the N depth layers at once. Instead, the intersections are calculated layer by layer, and then subsequent judgments are made on the intersections of the target line of sight vector and the depth layer. If the judgment conditions are met, there is no need to perform intersection operations on the subsequent depth layers, which reduces computing power and saves computing resources.

[0089] Continue to refer Figure 5 and Figure 3 , the intersection of the target sight line vector and the first depth layer can be calculated first, and then the projection point P1 of the intersection on the plane grid can be calculated. According to the UV coordinate of the projection point P1 on the plane grid, the depth value of the projection point P1 in the image 31 can be obtained. If the depth difference between the depth value of the projection point P1 in the image 31 and the depth value of the first depth layer is less than or equal to 0, the target UV coordinate of the pixel point is determined according to the UV coordinate of the projection point P1 on the plane grid.

[0090] If the depth difference between the depth value of the projection point P1 in image 31 and the depth value of the first depth layer is greater than 0, continue to calculate the intersection of the target sight line vector and the second depth layer, and then calculate the projection point P2 of the intersection of the target sight line vector and the second depth layer on the plane grid. According to the UV coordinate of the projection point P2 on the plane grid, the depth value of the projection point P2 in the image 31 can be obtained. If the depth difference between the depth value of the projection point P2 in image 31 and the depth value of the second depth layer is less than or equal to 0, then according to the UV coordinate of the projection point P2 on the plane grid, the target UV coordinate of the pixel point is determined.

[0091] If the depth difference between the depth value of the projection point P2 in the image 31 and the depth value of the second depth layer is greater than 0, continue to calculate the intersection of the target sight line vector and the third depth layer, and so on, until the intersection of the target sight line vector and the tenth depth layer is calculated.

[0092] In an embodiment of the present application, the intersection of the target sight line vector and the N depth layers is calculated layer by layer, the projection point of the intersection on the plane grid is calculated, and then the first depth value of the projection point in the first depth image can be obtained according to the UV coordinate of the projection point on the plane grid, and then it is determined whether the depth value of the projection point meets the requirements, and the target UV coordinate of the pixel can be accurately determined. In addition, when calculating the intersection of the target sight line vector and the depth layer, it is not necessary to calculate all the intersections with each of the N depth layers at one time, which reduces the computing power and saves computing resources.

[0093] In some embodiments of the present application, before acquiring the first depth value of the first projection point in the first depth image according to the UV coordinate of the first projection point on the plane grid, the method further includes:

[0094] Calculate the offset between the first projection point and the pixel point;

[0095] The UV coordinates of the first projection point on the plane grid are determined according to the offset and the position information of the pixel point.

[0096] In some embodiments of the present application, the UV coordinates of the first projection point on the plane grid may be determined based on the offset between the first projection point and the pixel point, and the position information of the pixel point.

[0097] Continue to refer Figure 5 , P1, P2, ..., P 10 are the projection points of the intersection of the target sight vector and the N depth layers on the plane grid. According to the principle of similar triangles, P1, P2, ..., P 10 The distances between adjacent projection points are equal. For the nth projection point, its texture coordinate offset relative to P0 is n·v'_n.xy / N, where v'_n.xy is the coordinate of the nth intersection point.

[0098] Then, according to the texture coordinate offset n·v'_n.xy / N of the nth projection point relative to the pixel point P0, and the position information of the pixel point P0, the UV coordinate of the nth projection point on the plane grid can be obtained according to the following formula: n·v'_n.xy / N-P0.xy, where P0.xy is the position information of the pixel point P0, specifically the position information of the pixel point P0 in the plane grid.

[0099] In an embodiment of the present application, the UV coordinates of the first projection point on the plane grid can be accurately determined through the offset between the first projection point and the pixel point and the position information of the pixel point, and then the target UV coordinates of the pixel point can be accurately determined based on the UV coordinates of the first projection point on the plane grid.

[0100] In some embodiments of the present application, in order to further accurately determine the target UV coordinates of the pixel point, when the depth difference between the first depth value and the second depth value of the j-th depth layer is less than or equal to 0, determining the target UV coordinates of the pixel point according to the UV coordinates of the first projection point on the plane grid may specifically include:

[0101] When the depth difference between the first depth value and the second depth value of the j-th depth layer is equal to 0, the UV coordinate of the first projection point on the plane grid is used as the target UV coordinate of the pixel point;

[0102] When the depth difference between the first depth value and the second depth value of the j-th depth layer is less than 0, the target UV coordinate of the pixel point is determined according to the UV coordinate of the first projection point on the plane grid and the UV coordinate of the target sight vector and the projection point of the j-1-th depth layer in the N depth layers on the plane grid.

[0103] In some embodiments of the present application, if the depth difference between the first depth value and the second depth value of the j-th depth layer is equal to 0, that is, the first depth value is equal to the second depth value, then it means The intersection point with the object in the 3D scene is located at At the intersection with the j-th depth layer, the UV coordinate of the first projection point on the plane grid can be used as the target UV coordinate of the pixel point.

[0104] If the depth difference between the first depth value and the second depth value of the j-th depth layer is less than 0, that is, the first depth value is less than the second depth value, it means that the intersection of V' and the object in the three-dimensional scene is between the intersection of V' and the j-th depth layer and the intersection of V' and the j-1-th depth layer, such as Figure 6 As shown, Figure 6 for Figure 5 The simplified schematic diagram is Figure 6 In the example, the intersection point P' between V' and the object in the three-dimensional scene is located between the intersection point P6' of V' and the intersection point P5' of the sixth depth layer. Figure 6 As shown, P6" and P5" are the intersection points of P6' and P5' with the objects in the three-dimensional scene, that is, the set of depth values ​​of P5 and P6 in the depth map of image 31 (ie Figure 3 The depth position sampled in (label 32).

[0105] In the embodiments of the present application, the target UV coordinates of the pixel points can be accurately determined according to different situations of the depth difference.

[0106] In some embodiments of the present application, when it is determined that the first depth value is less than the second depth value, the target UV coordinate of the pixel point can be determined according to the UV coordinate of the first projection point on the plane grid, and the UV coordinate of the projection point of the target sight vector and the j-1th depth layer in the N depth layers on the plane grid. For details, continue to refer to Figure 6 , can be layer 6 or layer 5 and The projection point P5 or P6 corresponding to the intersection point replaces the projection point P of P', and the UV coordinates of P5 or the UV coordinates of P6 are used as the final texture sampling coordinates of P0.

[0107] In the above, the UV coordinates of P5 or P6 are used as the final texture sampling coordinates of P0. The final texture sampling coordinates of P0 may not be very accurate. Therefore, in order to improve the accuracy of the final texture sampling coordinates of P0, the texture sampling coordinates corresponding to P6' or the texture sampling coordinates corresponding to P5' can be linearly interpolated to obtain the final texture sampling coordinates of P0, that is, the line between P6" and P5" is used. The intersection point P m 'Replace P', that is, P m 'The projection point P m The UV coordinates are used as the final texture sampling coordinates of P0.

[0108] That is, in some embodiments of the present application, in order to further accurately determine the target UV coordinates of the pixel point, the target UV coordinates of the pixel point are determined according to the UV coordinates of the first projection point on the plane grid, and the target sight line vector and the UV coordinates of the projection point of the j-1th depth layer in the N depth layers on the plane grid, which may specifically include:

[0109] Interpolate the UV coordinates of the first projection point on the plane grid, and the UV coordinates of the target sight vector and the projection point of the j-1th depth layer in the N depth layers on the plane grid to obtain the interpolated UV coordinates;

[0110] Use the interpolated UV coordinates as the target UV coordinates of the pixel.

[0111] In some embodiments of the present application, the UV coordinates of the first projection point on the plane grid, as well as the UV coordinates of the target sight vector and the projection point of the j-1th depth layer in the N depth layers on the plane grid can be interpolated to obtain the interpolated UV coordinates, and then the interpolated UV coordinates are used as the target UV coordinates of the pixel point.

[0112] In an embodiment of the present application, an interpolation calculation is performed by using the UV coordinates of the first projection point on the plane grid, and the UV coordinates of the target sight vector and the projection point of the j-1th depth layer in the N depth layers on the plane grid to obtain the interpolated UV coordinates, and the interpolated UV coordinates are used as the target UV coordinates of the pixel point, thereby improving the accuracy of determining the target UV coordinates of the pixel point.

[0113] Step 150: Sample the target color value of the pixel in the first rendered image according to the target UV coordinate.

[0114] The target color value may be the color value of a pixel sampled in the first rendered image according to the target UV coordinates, that is, the color value at a position corresponding to the target UV coordinates in the first rendered image.

[0115] In some embodiments of the present application, since the plane grid is parallel to the posture corresponding to the first rendered image, each pixel point in the plane grid can find a corresponding position in the first rendered image, so the target color value of each pixel point can be sampled in the first rendered image according to the target UV coordinate of each pixel point in the plane grid.

[0116] Step 160: Obtain a second rendered image according to the target color value corresponding to each pixel.

[0117] The second rendered image may be a rendered image at a second moment.

[0118] In order to better understand the solution of the embodiment of the present application, the image processing method provided by the embodiment of the present application is described below with specific examples. Figure 7 As shown, the image processing method may include steps 701 to 714.

[0119] Step 701: Obtain a first rendering image and a first depth image.

[0120] Step 702: Obtain a plane grid.

[0121] Step 703: Divide the depth values ​​of the plane grid into N layers.

[0122] Step 704: Create a sight line vector between the target pixel and the image acquisition device.

[0123] In step 704, the target pixel point may be any pixel point in the plane grid.

[0124] Step 705: Convert the sight line vector to the tangent space to obtain the target sight line vector.

[0125] Step 706: Normalize the target sight line vector.

[0126] In step 706, the target sight line vector is normalized to set the depth value range of the tangent space of the target sight line vector to between 0 and 1. For details, please refer to the description in the above embodiment, which will not be repeated here.

[0127] Step 707: Calculate the UV coordinates of the intersection of the target sight line vector and the j-th depth layer among the N depth layers projected onto the plane grid.

[0128] In step 707, the UV coordinates of the intersection of the target sight line vector and the jth depth layer among the N depth layers projected onto the plane grid can be calculated by referring to the calculation method of the UV coordinates of the intersection of the target sight line vector and the jth depth layer among the N depth layers projected onto the plane grid in the above embodiment, which will not be repeated here.

[0129] Step 708: Sample the first depth image according to the UV coordinates to obtain a first depth value.

[0130] Step 709, determine the size of the first depth value and the second depth value of the depth layer corresponding to the UV coordinate. If they are equal, execute step 710. If they are greater, continue processing the next depth layer and return to step 707. If they are less than, execute step 711.

[0131] Step 710: Use the UV coordinate as the target UV coordinate of the target pixel.

[0132] Step 711: linear interpolation is performed to obtain the target UV coordinates of the target pixel.

[0133] Step 712: Sample the first rendered image according to the target UV coordinates to obtain a target color value of the target pixel.

[0134] Step 713, determine whether all pixel points have been processed. If so, execute step 714. If not, calculate the next target pixel point, and return to execute step 704.

[0135] Step 714: Obtain a second rendered image according to the target color value corresponding to each pixel.

[0136] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0137] Figure 8 FIG. 1 is a schematic diagram showing the structure of an image processing device according to an exemplary embodiment. Figure 8 As shown, the image processing device 800 may include:

[0138] A first acquisition module 810 is used to acquire a first rendered image and a plane mesh, wherein the first rendered image is determined based on an image acquired by an image acquisition device with a first pose information at a first moment, the plane mesh is a mesh created at a second moment and parallel to a near clipping plane under the first pose information, and the pose information of the image acquisition device at the second moment is the second pose information;

[0139] A creation module 820, configured to create, for each pixel point in the plane grid, a sight line vector between the pixel point and the image acquisition device;

[0140] A division module 830, configured to divide the depth value of the plane grid into N layers, where N is a positive integer, and the depth value of each depth layer in the N depth layers is different;

[0141] A first determination module 840 is used to determine the target UV coordinate of the pixel point according to the UV coordinate projected onto the plane grid by the target sight line vector and the intersection points of the N depth layers, wherein the target sight line vector is the sight line vector converted into the tangent space;

[0142] A sampling module 850, configured to obtain a target color value of the pixel point by sampling in the first rendered image according to the target UV coordinate;

[0143] The second determination module 860 is used to obtain a second rendering image according to the target color value corresponding to each pixel point, where the second rendering image is the rendering image at the second moment.

[0144] In an embodiment of the present application, the depth values ​​of the plane grid corresponding to the first rendered image are layered, and for each pixel point of the plane grid, a sight vector between the pixel point and the image acquisition device is created, and then the sight vector is converted into the tangent space to obtain a target sight vector. In this way, the target UV coordinates of the pixel point can be determined based on the intersection of the target sight vector and the N depth layers projected onto the UV coordinates on the plane grid, thereby realizing pixel reprojection. Compared with the prior art in which reprojection calculation is performed on each pixel point in the first rendered image that is rendered for the image captured at the first moment, the scheme of the embodiment of the present application has a smaller amount of calculation. Compared with the prior art in which the image acquisition device is converted from the first pose information at the first moment to the second pose information at the second moment, an image buffer area needs to be added to the central processing unit to perform movement calculation of each pixel point when performing pixel reprojection calculation. The scheme of the embodiment of the present application does not need to perform movement calculation of each pixel point, nor does it need to increase the image buffer area, thereby saving memory usage. Furthermore, the solution of the embodiment of the present application is to determine the target UV coordinates of the pixel points based on the UV coordinates projected onto the plane grid based on the intersection of the sight vector and the N depth layers, and then add the target color value corresponding to each pixel point to the plane grid to obtain a second rendered image. That is, the solution of the embodiment of the present application is to directly start from the pixel points seen by the sight vector at the second moment, and sample from the first rendered image to achieve the purpose of reprojection, so the problem of holes formed by missing pixels will not occur, which solves the problem of holes formed due to missing pixels in the prior art.

[0145] In some embodiments of the present application, the first determination module 840 may include:

[0146] A first calculation unit is used to calculate the intersection point of the target sight line vector and the j-th depth layer among the N depth layers to obtain a first intersection point, where the initial value of j is 1 and j is a positive integer;

[0147] A second calculation unit, used for calculating a projection point of the first intersection point on the plane grid to obtain a first projection point;

[0148] A first acquisition unit, configured to acquire a first depth value of the first projection point in a first depth image according to a UV coordinate of the first projection point on the plane grid, where the first depth image is a depth image corresponding to the first rendered image;

[0149] a first determining unit, configured to determine, when a depth difference between the first depth value and the second depth value of the j-th depth layer is less than or equal to 0, a target UV coordinate of the pixel point according to the UV coordinate of the first projection point on the plane grid;

[0150] The second determination unit is used to update j to j+1 when the depth difference between the first depth value and the second depth value of the j-th depth layer is greater than 0, return to execute calculation of the intersection of the target sight line vector and the j-th depth layer among the N depth layers, calculate the projection point of the first intersection on the plane grid, and obtain the first projection point; when the depth difference between the first depth value and the second depth value of the j-th depth layer is less than or equal to 0, determine the target UV coordinate of the pixel point according to the UV coordinate of the first projection point on the plane grid until j=N.

[0151] In some embodiments of the present application, the above-mentioned device may further include:

[0152] A second acquisition module, configured to calculate an offset between the first projection point and the pixel point before acquiring a first depth value of the first projection point in the first depth image according to the UV coordinate of the first projection point on the plane grid;

[0153] The third determination module is used to determine the UV coordinates of the first projection point on the plane grid according to the offset and the position information of the pixel point.

[0154] In some embodiments of the present application, the first determining unit includes:

[0155] A first determining subunit is configured to use, when a depth difference between the first depth value and the second depth value of the j-th depth layer is equal to 0, the UV coordinate of the first projection point on the plane grid as the target UV coordinate of the pixel point;

[0156] The second determination subunit is used to determine the target UV coordinate of the pixel point according to the UV coordinate of the first projection point on the plane grid, and the UV coordinate of the projection point of the target line of sight vector and the j-1th depth layer in the N depth layers on the plane grid when the depth difference between the first depth value and the second depth value of the j-th depth layer is less than 0.

[0157] In some embodiments of the present application, the second determining subunit is specifically used for:

[0158] Interpolate the UV coordinates of the first projection point on the plane grid, and the UV coordinates of the target sight line vector and the projection point of the j-1th depth layer among the N depth layers on the plane grid to obtain interpolated UV coordinates;

[0159] The interpolated UV coordinates are used as the target UV coordinates of the pixel point.

[0160] The image processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmentedreality, AR) / virtual reality (virtualreality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personalcomputer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0161] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0162] The image processing device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0163] Alternatively, if Fig. 9 As shown, an embodiment of the present application further provides an electronic device 900, including a processor 901 and a memory 902, wherein the memory 902 stores programs or instructions that can be executed on the processor 901, and when the program or instructions are executed by the processor 901, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0164] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0165] Fig.10 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0166] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.

[0167] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.10 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0168] Among them, the processor 1010 is used to obtain a first rendered image and a plane grid, wherein the first rendered image is determined based on an image captured by an image acquisition device with a first pose information at a first moment, the plane grid is a grid created at a second moment and parallel to the near clipping plane under the first pose information, and the pose information of the image acquisition device at the second moment is the second pose information; for each pixel point in the plane grid, a sight vector between the pixel point and the image acquisition device is created; the depth value of the plane grid is divided into N layers, N is a positive integer, and the depth value of each depth layer in the N depth layers is different; according to the UV coordinates of the intersection of the target sight vector and the N depth layers projected onto the plane grid, the target UV coordinates of the pixel point are determined, and the target sight vector is the sight vector after the sight vector is converted to the tangent space; according to the target UV coordinates, the target color value of the pixel point is sampled in the first rendered image; according to the target color value corresponding to each pixel point, a second rendered image is obtained, and the second rendered image is the rendered image at the second moment.

[0169] In this way, by layering the depth value of the plane grid corresponding to the first rendered image, for each pixel point of the plane grid, a sight line vector between the pixel point and the image acquisition device is created, and then the sight line vector is converted into the tangent space to obtain the target sight line vector. In this way, the target UV coordinate of the pixel point can be determined based on the intersection of the target sight line vector and the N depth layers projected onto the UV coordinate on the plane grid, thereby realizing the reprojection of the pixel. Compared with the prior art in which reprojection calculation is performed on each pixel point in the first rendered image that is rendered by the image captured at the first moment, the scheme of the embodiment of the present application has a smaller amount of calculation. Compared with the prior art in which the image acquisition device is converted from the first pose information at the first moment to the second pose information at the second moment, an image buffer area needs to be added in the central processing unit to perform the movement calculation of each pixel point when performing the reprojection calculation of the pixel point. The scheme of the embodiment of the present application does not need to perform the movement calculation of each pixel point, nor does it need to increase the image buffer area, thereby saving memory usage. Furthermore, the solution of the embodiment of the present application is to determine the target UV coordinates of the pixel points based on the UV coordinates projected onto the plane grid based on the intersection of the sight vector and the N depth layers, and then add the target color value corresponding to each pixel point to the plane grid to obtain a second rendered image. That is, the solution of the embodiment of the present application is to directly start from the pixel points seen by the sight vector at the second moment, and sample from the first rendered image to achieve the purpose of reprojection, so the problem of holes formed by missing pixels will not occur, which solves the problem of holes formed due to missing pixels in the prior art.

[0170] Optionally, the processor 1010 is further configured to calculate the intersection of the target sight line vector and the j-th depth layer among the N depth layers to obtain a first intersection point, where the initial value of j is 1 and j is a positive integer; calculate the projection point of the first intersection point on the plane grid to obtain a first projection point; obtain a first depth value of the first projection point in a first depth image according to the UV coordinate of the first projection point on the plane grid, where the first depth image is a depth image corresponding to the first rendered image; when the depth difference between the first depth value and the second depth value of the j-th depth layer is less than or equal to 0, obtain a first depth value of the first projection point in the plane grid according to the UV coordinate of the first projection point on the plane grid. The UV coordinates on the plane grid are used to determine the target UV coordinates of the pixel point; when the depth difference between the first depth value and the second depth value of the j-th depth layer is greater than 0, j is updated to j+1, and the intersection of the target sight line vector and the j-th depth layer in the N depth layers is returned to calculate, and the projection point of the first intersection on the plane grid is calculated to obtain a first projection point; when the depth difference between the first depth value and the second depth value of the j-th depth layer is less than or equal to 0, the target UV coordinates of the pixel point are determined according to the UV coordinates of the first projection point on the plane grid, until j=N.

[0171] In this way, by calculating the intersection of the target sight line vector and the N depth layers layer by layer, calculating the projection point of the intersection on the plane grid, and then according to the UV coordinate of the projection point on the plane grid, the first depth value of the projection point in the first depth image can be obtained, and then it is determined whether the depth value of the projection point meets the requirements, and then the target UV coordinate of the pixel can be accurately determined. In addition, when calculating the intersection of the target sight line vector and the depth layer, there is no need to calculate all the intersections with each of the N depth layers at one time, which reduces the computing power and saves computing resources.

[0172] Optionally, the processor 1010 is further used to calculate an offset between the first projection point and the pixel point; and determine a UV coordinate of the first projection point on the plane grid according to the offset and position information of the pixel point.

[0173] In this way, the UV coordinates of the first projection point on the plane grid can be accurately determined through the offset between the first projection point and the pixel point and the position information of the pixel point, and then the target UV coordinates of the pixel point can be accurately determined based on the UV coordinates of the first projection point on the plane grid.

[0174] Optionally, the processor 1010 is further used to, when the depth difference between the first depth value and the second depth value of the j-th depth layer is equal to 0, use the UV coordinates of the first projection point on the plane grid as the target UV coordinates of the pixel point; when the depth difference between the first depth value and the second depth value of the j-th depth layer is less than 0, determine the target UV coordinates of the pixel point according to the UV coordinates of the first projection point on the plane grid, and the target line of sight vector and the UV coordinates of the projection point of the j-1-th depth layer among the N depth layers on the plane grid.

[0175] In this way, the target UV coordinates of the pixel points can be accurately determined according to different situations of the depth difference.

[0176] Optionally, the processor 1010 is further used to interpolate the UV coordinates of the first projection point on the plane grid, and the UV coordinates of the target sight vector and the projection point of the j-1th depth layer in the N depth layers on the plane grid to obtain interpolated UV coordinates; and use the interpolated UV coordinates as the target UV coordinates of the pixel point.

[0177] In this way, the interpolated UV coordinates are obtained by interpolating the UV coordinates of the first projection point on the plane grid, the target sight vector and the UV coordinates of the projection point of the j-1th depth layer in the N depth layers, and the interpolated UV coordinates are used as the target UV coordinates of the pixel point, thereby improving the accuracy of determining the target UV coordinates of the pixel point.

[0178] It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of the static picture or video obtained by the image capture device (such as a color camera) in the video capture mode or the image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0179] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0180] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.

[0181] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0182] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0183] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0184] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0185] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0186] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0187] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0188] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first rendered image and a plane mesh, wherein the first rendered image is determined based on an image acquired by an image acquisition device with a first pose information at a first moment, the plane mesh is a mesh created at a second moment and parallel to a near clipping plane under the first pose information, and the pose information of the image acquisition device at the second moment is the second pose information; For each pixel point in the plane grid, create a sight line vector between the pixel point and the image acquisition device; Dividing the depth value of the plane grid into N layers, where N is a positive integer, and the depth value of each depth layer in the N depth layers is different; Determine the target UV coordinate of the pixel point according to the UV coordinate projected onto the plane grid by the target sight line vector and the intersection of the N depth layers, wherein the target sight line vector is the sight line vector after the sight line vector is converted into the tangent space; Sampling the first rendered image according to the target UV coordinates to obtain a target color value of the pixel; A second rendered image is obtained according to the target color value corresponding to each pixel, where the second rendered image is the rendered image at the second moment.

2. The method according to claim 1, characterized in that Determining the target UV coordinates of the pixel point according to the UV coordinates projected onto the plane grid by the intersection of the target sight line vector and the N depth layers includes: Calculate the intersection of the target sight line vector and the jth depth layer among the N depth layers to obtain the first intersection point, where the initial value of j is 1 and j is a positive integer; Calculate the projection point of the first intersection point on the plane grid to obtain a first projection point; Acquire, according to the UV coordinates of the first projection point on the plane grid, a first depth value of the first projection point in a first depth image, where the first depth image is a depth image corresponding to the first rendered image; When a depth difference between the first depth value and the second depth value of the j-th depth layer is less than or equal to 0, determining a target UV coordinate of the pixel point according to the UV coordinate of the first projection point on the plane grid; When the depth difference between the first depth value and the second depth value of the j-th depth layer is greater than 0, j is updated to j+1, and the calculation of the intersection of the target sight line vector and the j-th depth layer in the N depth layers is returned, and the projection point of the first intersection on the plane grid is calculated to obtain the first projection point; when the depth difference between the first depth value and the second depth value of the j-th depth layer is less than or equal to 0, the target UV coordinate of the pixel point is determined according to the UV coordinate of the first projection point on the plane grid until j=N.

3. The method according to claim 2, characterized in that Before acquiring a first depth value of the first projection point in the first depth image according to the UV coordinate of the first projection point on the plane grid, the method further includes: Calculating the offset between the first projection point and the pixel point; Determine the UV coordinate of the first projection point on the plane grid according to the offset and the position information of the pixel point.

4. The method according to claim 2, characterized in that: The determining, when a depth difference between the first depth value and the second depth value of the j-th depth layer is less than or equal to 0, a target UV coordinate of the pixel point according to the UV coordinate of the first projection point on the plane grid, comprises: When a depth difference between the first depth value and the second depth value of the j-th depth layer is equal to 0, using the UV coordinate of the first projection point on the plane grid as the target UV coordinate of the pixel point; When the depth difference between the first depth value and the second depth value of the j-th depth layer is less than 0, the target UV coordinate of the pixel point is determined according to the UV coordinate of the first projection point on the plane grid, and the target sight vector and the UV coordinate of the projection point of the j-1-th depth layer in the N depth layers on the plane grid.

5. The method according to claim 4, characterized in that The determining the target UV coordinate of the pixel point according to the UV coordinate of the first projection point on the plane grid and the UV coordinate of the projection point of the target sight vector and the j-1th depth layer in the N depth layers on the plane grid includes: Interpolate the UV coordinates of the first projection point on the plane grid, and the UV coordinates of the target sight line vector and the projection point of the j-1th depth layer among the N depth layers on the plane grid to obtain interpolated UV coordinates; The interpolated UV coordinates are used as the target UV coordinates of the pixel point.

6. An image processing device, characterized in that: The device comprises: A first acquisition module is used to acquire a first rendered image and a plane mesh, wherein the first rendered image is determined based on an image acquired by an image acquisition device with a first pose information at a first moment, the plane mesh is a mesh created at a second moment and parallel to a near clipping plane under the first pose information, and the pose information of the image acquisition device at the second moment is second pose information; A creation module, used for creating a sight line vector between each pixel point in the plane grid and the image acquisition device; A division module, used for dividing the depth value of the plane grid into N layers, where N is a positive integer, and the depth value of each depth layer in the N depth layers is different; A first determination module is used to determine the target UV coordinates of the pixel point according to the UV coordinates projected onto the plane grid by the target sight line vector and the intersection points of the N depth layers, wherein the target sight line vector is the sight line vector after the sight line vector is converted into the tangent space; A sampling module, used for sampling the first rendered image according to the target UV coordinates to obtain the target color value of the pixel point; The second determining module is used to obtain a second rendering image according to the target color value corresponding to each pixel point, where the second rendering image is the rendering image at the second moment.

7. The device according to claim 6, characterized in that The first determining module comprises: A first calculation unit is used to calculate the intersection point of the target sight line vector and the j-th depth layer among the N depth layers to obtain a first intersection point, where the initial value of j is 1 and j is a positive integer; A second calculation unit, used for calculating a projection point of the first intersection point on the plane grid to obtain a first projection point; A first acquisition unit, configured to acquire a first depth value of the first projection point in a first depth image according to a UV coordinate of the first projection point on the plane grid, where the first depth image is a depth image corresponding to the first rendered image; a first determining unit, configured to determine, when a depth difference between the first depth value and the second depth value of the j-th depth layer is less than or equal to 0, a target UV coordinate of the pixel point according to the UV coordinate of the first projection point on the plane grid; The second determination unit is used to update j to j+1 when the depth difference between the first depth value and the second depth value of the j-th depth layer is greater than 0, return to execute calculation of the intersection of the target sight line vector and the j-th depth layer among the N depth layers, calculate the projection point of the first intersection on the plane grid, and obtain the first projection point; when the depth difference between the first depth value and the second depth value of the j-th depth layer is less than or equal to 0, determine the target UV coordinate of the pixel point according to the UV coordinate of the first projection point on the plane grid until j=N.

8. The device according to claim 7, characterized in that The device also includes: A second acquisition module, configured to calculate an offset between the first projection point and the pixel point before acquiring a first depth value of the first projection point in the first depth image according to the UV coordinate of the first projection point on the plane grid; The third determination module is used to determine the UV coordinates of the first projection point on the plane grid according to the offset and the position information of the pixel point.

9. The device according to claim 8, characterized in that The first determining unit includes: A first determining subunit is configured to use, when a depth difference between the first depth value and the second depth value of the j-th depth layer is equal to 0, the UV coordinate of the first projection point on the plane grid as the target UV coordinate of the pixel point; The second determination subunit is used to determine the target UV coordinate of the pixel point according to the UV coordinate of the first projection point on the plane grid, and the UV coordinate of the projection point of the target line of sight vector and the j-1th depth layer in the N depth layers on the plane grid when the depth difference between the first depth value and the second depth value of the j-th depth layer is less than 0.

10. The device according to claim 9, characterized in that The second determining subunit is specifically used for: Interpolate the UV coordinates of the first projection point on the plane grid, and the UV coordinates of the target sight line vector and the projection point of the j-1th depth layer among the N depth layers on the plane grid to obtain interpolated UV coordinates; The interpolated UV coordinates are used as the target UV coordinates of the pixel point.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 5 are implemented.

12. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image processing method according to any one of claims 1 to 5 are implemented.

Images