Image processing method and device and storage medium

Abstract

The invention provides an image processing method and device and a storage medium, and the method comprises the steps: carrying out the depth estimation of an initial image, and obtaining an initial depth map; performing expansion processing based on the initial depth map to obtain a target depth map texture; and rendering based on the viewpoint offset determined by the interaction event, the target depth map texture and the initial image to obtain a target rendering frame. According to the method, the initial depth map obtained by estimating the depth of the initial image is subjected to expansion processing, so that one or more pixels are expanded from a region with brighter pixels to a region with darker pixels, and a depth region slightly larger than the visual contour of an object is formed; and then, rendering is performed by using the viewpoint offset, the target depth map texture and the initial image to determine a target rendering frame, so that the problem of visual artifacts during rendering can be avoided.

Description

Technical Field

[0001] This invention relates to the intersection of computer graphics and artificial intelligence, and in particular to an image processing method, apparatus and storage medium. Background Technology

[0002] Predicting depth information from a single 2D image using artificial intelligence models offers new possibilities for achieving pseudo-3D effects in lightweight environments such as the web. Developers can add a sense of depth to ordinary images without complex 3D modeling. Existing technologies inherently exhibit "uncertainty" when predicting the precise contours of objects. Specifically, at the boundary between foreground and background, AI-generated depth maps often suffer from noise, jagged edges, blurriness, or fluctuating depth values ​​between the two.

[0003] When a depth map with uncertain edges is rendered directly, the resulting image will exhibit severe visual artifacts such as violent flickering at the edges of objects, holes, and the background "transparent" into the foreground, which greatly damages the immersive experience. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide an image processing method, apparatus, and storage medium to solve the problem of visual artifacts occurring when rendering depth maps with edge uncertainties in the prior art.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] The first aspect illustrates an image processing method, the method comprising:

[0007] Get the initial image and interaction events input by the user;

[0008] Depth estimation is performed on the initial image to obtain an initial depth map;

[0009] Dilation processing is performed on the initial depth map to obtain the target depth map texture;

[0010] Rendering is performed based on the viewpoint offset determined by the interaction event, the target depth map texture, and the initial image to obtain the target rendering frame.

[0011] Optionally, depth estimation is performed on the initial image to obtain an initial depth map, including:

[0012] The initial image is decoded to obtain the corresponding source image object;

[0013] Convert the source image object into a tensor;

[0014] The pre-loaded model object is invoked to perform depth prediction on the tensor, and the original depth tensor is obtained;

[0015] The original depth tensor is restored to obtain the initial depth map.

[0016] Optionally, dilation processing is performed based on the initial depth map to obtain the target depth map texture, including:

[0017] The initial depth map texture is determined by processing the texture corresponding to the initial depth map.

[0018] For each pixel in the initial depth map texture, determine the corresponding texture coordinates of the pixel in the initial depth map texture;

[0019] Based on the preset structural elements and the texture coordinates of the pixel, the pixel is dilated in the initial depth map texture to determine the corresponding dilated pixel and the corresponding texture coordinates.

[0020] The initial depth map texture is updated based on the texture coordinates of each pixel and the texture coordinates of its corresponding dilated pixel to obtain the target depth map texture.

[0021] Optionally, the initial depth map texture is updated based on the texture coordinates of each pixel and the texture coordinates of its corresponding dilated pixel to obtain the target depth map texture, including:

[0022] For each pixel of the initial depth map texture, the initial depth value of the pixel is read from the initial depth map texture based on the texture coordinates of the pixel;

[0023] The initial depth value of the dilated pixel is read from the initial depth map texture according to the texture coordinates of the dilated pixel;

[0024] For each dilated pixel, the target depth value of the pixel is determined based on the initial depth value of the dilated pixel and the initial depth value of the pixel.

[0025] The initial depth map texture is updated based on the target depth value of each pixel to obtain the target depth map texture.

[0026] Optionally, based on the viewpoint offset determined by the interaction event, the target depth map texture, and the initial image, rendering is performed to obtain a target rendering frame, including:

[0027] The parameters corresponding to the interactive event are converted into a first vector, and the first vector is used as the viewpoint offset.

[0028] Rendering operations are performed based on the source image object determined by the initial image, the target depth map texture, and the viewpoint offset to obtain the target rendering frame.

[0029] Optionally, a rendering operation is performed based on the source image object determined from the initial image, the target depth map texture, and the viewpoint offset to obtain the target rendering frame, including:

[0030] For each pixel in the target depth map texture, a preset ray is invoked to render the pixel based on the pixel's texture coordinates and the viewpoint offset to determine the corresponding sampling coordinates;

[0031] Based on the sampling coordinates, the color corresponding to the sampling coordinates is obtained from the initial image and used as the target color of the pixel;

[0032] The target color of each pixel in the target depth map texture is collected to obtain the target rendering frame.

[0033] Optionally, for each pixel in the target depth map texture, a preset ray is invoked to render the pixel based on the pixel's texture coordinates and the viewpoint offset to determine the corresponding sampling coordinates, including:

[0034] For each pixel in the target depth map texture, ray stepping information is determined based on the texture coordinates of the pixel and the viewpoint offset;

[0035] A preset ray is invoked to simulate viewpoint movement on the target depth map texture according to the ray step information;

[0036] In each step of the simulated viewpoint movement, the current ray depth and texture coordinates corresponding to the sampling point at the intersection of the preset ray and the target depth map texture are obtained;

[0037] The sampling coordinates are determined based on the current ray depth and texture coordinates corresponding to the sampling point, and the target depth map texture.

[0038] Optionally, the sampling coordinates are determined based on the current ray depth and texture coordinates corresponding to the sampling point, and the target depth map texture, including:

[0039] Calculate the disparity parameters based on the current ray depth and viewpoint offset corresponding to the sampling point;

[0040] The initial coordinates are determined based on the texture coordinates and disparity parameters corresponding to the sampling points;

[0041] The scene depth of the sampling point is read from the target depth map texture using the initial coordinates;

[0042] Determine whether the initial coordinates need to be updated based on the scene depth and the current ray depth corresponding to the sampling point;

[0043] If necessary, the initial coordinates can be used as sampling coordinates.

[0044] A second aspect discloses an image processing apparatus, the apparatus comprising:

[0045] The acquisition unit is used to acquire the initial image and interaction events input by the user;

[0046] A depth estimation unit is used to perform depth estimation on the initial image to obtain an initial depth map;

[0047] The processing unit is used to perform dilation processing based on the initial depth map to obtain the target depth map texture;

[0048] The rendering unit is used to render the target rendering frame based on the viewpoint offset determined by the interaction event, the target depth map texture, and the initial image.

[0049] The third aspect discloses a storage medium including a stored program, wherein, when the program is executed, it controls the device where the storage medium is located to perform an image processing method as described in any of the first aspects of the present invention, 8.

[0050] Based on the above embodiments of the present invention, an image processing method, apparatus, and storage medium are provided. The method includes: acquiring an initial image and an interaction event input by a user; performing depth estimation on the initial image to obtain an initial depth map; performing dilation processing on the initial depth map to obtain a target depth map texture; and rendering based on a viewpoint offset determined by the interaction event, the target depth map texture, and the initial image to obtain a target rendering frame. In this embodiment of the present invention, the initial depth map is uploaded to a texture to perform dilation processing on the texture of the initial depth map, so that brighter pixel areas expand one or more pixels into darker areas to form a depth region slightly larger than the visual outline of the object; then, the target rendering frame is determined by rendering using the viewpoint offset, the target depth map texture, and the initial image to avoid visual artifacts during rendering. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0052] Figure 1 This is a schematic diagram illustrating the interaction between the client and the server in an embodiment of the present invention;

[0053] Figure 2This is a schematic flowchart illustrating an image processing method according to an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram of the image processing architecture shown in an embodiment of the present invention;

[0055] Figure 4 This is a schematic diagram of the structure of an image processing device according to an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0058] It should be noted that the descriptions involving "first," "second," etc., in this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0059] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0060] As the background technology indicates, the conventional approach to addressing data inaccuracy is to perform smoothing processes, such as Gaussian blurring. However, this method indiscriminately blurs all depth details, resulting in a blurry and indistinct pseudo-3D effect lacking depth. Therefore, there is an urgent need in this field for a technical solution that can preserve depth details while specifically addressing the uncertainty at the edges of AI depth maps.

[0061] See Figure 1 This is a schematic diagram illustrating the interaction between the client and the server in an embodiment of the present invention.

[0062] The client 10 is communicatively connected to the server 20;

[0063] There are multiple clients, including client 10.

[0064] Based on the above architecture, the specific implementation of image processing, namely the process of AI pseudo-3D rendering method based on depth safe zones, is as follows: Figure 2 The diagram shown is a flowchart illustrating an image processing method according to an embodiment of the present invention. The method includes:

[0065] Step S201: Obtain the initial image and interaction events input by the user;

[0066] Optionally, during the pseudo-3D rendering of the image, the user can specify a 2D source image through the client and trigger interactive events for the 2D source image through the client.

[0067] The 2D source image is the initial image.

[0068] It should be noted that the interactive events include operations such as mouse movement, touch swipe, and client device rotation.

[0069] In the specific implementation step S201, the server captures the initial image specified by the user for the client, as well as the triggered interaction events, through a listener.

[0070] Optionally, before executing step S201, i.e. before performing pseudo-3D rendering of the image, the 2D rendering engine is triggered to start, so that a 2D rendering engine (such as Pixi.js) is loaded and initialized in the client (such as a web browser), and a canvas is created to display the final effect; this indicates that the process of performing pseudo-3D rendering of the image has begun.

[0071] Step S202: Perform depth estimation on the initial image to obtain an initial depth map (initialDepthMapData).

[0072] It should be noted that the specific implementation of step S202 includes the following steps.

[0073] Step S11: Decode the initial image to obtain the corresponding source image object sourceImageObject;

[0074] In the specific implementation step S11, the user-specified 2D source image is loaded and decoded into a pixel data format accessible to the program to obtain the source image object sourceImageObject.

[0075] The source image object can be a webpage image processing object, HTMLImageElement.

[0076] Step S12: Convert the source image object into a tensor;

[0077] In this application, since the model encapsulated in the aiModel object cannot directly process the original source image object sourceImageObject, a series of preprocessing steps must be performed on the source image before depth estimation, as shown in steps S21 to S22, and then it is converted into a standardized numerical format that the model can understand, namely a tensor.

[0078] It should be noted that the following steps are included before the specific implementation of step S12.

[0079] Step S21: Adjust the size of the source image object to obtain a source image object of a preset size;

[0080] In the specific implementation of step S21, regardless of the size of the original image, a size adjustment operation needs to be performed to ensure the consistency of the input data dimensions; specifically, the size of the source image object sourceImageObject is adjusted to the preset size used during aiModel training to obtain a source image object of the preset size.

[0081] It should be noted that the preset size is set in advance based on multiple experiments, and can generally be set to 384x384 pixels.

[0082] Step S22: Convert the pixels of the source image object of the preset size to obtain a source image object that conforms to the preset data range.

[0083] In the specific implementation step S22, for each pixel in the source image object of the preset size, the pixel value is scaled to a preset range; then, the color channel of the pixel value is standardized according to the mean and standard deviation used in the model training, so as to obtain a source image object that conforms to the preset data range.

[0084] It should be noted that the preset interval is set in advance based on multiple experiments, and can generally be set to [0, 1].

[0085] The preset data range refers to the expected range of values ​​for the model, which is set in advance based on multiple experiments.

[0086] This application can eliminate the influence of external factors such as light on the model's predictions and interference through the above-mentioned methods.

[0087] In the specific implementation of step 12, the source image object that conforms to the preset data range, that is, the image data that has been resized and normalized, is converted into a multidimensional data that conforms to the preset dimensional order, namely a tensor.

[0088] It should be noted that the preset dimension order refers to the multidimensional data set based on multiple experiments, such as a multidimensional array of [batch size, number of channels, height, width].

[0089] Step S13: Call the pre-loaded model object to perform depth prediction on the tensor to obtain the original depth tensor.

[0090] The model object encapsulates a deep learning model, which is trained based on historical multidimensional data.

[0091] In the specific implementation of step S13, firstly, the processed input tensor is passed to the inference function of the aiModel object (for example, a method named .predict() or .run()). The aiModel object calls its internally encapsulated deep learning model to perform depth prediction on the tensor, obtaining an output tensor containing the prediction results, i.e., the original depth tensor.

[0092] It should be noted that deep learning models include AI inference libraries. AI inference libraries call on underlying computing resources (CPU or GPU) to efficiently perform all mathematical operations in the network. In other words, they perform deep predictions on tensors. After execution, aiModel returns an output tensor containing the prediction results.

[0093] Each value in the output tensor corresponds to the original depth prediction value at a location on the initial depth map.

[0094] Step S14: Restore the original depth tensor to obtain the initial depth map initialDepthMapData.

[0095] In the specific implementation of step S14, firstly, since the size of the depth map output by the model may be smaller than the size of the original image, in order to correspond to each pixel of the source image, the size of the original depth tensor is restored to the size of the source image object sourceImageObject.

[0096] Specifically, the output depth map needs to be enlarged using an interpolation algorithm (such as bilinear interpolation) to restore it to the same original resolution as the sourceImageObject, so that the size of the original depth tensor is consistent with the size of the source image object.

[0097] Next, the values ​​in the original depth tensor are formatted to obtain the initial depth map, initialDepthMapData.

[0098] Specifically, values ​​are extracted from the original depth tensor after size restoration, and arranged into a one-dimensional array. The array is then used to form the initial depth map, initialDepthMapData.

[0099] Each element of the array represents the depth value of the corresponding location in the source image object.

[0100] In this embodiment of the invention, the source image object is preprocessed to convert it into a standard tensor; then, aiModel is called to perform depth estimation through the model to infer and calculate the original depth tensor; finally, through post-processing, formatted initial depth map data initialDepthMapData corresponding one-to-one with the pixels of the source image is generated. At this time, the initial depth map contains data with edge uncertainty.

[0101] Step S203: Perform dilation processing based on the initial depth map to obtain the target depth map texture.

[0102] It should be noted that the specific implementation of step S203 includes the following steps.

[0103] Step S31: Process the texture corresponding to the initial depth map to determine the initial depth map texture;

[0104] In the specific implementation step S31, the server calls the rendering engine's application programming interface API to create a new texture object in the GPU memory and uploads the initial depth map to this new texture object to form the initial depth map texture initialDepthMapTexture.

[0105] Step S32: For each pixel of the initial depth map texture, determine the texture coordinates corresponding to the pixel in the initial depth map texture;

[0106] In the specific implementation of step S32, a floating-point variable is first set, for example, float maxDepth = 0.0; for each pixel of the initial depth map texture, i.e., the pixel point, the texture coordinates (UV) corresponding to the pixel are first determined by the texture coordinates (UV) corresponding to the pixel, which is the position of the pixel point;

[0107] The texture coordinates (UV) point to the position in the initial depth map texture corresponding to the current output pixel.

[0108] Step S33: Based on the preset structural element and the texture coordinates of the pixel, the pixel is dilated in the initial depth map texture to determine the corresponding dilated pixel and the corresponding texture coordinates.

[0109] It should be noted that the preset structural element is a pre-set pixel matrix.

[0110] In the specific implementation of step S33, the shader needs to check the pixels defined by the preset structuring element in sequence. Taking a 3x3 square kernel as an example, the 3x3 square is treated as a matrix, meaning that the preset structuring element can be set as a matrix;

[0111] The shader performs dilation on the pixel according to its position in the initial depth map texture, determines the corresponding dilated pixel and its texture coordinates. Specifically, the center point of the preset structuring element is aligned with the pixel in the initial depth map texture to perform dilation operation. The effect of the dilation operation is observed, that is, the neighboring pixels adjacent to the center point covered by this matrix in the initial depth map texture (for example, for a 3x3 matrix, it is the center point and its 8 surrounding neighbors, i.e., neighboring pixels) are observed, and the neighboring pixels are used as dilated pixels.

[0112] Next, each dilated pixel is traversed to obtain its texture width (textureWidth) and texture height (textureHeight) in the coordinate system corresponding to the initial depth map texture. Then, the texel size corresponding to the dilated pixel can be calculated using vec2(1.0 / textureWidth, 1.0 / textureHeight). Then, by adding or subtracting a preset integer multiple of the texel size from the texture coordinates currentUV at the center point, the accurate texture coordinates of the dilated pixel can be obtained.

[0113] Step S34: Update the initial depth map texture based on the texture coordinates of each pixel and the texture coordinates of the corresponding dilated pixel to obtain the target depth map texture.

[0114] It should be noted that the specific implementation of step S34 includes the following steps.

[0115] Step S341: For each pixel of the initial depth map texture, read the initial depth value of the pixel from the initial depth map texture based on the texture coordinates of the pixel;

[0116] Specifically, the initial depth value of the pixel is read from the texture coordinates of the pixel in the initial depth map texture `initialDepthMapTexture`. That is, the initial depth value is obtained from the `initialDepthMapTexture` according to the pixel's position in the initial depth map texture, i.e., its texture coordinates. Then, the initial depth value is assigned to `maxDepth`. This ensures that the depth of all neighboring (dilated) pixels is smaller than that of the center point, and the final result is at least the depth of the center point itself.

[0117] Step S342: Read the initial depth value of the dilated pixel from the initial depth map texture according to the texture coordinates of the dilated pixel;

[0118] In the specific implementation step S342, texture2D(initialDepthMapTexture,neighborUV) is used to sample the depth value neighborDepth corresponding to the texture coordinates of the dilated pixels in the initial depth map texture, and this value is used as the initial depth value.

[0119] Step S343: For each dilated pixel, determine the target depth value of the pixel based on the initial depth value of the dilated pixel and the initial depth value of the pixel.

[0120] In the specific implementation step S343, the initial depth value of the pixel and each adjacent dilated pixel are compared cyclically until the largest initial depth value is determined; and the largest initial depth value is taken as the target depth value of the pixel.

[0121] Specifically, all neighbors are traversed through a nested loop (or by direct expansion). For example, from the top left corner `(-1,-1)` to the bottom right corner `(1, 1)`; taking this as an example (not the first comparison), the initial depth value `neighborDepth` of the dilated pixel is compared with the current depth value `maxDepth`; if the depth value `neighborDepth` of a neighboring pixel is greater than the current depth value `maxDepth`, then `maxDepth = neighborDepth` is updated, that is, the depth value `neighborDepth` of the neighboring pixel is used as the current depth value `maxDepth`.

[0122] When all neighboring pixels have finished cycling, the maxDepth variable will store the maximum depth value among all 9 pixels in the entire 3x3 neighborhood (including the center point), and use it as the target depth value of the pixel.

[0123] It should be noted that if this is the first comparison, the current depth value maxDepth is the initial depth value of the pixel.

[0124] Finally, the shader uses the calculated target depth value, maxDepth, as the color output of the current pixel, and so on, to determine the target depth value of each pixel in the initial depth map texture. In other words, the GPU efficiently calculates the maximum depth in the neighborhood of each pixel in the output texture.

[0125] Since the output texture only cares about depth (usually a single channel), this value is written to its red channel, and other channels can be assigned any value.

[0126] Step S344: Update the initial depth map texture based on the target depth value of each pixel to obtain the target depth map texture.

[0127] In the specific implementation of step S344, the morphological dilation operation of each pixel in the initial depth map texture is completed in accordance with the above steps S341 to S343, generating a stabilizedDepthMapTexture containing a "depth safe zone".

[0128] In other words, from the initial 9 pixels of the depth map texture, find the pixel value with the highest brightness (brightest), that is, the highest pixel value; finally, use the highest pixel value as the new depth value of the pixel, that is, the target depth value, thereby creating a depth safe zone, which is used as the new pixel value at the pixel position in the output image; repeat the above process on every pixel of the entire image to complete the dilation operation of the entire image.

[0129] In other words, for each pixel in the initial depth map texture, we look at its depth value and that of its neighbors, and then take the largest depth value (that is, the depth closest to the foreground) as the new depth value for this pixel; to create a "depth safe zone";

[0130] In this embodiment of the invention, uncertain noise that oscillates between foreground and background depths is generated at the edges of foreground objects. A dilation operation expands the edges of the foreground object (high depth / high brightness) outwards; this expansion "covers" the uncertain, noisy pixels originally located at the edges, forming a "buffer zone"—a "depth safe zone"—around the object's original outline, with a width determined by the size of the "structuring element" and a stable depth value equal to the foreground depth. This ensures that in subsequent rendering, even if sampling points fall into the original, uncertain edge region, the correct foreground depth can be stably obtained, thereby fundamentally eliminating flickering and hole-like issues.

[0131] It should be noted that morphological dilation expands the area of ​​highlighted regions (white parts) in an image by selecting a structuring element and sliding it across the image, assigning the maximum pixel value of the covered area to the anchor point.

[0132] Step S204: Render the target rendering frame based on the viewpoint offset determined by the interaction event, the target depth map texture, and the initial image.

[0133] Step S41: Convert the parameters corresponding to the interactive event into a first vector, and use the first vector as the viewpoint offset;

[0134] The first vector is a two-dimensional vector.

[0135] In the specific implementation step S41, the parameters corresponding to the interactive event are obtained, and then the parameters are converted into a two-dimensional vector, and the two-dimensional vector is used as the viewpoint offset.

[0136] It should be noted that the parameters here include screen coordinates and / or rotation angle;

[0137] Viewpoint offset refers to the offset of the user's viewpoint relative to the center; that is, it is the offset of the current interaction state relative to the initial center viewpoint.

[0138] For example, when the user is not interacting or the mouse is in the center, the viewpoint offset is {x: 0, y: 0}. When an interaction event is detected, such as when the user moves the mouse to the right side of the screen, there is a viewpoint offset from {x: 0, y: 0}, meaning the viewpoint offset will have a new value, such as {x: 0.5, y: 0}. This allows the renderer to subsequently render the image in this frame based on the viewpoint offset {x: 0.5, y: 0}, simulating the parallax effect of viewing from left to right.

[0139] Step S42: Perform a rendering operation based on the source image object determined by the initial image, the target depth map texture, and the viewpoint offset to obtain the target rendering frame.

[0140] It should be noted that the specific implementation of step S42 includes the following steps.

[0141] Step S421: For each pixel in the target depth map texture, render the pixel using a preset ray based on the pixel's texture coordinates and the viewpoint offset, and determine the corresponding sampling coordinates.

[0142] It should be noted that the specific implementation of step S421 includes the following steps.

[0143] Step S4211: For each pixel in the target depth map texture, determine the ray stepping information based on the texture coordinates of the pixel and the viewpoint offset;

[0144] It should be noted that the light stepping information includes the starting position of the light and the direction of the light.

[0145] In the specific implementation step S4211, the difference between the texture coordinates of the pixel and the viewpoint offset is calculated to obtain the starting position of the light ray. That is, the user's line of sight moves from the viewpoint offset to the starting position of the light ray. Next, a direction perpendicular to the screen (Z-axis direction) is set as the direction of the light ray.

[0146] Step S4212: Call a preset ray to simulate viewpoint movement on the target depth map texture according to the ray stepping information;

[0147] In the specific implementation step S4212, the depth currentRayDepth of the current ray at the starting position is set, that is, the preset value, the current step size and the number of loops are set; the ray is made to move from the starting position on the target depth map texture according to the ray direction and the current step size, so as to simulate the viewpoint movement on the target depth map texture;

[0148] The current step size is the ratio of the current ray depth (currentRayDepth) to the number of iterations.

[0149] Step S4213: In each step of the simulated viewpoint movement, obtain the current ray depth and texture coordinates corresponding to the sampling point where the preset ray intersects with the target depth map texture;

[0150] In the specific implementation of step S4213, in each step of the simulated viewpoint movement, the position of the preset ray on the target depth map texture plane when the ray steps to the current ray depth is obtained, and the texture coordinates of the sampling point corresponding to that position are obtained.

[0151] If the current step is the first step of the simulated viewpoint movement, the current ray depth is the preset value; if the current step is not the first step of the simulated viewpoint movement, the current ray depth is the current step size corresponding to the previous sampling point.

[0152] Step S4214: Determine the sampling coordinates based on the current ray depth and texture coordinates corresponding to the sampling point, and the target depth map texture.

[0153] It should be noted that the specific implementation of step S4214 includes the following steps.

[0154] A1: Calculate the parallax parameter based on the current ray depth and viewpoint offset corresponding to the sampling point;

[0155] In the specific implementation of step A1, in each step of simulating viewpoint movement, since the light moves to the depth of the current light, the position of the preset light on the initial image plane will produce a parallax effect. Therefore, the current light depth currentRayDepth and the viewpoint offset viewpointOffset are substituted into formula (1) for calculation to obtain the parallax parameter parallaxEffect.

[0156] Formula (1):

[0157]

[0158] A2: Determine the initial coordinates based on the texture coordinates and disparity parameters corresponding to the sampling points;

[0159] It should be noted that the initial coordinates are scene coordinates.

[0160] Specifically, the disparity parameter parallaxEffect is calculated in relation to the texture coordinates of the sampling points. The sum of the midpoints is used to obtain the new initial coordinates sampleUV. That is, the parallax parameter parallaxEffect and the texture coordinates of the sampling point are substituted into formula (2) for calculation to obtain the initial coordinates sampleUV.

[0161] Formula (2):

[0162]

[0163] A3: Use the initial coordinates to read the scene depth of the sampling point from the target depth map texture;

[0164] Specifically, the initial coordinates sampleUV are used to read the depth value corresponding to the initial coordinates of the sampling point from the stabilizedDepthMapTexture of the target depth map, and this value is used as the actual scene depth sceneDepth, as shown in formula (3).

[0165] Formula (3):

[0166]

[0167] Where r is a preset coefficient, which is set in advance.

[0168] A4: Based on the scene depth and the current ray depth corresponding to the sampling point, determine whether the initial coordinates need to be updated. If so, execute step A5; otherwise, execute step S4212 to perform the next loop and let the ray move one more step.

[0169] In the specific implementation of step A4, the current ray depth is compared with the actual scene depth. If the current ray depth is greater than or equal to the actual scene depth, the loop ends and step A5 is executed. Otherwise, it means that the ray is still in front of the pixel and is "empty". In this case, the current step size is used as the next current ray depth. That is, the current ray depth currentRayDepth = stepSize is updated by the current step size to obtain the next current ray depth, and the loop continues for the next time. That is, the execution of step S4212 is returned to let the ray move one more step.

[0170] A5: Use the initial coordinates as the sampling coordinates;

[0171] Step S422: Obtain the color corresponding to the sampling coordinates from the initial image based on the sampling coordinates, and use it as the target color of the pixel;

[0172] In the specific implementation step S422, the sampling coordinate sampleUV obtained during the collision is used to obtain the color finalPixelColor at the sampling coordinate from the original source image object, i.e. the initial image sourceTexture, as shown in formula (4), and the color is used as the target color of the pixel.

[0173] Formula (4):

[0174]

[0175] Where {x, y} are the sampling coordinates sampleUV.

[0176] Step S423: Collect the target color of each pixel in the target depth map texture to obtain the target rendering frame.

[0177] In the specific implementation step S423, the GPU performs the aforementioned "ray stepping" process completely for each pixel on the screen. Since each pixel has a different texture coordinate (currentUV), their calculated final sampling coordinates (sampleUV) are also different, resulting in a three-dimensional distortion and displacement effect. The sampled colors (finalPixelColor) of all pixels are combined to form a complete image with a pseudo-3D effect.

[0178] This complete image is the finalRenderedFrame, which is ultimately drawn onto the canvas and presented to the user.

[0179] The role of the "deep safe zone": In this process, if sampleUV happens to fall on the edge of the object, because the sampled is stabilizedDepthMapTexture, sceneDepth will stably return the foreground depth, ensuring that the collision detection result does not jump between the foreground and the background, thus ensuring the stability of the visual effect.

[0180] Optionally, the browser will automatically call this callback function before each frame is redrawn. Inside the function, the latest viewpointOffset is first obtained based on the interaction event, and then used as input in step S204 to generate and display a new frame.

[0181] Optionally, it also includes: tightly linking user interaction and the final target rendering frame through a loop mechanism, achieving real-time and smooth synchronization between user operation and visual feedback.

[0182] This application, through the close connection and data flow of the above steps, transforms an ordinary 2D image into a high-quality, artifact-free, and smoothly interactive pseudo-3D visual experience through AI inference, core "deep safe zone" generation, and final high-precision rendering.

[0183] It should be noted that the specific implementation process of steps S201 to S204 can also be achieved as follows: Figure 3 Please provide an explanation.

[0184] In this embodiment of the invention, an initial depth map is uploaded to a texture to obtain an initial depth map texture. Then, a morphological dilation operation is performed on the initial depth map texture to expand the brighter areas of the pixels to the darker areas by one or more pixels, so as to form a depth region that is slightly larger than the visual outline of the object. Then, the target rendering frame is determined by using the viewpoint offset, the target depth map texture and the initial image to avoid visual artifacts during rendering.

[0185] Optionally, based on the image processing method shown in the above embodiments of the present invention, correspondingly, the embodiments of the present invention also show an image processing apparatus, such as... Figure 4 As shown.

[0186] The acquisition unit 401 is used to acquire the initial image and interaction events input by the user;

[0187] Depth estimation unit 402 is used to perform depth estimation on the initial image to obtain an initial depth map;

[0188] Processing unit 403 is used to perform dilation processing based on the initial depth map to obtain the target depth map texture;

[0189] The rendering unit 404 is used to render the target rendering frame based on the viewpoint offset determined by the interaction event, the target depth map texture, and the initial image.

[0190] The specific principles and execution processes of each unit in the image processing apparatus disclosed in the above embodiments of the present invention are the same as the corresponding contents in the image processing method provided in the above embodiments of the present invention. Please refer to the corresponding parts in the image processing method disclosed in the above embodiments of the present invention, and they will not be repeated here.

[0191] In this embodiment of the invention, an initial depth map is uploaded to a texture to obtain an initial depth map texture. Then, a morphological dilation operation is performed on the initial depth map texture to expand the brighter areas of the pixels to the darker areas by one or more pixels, so as to form a depth region that is slightly larger than the visual outline of the object. Then, the target rendering frame is determined by using the viewpoint offset, the target depth map texture and the initial image to avoid visual artifacts during rendering.

[0192] Optionally, based on the image processing apparatus shown in the above embodiments of the present invention, the depth estimation unit 402 is specifically used for:

[0193] The initial image is decoded to obtain the corresponding source image object;

[0194] Convert the source image object into a tensor;

[0195] The preloaded model object is invoked to perform depth prediction on the tensor to obtain the original depth tensor. The model object encapsulates a deep learning model, which is trained based on historical tensors.

[0196] The original depth tensor is restored to obtain the initial depth map.

[0197] Optionally, based on the image processing apparatus shown in the above embodiments of the present invention, the processing unit 403 is specifically used for:

[0198] The initial depth map texture is determined by processing the texture corresponding to the initial depth map.

[0199] For each pixel in the initial depth map texture, determine the corresponding texture coordinates of the pixel in the initial depth map texture;

[0200] Based on the preset structural elements and the texture coordinates of the pixel, the pixel is dilated in the initial depth map texture to determine the corresponding dilated pixel and the corresponding texture coordinates.

[0201] The initial depth map texture is updated based on the texture coordinates of each pixel and the texture coordinates of its corresponding dilated pixel to obtain the target depth map texture.

[0202] The processing unit 403, which updates the initial depth map texture based on the texture coordinates of each pixel and the texture coordinates of its corresponding dilated pixel to obtain the target depth map texture, is specifically used for:

[0203] For each pixel of the initial depth map texture, the initial depth value of the pixel is read from the initial depth map texture based on the texture coordinates of the pixel;

[0204] The initial depth value of the dilated pixel is read from the initial depth map texture according to the texture coordinates of the dilated pixel;

[0205] For each dilated pixel, the target depth value of the pixel is determined based on the initial depth value of the dilated pixel and the initial depth value of the pixel.

[0206] The initial depth map texture is updated based on the target depth value of each pixel to obtain the target depth map texture.

[0207] Optionally, based on the image processing apparatus shown in the above embodiments of the present invention, the rendering unit 404 is specifically used for:

[0208] The parameters corresponding to the interactive event are converted into a first vector, and the first vector is used as the viewpoint offset.

[0209] Rendering operations are performed based on the source image object determined by the initial image, the target depth map texture, and the viewpoint offset to obtain the target rendering frame.

[0210] Optionally, based on the image processing apparatus shown in the above embodiments of the present invention, a rendering unit 404, which performs a rendering operation based on the source image object determined from the initial image, the target depth map texture, and the viewpoint offset to obtain the target rendering frame, is specifically used for:

[0211] For each pixel in the target depth map texture, a preset ray is invoked to render the pixel based on the target depth map texture and the viewpoint offset to determine the corresponding sampling coordinates;

[0212] Based on the sampling coordinates, the color corresponding to the sampling coordinates is obtained from the initial image and used as the target color of the pixel;

[0213] The target color of each pixel in the target depth map texture is collected to obtain the target rendering frame.

[0214] Optionally, based on the image processing apparatus shown in the above embodiments of the present invention, the rendering unit 404, which renders each pixel in the target depth map texture by calling a preset ray based on the target depth map texture and the viewpoint offset, and determines the corresponding sampling coordinates, is specifically used for:

[0215] For each pixel in the target depth map texture, ray stepping information is determined based on the texture coordinates of the pixel and the viewpoint offset;

[0216] A preset ray is invoked to simulate viewpoint movement on the target depth map texture according to the ray step information;

[0217] In each step of the simulated viewpoint movement, the current ray depth and texture coordinates corresponding to the sampling point at the intersection of the preset ray and the target depth map texture are obtained;

[0218] The sampling coordinates are determined based on the current ray depth and texture coordinates corresponding to the sampling point, and the target depth map texture.

[0219] Optionally, based on the image processing apparatus shown in the above embodiments of the present invention, the rendering unit 404, which determines the sampling coordinates based on the current ray depth and texture coordinates corresponding to the sampling point and the target depth map texture, is specifically used for:

[0220] Calculate the disparity parameters based on the current ray depth and viewpoint offset corresponding to the sampling point;

[0221] The initial coordinates are determined based on the texture coordinates and disparity parameters corresponding to the sampling points;

[0222] The scene depth of the sampling point is read from the target depth map texture using the initial coordinates;

[0223] Determine whether the initial coordinates need to be updated based on the scene depth and the current ray depth corresponding to the sampling point;

[0224] If necessary, the initial coordinates can be used as sampling coordinates.

[0225] This application provides an electronic device, which includes a processor and a memory. The memory is used to store image processing program code and data, and the processor is used to call the program instructions in the memory to execute the steps shown in the image processing method described in the above embodiments.

[0226] This invention provides a storage medium, namely a computer-readable storage medium, which includes a stored program, wherein the program controls the device where the storage medium is located to execute the image processing method described in the above embodiments when it is running.

[0227] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0228] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0229] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that, The method includes: Get the initial image and interaction events input by the user; Depth estimation is performed on the initial image to obtain an initial depth map; Dilation processing is performed on the initial depth map to obtain the target depth map texture; Rendering is performed based on the viewpoint offset determined by the interaction event, the target depth map texture, and the initial image to obtain the target rendering frame.

2. The method according to claim 1, characterized in that, Depth estimation is performed on the initial image to obtain an initial depth map, including: The initial image is decoded to obtain the corresponding source image object; Convert the source image object into a tensor; The pre-loaded model object is invoked to perform depth prediction on the tensor, and the original depth tensor is obtained; The original depth tensor is restored to obtain the initial depth map.

3. The method according to claim 1, characterized in that, Based on the initial depth map, dilation processing is performed to obtain the target depth map texture, including: The initial depth map texture is determined by processing the texture corresponding to the initial depth map. For each pixel in the initial depth map texture, determine the corresponding texture coordinates of the pixel in the initial depth map texture; Based on the preset structural elements and the texture coordinates of the pixel, the pixel is dilated in the initial depth map texture to determine the corresponding dilated pixel and the corresponding texture coordinates. The initial depth map texture is updated based on the texture coordinates of each pixel and the texture coordinates of its corresponding dilated pixel to obtain the target depth map texture.

4. The method according to claim 3, characterized in that, The initial depth map texture is updated based on the texture coordinates of each pixel and the texture coordinates of its corresponding dilated pixel to obtain the target depth map texture, including: For each pixel of the initial depth map texture, the initial depth value of the pixel is read from the initial depth map texture based on the texture coordinates of the pixel; The initial depth value of the dilated pixel is read from the initial depth map texture according to the texture coordinates of the dilated pixel; For each dilated pixel, the target depth value of the pixel is determined based on the initial depth value of the dilated pixel and the initial depth value of the pixel. The initial depth map texture is updated based on the target depth value of each pixel to obtain the target depth map texture.

5. The method according to claim 1, characterized in that, Rendering is performed based on the viewpoint offset determined by the interaction event, the target depth map texture, and the initial image to obtain a target rendering frame, including: The parameters corresponding to the interactive event are converted into a first vector, and the first vector is used as the viewpoint offset. Rendering operations are performed based on the source image object determined by the initial image, the target depth map texture, and the viewpoint offset to obtain the target rendering frame.

6. The method according to claim 5, characterized in that, A rendering operation is performed based on the source image object determined from the initial image, the target depth map texture, and the viewpoint offset to obtain the target rendering frame, including: For each pixel in the target depth map texture, a preset ray is invoked to render the pixel based on the pixel's texture coordinates and the viewpoint offset to determine the corresponding sampling coordinates; Based on the sampling coordinates, the color corresponding to the sampling coordinates is obtained from the initial image and used as the target color of the pixel; The target color of each pixel in the target depth map texture is collected to obtain the target rendering frame.

7. The method according to claim 6, characterized in that, For each pixel in the target depth map texture, a preset ray is invoked to render the pixel based on the pixel's texture coordinates and the viewpoint offset to determine the corresponding sampling coordinates, including: For each pixel in the target depth map texture, ray stepping information is determined based on the texture coordinates of the pixel and the viewpoint offset; A preset ray is invoked to simulate viewpoint movement on the target depth map texture according to the ray step information; In each step of the simulated viewpoint movement, the current ray depth and texture coordinates corresponding to the sampling point at the intersection of the preset ray and the target depth map texture are obtained; The sampling coordinates are determined based on the current ray depth and texture coordinates corresponding to the sampling point, and the target depth map texture.

8. The method according to claim 7, characterized in that, Determining the sampling coordinates based on the current ray depth and texture coordinates corresponding to the sampling point, and the target depth map texture, includes: Calculate the disparity parameters based on the current ray depth and viewpoint offset corresponding to the sampling point; The initial coordinates are determined based on the texture coordinates and disparity parameters corresponding to the sampling points; The scene depth of the sampling point is read from the target depth map texture using the initial coordinates; Determine whether the initial coordinates need to be updated based on the scene depth and the current ray depth corresponding to the sampling point; If necessary, the initial coordinates can be used as sampling coordinates.

9. An image processing apparatus, characterized in that, The device includes: The acquisition unit is used to acquire the initial image and interaction events input by the user; A depth estimation unit is used to perform depth estimation on the initial image to obtain an initial depth map; The processing unit is used to perform dilation processing based on the initial depth map to obtain the target depth map texture; The rendering unit is used to render the target rendering frame based on the viewpoint offset determined by the interaction event, the target depth map texture, and the initial image.

10. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, it controls the device where the storage medium is located to perform the image processing method as described in any one of claims 1-8.

Images