An image processing method, apparatus and device

Abstract

Embodiments of the present specification disclose an image processing method, device and equipment, the method comprising: acquiring an original image to be processed, the original image being a two-dimensional image; based on spatial distance information of pixel points in the original image, splitting a plurality of layers of sub-images from the original image, and performing content completion on the plurality of layers of sub-images to obtain a plurality of layers of target sub-images, the spatial distance information of the pixel points being used to represent the relative distance between different pixel points and a same reference surface in the original image; based on the spatial distance information of the pixel points in the target sub-images, determining a rendering strategy of the pixel points in the target sub-images, and re-rendering the pixel points in the target sub-images according to the rendering strategy to obtain a pseudo three-dimensional sub-image corresponding to the target sub-images; and according to the hierarchical order between the plurality of layers of target sub-images, superimposing the pseudo three-dimensional sub-images corresponding to the plurality of layers of target sub-images to obtain a target pseudo three-dimensional image corresponding to the original image.

Description

Technical Field

[0001] This document relates to the field of image processing technology, and in particular to an image processing method, apparatus and device. Background Technology

[0002] With the popularization of WebGL / WebGPU technology, the demand for generating pseudo-3D images based on single 2D images is constantly growing. Pseudo-3D images are a type of image that simulates 3D visual effects using 2D techniques. By utilizing visual elements such as perspective, light and shadow, and texture, it can create images with a sense of depth and space on a 2D plane. Pseudo-3D images can be widely used in interactive advertising, AI content generation, and e-commerce displays.

[0003] Currently, pseudo-3D images can be generated based on a single color map and a single depth map. However, when occlusion exists in the image (such as a hand blocking an arm or an object obscuring the background), the depth map lacks true pixel and depth information in the occluded areas. This leads to large-scale pixel stretching, tearing, and holes due to parallax shift, resulting in severely distorted and poor-quality pseudo-3D images. Therefore, a technical solution is needed to improve the quality of pseudo-3D images. Summary of the Invention

[0004] The purpose of the embodiments in this specification is to provide a technical solution that can improve the quality of pseudo-3D images.

[0005] To achieve the above technical solution, the embodiments in this specification are implemented as follows: This specification provides an image processing method, comprising: acquiring an original image to be processed, wherein the original image is a two-dimensional image; splitting multiple sub-images from the original image based on the spatial distance information of pixels within the original image, and performing content completion on the multiple sub-images to obtain multiple target sub-images, wherein the spatial distance information of pixels is used to characterize the relative distance between different pixels in the original image and the same reference surface; determining a rendering strategy for pixels in the target sub-images based on the spatial distance information of pixels within the target sub-images, and re-rendering the pixels in the target sub-images according to the rendering strategy to obtain a pseudo-three-dimensional sub-image corresponding to the target sub-images; and superimposing the pseudo-three-dimensional sub-images corresponding to the multiple target sub-images according to the hierarchical order between the multiple target sub-images to obtain a target pseudo-three-dimensional image corresponding to the original image.

[0006] This specification provides an image processing apparatus, comprising: a layering module for acquiring an original image to be processed, wherein the original image is a two-dimensional image; splitting multiple sub-images from the original image based on the spatial distance information of pixels within the original image, and performing content completion on the multiple sub-images to obtain multiple target sub-images, wherein the spatial distance information of pixels is used to characterize the relative distance between different pixels in the original image and the same reference surface; a rendering module for determining a rendering strategy for pixels in the target sub-images based on the spatial distance information of pixels within the target sub-images, and re-rendering the pixels in the target sub-images according to the rendering strategy to obtain a pseudo-three-dimensional sub-image corresponding to the target sub-images; and an overlay module for overlaying the pseudo-three-dimensional sub-images corresponding to the multiple target sub-images according to the hierarchical order between the multiple target sub-images to obtain a target pseudo-three-dimensional image corresponding to the original image.

[0007] This specification provides an image processing device comprising: a processor; and a memory arranged to store computer-executable instructions, wherein the executable instructions, when executed, cause the processor to: acquire a raw image to be processed, the raw image being a two-dimensional image; based on spatial distance information of pixels within the raw image, split the raw image into multiple sub-images, and perform content completion on the multiple sub-images to obtain multiple target sub-images, wherein the spatial distance information of pixels is used to characterize the relative distance between different pixels in the raw image and the same reference surface; based on the spatial distance information of pixels within the target sub-images, determine a rendering strategy for pixels in the target sub-images, and re-render the pixels in the target sub-images according to the rendering strategy to obtain a pseudo-three-dimensional sub-image corresponding to the target sub-images; and superimpose the pseudo-three-dimensional sub-images corresponding to the multiple target sub-images according to the hierarchical order between the multiple target sub-images to obtain a target pseudo-three-dimensional image corresponding to the raw image.

[0008] This specification also provides a storage medium for storing computer-executable instructions. When executed by a processor, the executable instructions implement the following process: acquiring an original image to be processed, wherein the original image is a two-dimensional image; based on the spatial distance information of pixels in the original image, splitting the original image into multiple sub-images and performing content completion on the multiple sub-images to obtain multiple target sub-images, wherein the spatial distance information of pixels is used to characterize the relative distance between different pixels in the original image and the same reference surface; based on the spatial distance information of pixels in the target sub-images, determining the rendering strategy of pixels in the target sub-images, and re-rendering the pixels in the target sub-images according to the rendering strategy to obtain a pseudo-3D sub-image corresponding to the target sub-images; and superimposing the pseudo-3D sub-images corresponding to the multiple target sub-images according to the hierarchical order between the multiple target sub-images to obtain a target pseudo-3D image corresponding to the original image.

[0009] This specification also provides a computer program product, including a computer program that, when executed by a processor, implements the following process: acquiring an original image to be processed, the original image being a two-dimensional image; based on the spatial distance information of pixels within the original image, splitting the original image into multiple sub-images, and performing content completion on the multiple sub-images to obtain multiple target sub-images, wherein the spatial distance information of pixels is used to characterize the relative distance between different pixels in the original image and the same reference surface; based on the spatial distance information of pixels within the target sub-images, determining a rendering strategy for pixels in the target sub-images, and re-rendering the pixels in the target sub-images according to the rendering strategy to obtain a pseudo-3D sub-image corresponding to the target sub-images; and superimposing the pseudo-3D sub-images corresponding to the multiple target sub-images according to the hierarchical order between the multiple target sub-images to obtain a target pseudo-3D image corresponding to the original image. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Figure 1 This is a schematic flowchart of an image processing method according to one embodiment of this specification; Figure 2 This is a schematic diagram of a process for splitting multiple sub-images according to one embodiment of this specification; Figure 3This is a schematic diagram of another process for splitting multi-layer sub-images according to one embodiment of this specification; Figure 4 This is a schematic diagram illustrating a process for content completion of multi-layer sub-images in one embodiment of this specification; Figure 5 This is a schematic diagram of a process for determining a pseudo-3D sub-image in one embodiment of this specification; Figure 6 This is a schematic flowchart of another image processing method according to one embodiment of this specification; Figure 7 This is a schematic diagram of an image processing apparatus according to one embodiment of this specification; Figure 8 This is a schematic diagram of an image processing device according to one embodiment of this specification. Detailed Implementation

[0011] This specification provides an image processing method, apparatus, and device through its embodiments.

[0012] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0013] This specification provides a novel pseudo-3D image generation mechanism that can be used to convert 2D images into pseudo-3D parallax images with stereoscopic effect and spatial depth. Current mainstream pseudo-3D image generation mechanisms typically render pseudo-3D images based on a single color map and a single depth map. When occlusion exists in the image (e.g., a hand blocking an arm, or an object obscuring the background), the depth map lacks real pixels and depth information in the occluded area, leading to large-area pixel stretching, tearing, and holes after parallax shift, resulting in severely distorted and poor-quality pseudo-3D images. Other existing pseudo-3D image generation mechanisms, such as Depth-Aware Warping, Meta 3D Photo, Immersity.AI, and H5 pseudo-3D plugins, cannot solve the problem of missing occluded areas or maintain the correct occlusion relationship in complex foreground and background scenes. They generally suffer from limitations such as small parallax range, fragmentation with even slight shifts, and severe edge distortion, resulting in poor-quality generated pseudo-3D images.

[0014] To address this, the present invention proposes a superior image processing scheme. The first step involves acquiring a two-dimensional image to be processed as the original image. Then, based on the spatial distance information of pixels within the original image, multiple sub-images are extracted from it. These sub-images are then augmented to obtain multiple target sub-images. The spatial distance information of pixels can be used to characterize the relative distances between different pixels in the original image and the same reference surface. Next, based on the spatial distance information of pixels within the target sub-images, a rendering strategy for the pixels in the target sub-images is determined. The pixels in the target sub-images are then re-rendered according to this strategy to obtain pseudo-3D sub-images corresponding to the target sub-images. Finally, the pseudo-3D sub-images corresponding to the multiple target sub-images are superimposed according to the hierarchical order between them to obtain the target pseudo-3D image corresponding to the original image. This extraction and augmentation of the original image ensures that each layer possesses complete image information, eliminating pixel stretching, tearing, and holes caused by occlusion. Then, pseudo-3D transformation is performed on each layer, and the resulting multiple pseudo-3D layers are accurately synthesized, effectively improving the quality of the pseudo-3D image. Specific processing details can be found in the following embodiments.

[0015] like Figure 1 As shown in the embodiments of this specification, an image processing method is provided. The execution subject of this method can be a terminal device or a server, etc. The terminal device can be a mobile terminal device such as a mobile phone or tablet computer, a computer device such as a laptop or desktop computer, or an IoT device (specifically, a smartwatch, an in-vehicle device, etc.). The server can be a single server or a server cluster composed of multiple servers. The server can be a backend server in fields such as finance or online shopping, or a backend server of an application. This embodiment uses a server as the execution subject for detailed description. For the case where the execution subject is a terminal device, please refer to the following server case processing, which will not be repeated here. The method may specifically include the following steps: In step S102, the original image to be processed is obtained, which is a two-dimensional image.

[0016] The original image is an RGB image, which can be obtained from an existing image library or acquired independently. This manual does not specify the source of the original image. A two-dimensional image is a matrix composed of pixels, each pixel containing information such as color and brightness, which together constitute the visual content on a plane. It only has two dimensions: length and width, and no depth information.

[0017] In step S104, based on the spatial distance information of pixels in the original image, multiple sub-images are extracted from the original image, and the content of the multiple sub-images is completed to obtain multiple target sub-images.

[0018] The spatial distance information of pixels is used to characterize the relative distance between different pixels in the original image and the same reference surface. The spatial distance information of pixels in the original image can be represented in various ways, such as depth maps, point clouds, implicit representations, parametric surfaces, etc.

[0019] In implementation, the spatial distance information of pixels within the original image reflects the distance between each pixel and the observation point. Based on this distance, pixels within different distance ranges are extracted from the original image. Sub-images are then constructed based on the pixels within each extracted distance range, with the positions of these pixels in the sub-images being the same as their positions in the original image. Multiple sub-images are constructed based on the extracted distance ranges, and these sub-images can be identified as foreground, midground, background, etc., according to the distance range associated with each sub-image.

[0020] The distance range can be determined based on a preset threshold, or based on distance clustering, or manually based on image semantics. This embodiment does not limit this.

[0021] After splitting the original image into multiple sub-images, for any given sub-image, the scene representation in the sub-image can be completed based on distance information within the sub-image and scene visual information provided by the original image. For example, the shape and outline of areas occluded by foreground objects in the sub-image can be completed, making the sub-image appear as it would be without occlusion; the texture and color of occluded areas can also be completed, allowing them to blend naturally with the surrounding background. This specification allows for the use of manual completion and automatic algorithmic completion methods to complete the content of the sub-images. The specific completion process will be described in detail below and will not be elaborated upon here.

[0022] In step S106, based on the spatial distance information of pixels in the target sub-image, the rendering strategy of pixels in the target sub-image is determined, and the pixels in the target sub-image are re-rendered according to the rendering strategy to obtain the pseudo-3D sub-image corresponding to the target sub-image.

[0023] The purpose of re-rendering the pixels in the target sub-image is to simulate the relative positional relationship of different pixels in the target sub-image in three-dimensional space, so as to reflect the visual effect of the object in the target sub-image in the three-dimensional scene, such as stereoscopic effect and perspective changes.

[0024] In implementation, the rendering strategy for pixels in the target sub-image can be determined based on the spatial distance information of pixels within the target sub-image. That is, the visual effect of objects in the target sub-image within a 3D scene is converted into pixel values ​​of a 2D image, and a rendering strategy is constructed based on these converted pixel values. The pixel values ​​of each pixel in the target sub-image are adjusted based on the pixel values ​​included in the rendering strategy, thus re-rendering the pixels in the target sub-image. This re-rendered target sub-image can then present a 3D visual effect formed by parallax offset, and is thus identified as the pseudo-3D sub-image corresponding to the target sub-image.

[0025] In step S108, the pseudo-3D sub-images corresponding to the multi-layer target sub-images are superimposed according to the hierarchical order between the multi-layer target sub-images to obtain the target pseudo-3D image corresponding to the original image.

[0026] As mentioned earlier, based on the distance range associated with each sub-image, these sub-images can be identified as foreground, midground, background, etc. Therefore, based on the distance range associated with each sub-image, the hierarchical order between multiple layers of target sub-images can be determined, and the hierarchical order between target sub-images can characterize the order of the distance range of pixels.

[0027] In implementation, pseudo-3D sub-images corresponding to multiple target sub-images are superimposed according to the hierarchical order between the target sub-images. This allows the pixels in the target sub-images to be superimposed together in order of distance from farthest to closest, fusing them into a single pseudo-3D image, which serves as the target pseudo-3D image corresponding to the original image. This target pseudo-3D image can provide a pseudo-3D stereoscopic representation of the image content in the original image.

[0028] This specification provides an image processing method that involves acquiring an original image to be processed, splitting it into multiple sub-images based on the spatial distance information of pixels within the original image, and then performing content completion on these sub-images to obtain multiple target sub-images. The spatial distance information of pixels can be used to characterize the relative distance between different pixels in the original image and the same reference surface. Then, based on the spatial distance information of pixels within the target sub-images, a rendering strategy for the pixels in the target sub-images is determined, and the pixels in the target sub-images are re-rendered according to this strategy to obtain pseudo-3D sub-images corresponding to the target sub-images. Subsequently, the pseudo-3D sub-images corresponding to the multiple target sub-images are superimposed according to the hierarchical order between the multiple target sub-images to obtain the target pseudo-3D image corresponding to the original image. In this way, by splitting and completing the original image, each layer can possess complete image information, eliminating problems such as pixel stretching, tearing, and holes caused by occlusion. Then, pseudo-3D transformation is performed on each layer, and the multiple pseudo-3D layers are accurately synthesized, effectively improving the quality of the pseudo-3D image.

[0029] In the above or following embodiments, the processing method for splitting multi-layer sub-images from the original image based on the spatial distance information of pixels within the original image in step S104 can be varied. One optional processing method is provided below, such as... Figure 2 As shown, this implementation method may specifically include the following steps S202~S204.

[0030] In step S202, a target depth map corresponding to the original image is determined. The target depth map is used to characterize the spatial distance information of pixels in the original image.

[0031] The original image is a multi-channel RGB image, while the target depth map is a single-channel grayscale image. The grayscale value of each pixel in the target depth map represents the distance (depth) from the corresponding point in the scene to the observation point. The values ​​in the depth map can be represented by 8 bits (0-255) or 16 bits (0-65535). The larger the value, the closer the distance to the observation point, and the smaller the value, the farther the distance to the observation point.

[0032] In practice, RGB images themselves do not contain depth information. Based on the scene content in the original image, algorithms or models can be used to infer the distances of objects in the scene. Then, the distance of each pixel is inferred from the object distances, and a target depth map corresponding to the original image is constructed based on the inferred pixel distances. Therefore, the target depth map can be used to represent the spatial distance information of pixels within the original image.

[0033] In step S204, multi-layer sub-images are extracted from the original image based on the depth distribution of pixels in the target depth map.

[0034] In the target depth map, the depth distribution of pixels refers to the statistical characteristics or distribution pattern of the depth values ​​corresponding to all pixels in the image, which reflects the spatial distance relationship between objects and observation points in the scene.

[0035] In practice, based on the depth distribution of pixels in the target depth map, the frequency of each depth value (or depth interval) in the depth map can be statistically analyzed to intuitively understand the distribution pattern of depth values. Then, based on the distribution pattern of depth values, multiple depth layers (such as foreground, midground, and background) existing in the target depth map scene can be determined. Based on these depth layers, the original image is then layered and split to obtain multiple sub-images contained within the original image. Each sub-image may correspond to a local region of the original image, and there is no pixel overlap between different sub-images.

[0036] Based on this, there are various ways to determine the target depth map corresponding to the original image in step S202. The following is an optional implementation method, which may specifically include the following steps.

[0037] Step S2022: Obtain a depth model for depth prediction of the image; Step S2024: Input the original image into the depth model, perform depth prediction on the original image through the depth model, and output the depth map corresponding to the original image. Step S2026: Normalize and invert the depth map to obtain the target depth map corresponding to the original image.

[0038] A depth model is an algorithm that predicts the distance between objects and the camera in a scene by analyzing visual information (such as texture, color, and contextual relationships) in an image. The depth model used in this embodiment is a pre-trained depth estimation model, such as Depth-Anything, MiDaS, DPT, MonoDepth2, etc.

[0039] In implementation, the original image can be input into a depth model. The depth model can extract image features from the original image, such as perspective, occlusion relationships, and texture gradients, and predict the depth of the original image based on the extracted features, outputting a single-channel depth map corresponding to the original image. Then, the depth map output by the depth model can be normalized to the range [0, 1] for visualization or subsequent processing. The normalized depth map is then inverted. By inverting the depth values ​​in the depth map, pixels that were originally far apart become closer and have larger depth values, while pixels that were originally close become farther away and have smaller depth values. That is, a larger value in the uninverted depth map indicates a greater distance, and a larger value in the inverted depth map indicates a closer distance. Inversion can emphasize the foreground in the image. Finally, the inverted depth map is determined as the target depth map corresponding to the original image.

[0040] exist Figure 2 Based on this, step S204 involves various methods for separating multiple sub-images from the original image according to the depth distribution of pixels in the target depth map. One optional method is provided below, such as... Figure 3 As shown, this implementation method may specifically include the following steps.

[0041] Step S302: Based on the grayscale value corresponding to each pixel in the target depth map, classify the pixels in the original image to obtain multiple pixel sets.

[0042] The target depth map is a grayscale image containing the grayscale values ​​corresponding to each pixel. The magnitude of the grayscale value can be used to characterize the distance between the corresponding pixel and the observation point.

[0043] In implementation, pixels in the original image can be classified based on their corresponding grayscale values. For example, multiple grayscale ranges can be preset, and pixels can be divided into corresponding grayscale ranges according to their grayscale values. Pixels in each grayscale range form a pixel set, resulting in multiple pixel sets. Of course, other pixel classification schemes can also be used, such as grayscale value-based clustering to divide the pixels in the original image into multiple pixel sets, or semantically guided deep aggregation to divide the pixels in the original image into multiple pixel sets, etc.

[0044] Step S304: Based on multiple pixel sets and the original image, generate a mask corresponding to each pixel set. The pixel region composed of the pixel sets represents the effective region in the mask.

[0045] The mask is a matrix of the same size as the original image, with each element corresponding to a pixel in the image. Typically, 0 represents invalid regions, and 1 or 255 represents valid regions. Masks can be used to extract, modify, or mask specific parts of an image.

[0046] In implementation, the pixel region occupied by each pixel set in the original image can be directly mapped to the valid region in the binary image, and the remaining regions in the original image can be mapped to invalid regions. The mask corresponding to the pixel set is then formed based on the mapped valid and invalid regions. Alternatively, an all-zero mask can be created for each pixel set, and the mask positions corresponding to the pixel coordinates in the pixel set can be set to 1 or 255 to obtain the mask corresponding to the pixel set.

[0047] It should be noted that after the pixel mapping is completed, morphological operations (such as dilation and closing operations) can be used to fill the holes to ensure that the effective area mapped by the pixel set in the mask is a continuous area.

[0048] Step S306: Based on the masks corresponding to multiple pixel sets, split the sub-image associated with each mask from the original image, and determine multi-layer sub-images based on the images associated with multiple masks.

[0049] In practice, for any set of pixels, each channel of the original image can be bitwise ANDed with the mask (e.g., Original Image × Mask = Subimage), so that the area with a mask value of 0 is set to black, and the area with a mask value of 1 retains its original value, so as to separate the sub-image associated with the mask from the original image, and determine the multiple sub-images associated with the mask as the multi-layer sub-images of the original image.

[0050] Furthermore, the target depth map corresponding to the original image can be further segmented. Similar to segmenting the original image, the depth sub-image associated with each mask can be extracted from the target depth map based on the masks corresponding to multiple pixel sets in step S304 above. Then, based on the multiple mask-associated depth sub-images, the multi-layer depth sub-images associated with the target depth map can be determined.

[0051] In implementation, for any set of pixels, the target depth map can be bitwise ANDed with the mask (e.g., Depth Image × Mask = Depth Subimage) so that the area with a mask value of 0 is set to black, and the area with a mask value of 1 retains its original value. This separates the depth sub-image associated with the mask from the target depth map, and the separated depth sub-images associated with the mask are determined as the multi-layer depth sub-image of the target depth map.

[0052] exist Figure 3 Based on this, step S104 performs content completion on the multi-layer sub-images to obtain multi-layer target sub-images. There are various processing methods for this; the following provides one optional processing method, such as... Figure 4 As shown, this processing method may specifically include the following steps S402~S406.

[0053] Step S402: Based on the mask corresponding to each set of pixels, perform boundary detection and erosion processing on the effective areas in the mask to obtain the processed mask.

[0054] Erosion is a basic morphological operation in image processing. It optimizes the shape or structure of the effective region by "eroding" the effective region in a binary image to reduce the boundaries of objects, remove small noise, separate sticky objects, or smooth edges.

[0055] In implementation, for the mask corresponding to each set of pixels, the edges of the effective region in the mask can be extracted using Canny, Sobel, or depth gradient. Then, the effective region in the mask is eroded to remove small noise in the effective region, separate closely connected objects, and smooth the edges of the effective region, resulting in a processed mask with continuous regions and smooth edges.

[0056] Step S404: Based on the processed mask, the content of the multi-layer depth sub-image is completed using a preset depth completion model to obtain a multi-layer target depth sub-image.

[0057] The core idea of ​​the preset depth completion model is to propagate information (such as depth values ​​or structures) from known regions to missing regions based on a diffusion mechanism, thereby filling in holes or noisy regions in the depth sub-image and restoring the spatial distance information of the missing regions. Preset depth completion models can include Depth Completion networks, BoostingMonocular Depth, and Deep Sparse Depth Prior, among others.

[0058] In practice, the processed mask and depth sub-image can be input into a preset depth completion model. The processed mask can be used as auxiliary information to guide the depth completion model to focus on the missing parts, and to focus on the content completion of the effective areas in the depth sub-image. The output of the preset depth completion model is then determined as the completed target depth sub-image.

[0059] In addition, this specification also allows for the use of geometric rules to complete depth sub-images. Examples include local plane fitting, neighboring layer depth inference, and multi-layer average depth estimation. The core of geometric rule-based depth map completion lies in utilizing prior assumptions about the scene (such as local planarity and spatial continuity) to infer missing values ​​through mathematical optimization or heuristic rules. This completion method eliminates the need for training and computation of depth completion models, resulting in high efficiency.

[0060] Step S406: Based on the processed mask, use a preset image completion model to complete the content of the multi-layer sub-images to obtain multi-layer target sub-images.

[0061] The core idea of ​​the preset image completion model is to repair lost or damaged areas (such as occlusions, scratches, text removal, etc.) in sub-images by modeling the context and generating visually reasonable and semantically consistent pixel content. The preset image completion model can be Image Inpainting based on diffusion mechanisms, AI inpainting models (such as Stable Diffusion Inpaint, SDXL-Inpaint, ControlNet-Inpaint, etc.), Lama, ZITS, etc.

[0062] In practice, the processed mask and sub-image can be input into a preset image completion model. The processed mask can be used as auxiliary information to guide the image completion model to focus on the missing parts, and to focus on the content completion of the effective areas in the sub-image. The output of the preset image completion model is then determined as the completed target sub-image.

[0063] It should be noted that neither the multi-layer target depth sub-image nor the multi-layer target sub-image obtained after content completion contains holes caused by parallax, which can effectively avoid image distortion during subsequent pseudo-3D conversion.

[0064] In practical applications, the process of splitting the original image and the target depth map, as well as the process of completing the content of multi-layer sub-images and multi-layer depth sub-images, can be completed online during image processing or offline during the preprocessing stage. This manual does not limit this process.

[0065] In the above or following embodiments, step S106 determines the rendering strategy of pixels in the target sub-image based on the spatial distance information of pixels within the target sub-image, and re-renders the pixels in the target sub-image according to the rendering strategy to obtain the pseudo-3D sub-image corresponding to the target sub-image. There are various processing methods for this, and the following provides one optional processing method, such as... Figure 5 As shown, this processing method may specifically include the following steps.

[0066] Step S502: Determine the pivot point in each target sub-image and construct a step ray starting from the pivot point.

[0067] In this context, the pivot point refers to the reference point or rotation center for disparity changes within the sub-image. The pivot point can be determined manually, using the bounding box center, semantic center, or depth center, etc., and this embodiment does not limit this method. The bounding box is the smallest cube or cuboid encompassing all objects in the sub-image scene, and the bounding box center is its geometric center. The semantic center can be a center point determined based on the semantic information of objects in the scene (such as importance and visibility). The depth center can be based on the centroid or the maximum depth point of the depth map.

[0068] In practice, a stepping ray can be understood as a series of sampling points obtained by discretizing a continuous ray, and the intersection with the object is detected by progressively advancing these points. A stepping ray can be represented as P(t) = pivot + t * D, where pivot is the pivot point, t is the step distance, and D is the ray direction.

[0069] Step S504: Based on the target depth sub-image corresponding to the target sub-image, control the stepping ray to perform a stepping operation on the target sub-image, and determine the intersection point between the stepping ray and the target sub-image.

[0070] The target sub-image and target depth sub-image are both derived from the processed mask, and there is a correspondence between the target sub-image and the target depth sub-image obtained from the same mask. A stepping distance threshold and a precision threshold can be preset for the stepping ray. When the stepping distance of the stepping ray exceeds the stepping distance threshold, it is considered that the ray has not hit any object, and stepping stops; when the distance of the stepping ray to the object surface is less than the precision threshold, it is considered that the ray has hit the object, and stepping stops.

[0071] In implementation, the pivot point can be used as the camera's optical center, and a three-dimensional coordinate system can be established with the pivot point as the origin (0,0,0). The two-dimensional pixel coordinates (u,v) of the pixels in the target sub-image can then be transformed into this three-dimensional coordinate system to obtain the corresponding three-dimensional spatial coordinates (x,y,z). The coordinate transformation formula can be: localUV is the pixel coordinate, and scale is a scaling factor used to control the scaling ratio of the coordinates.

[0072] For any pixel in the target sub-image, the direction from the pivot point to that pixel can be defined as the stepping direction. Starting from the pivot point, the stepping ray is controlled to advance gradually along the stepping direction (the step size can be fixed or adaptive). After each advance, it is checked whether the current ray position intersects with an object in the scene. Specifically, the distance from the current ray position to the surface of an object in the target sub-image scene is calculated based on the target depth sub-image, and it is determined whether this distance is less than a preset precision threshold. If the distance is less than the threshold, the stepping ray intersects with the surface of an object in the target sub-image scene. The ray then stops advancing, and the intersection point is determined and recorded, including its position and color information. If no intersection occurs, the ray continues advancing until its stepping distance exceeds a preset stepping distance threshold.

[0073] Based on this, multiple stepping directions can be determined for the stepping ray based on multiple pixels in the target sub-image. By controlling the stepping ray to move forward step by step along different stepping directions, multiple intersection points between the target sub-image and the stepping ray can be obtained. Each intersection point represents the position of a pixel in the target sub-image in a three-dimensional coordinate system based on the pivot point.

[0074] In this embodiment, during the stepping operation on the target sub-image, a binary search method can be used to achieve the stepping. By continuously dividing the search interval into two, the range of possible intersection points is quickly narrowed, significantly reducing the number of steps. Furthermore, binary search ensures the accuracy of the intersection point position, avoiding visual defects caused by step size errors. The number of steps or step size can also be adaptively and dynamically adjusted based on the interaction between light and the scene (such as distance and collision risk) or the user's interaction (such as changes in viewpoint), avoiding the inefficiency or insufficient accuracy caused by a fixed step size.

[0075] Step S506: Based on the intersection of the stepping ray and the target sub-image, calculate the rendering strategy of the pixels in the target sub-image, and re-render the pixels in the target sub-image according to the rendering strategy to obtain the pseudo 3D sub-image corresponding to the target sub-image.

[0076] It should be understood that the intersection of the stepping ray and the target sub-image can be represented as a pixel in the 3D scene. The intersection position is not the original pixel position on the target sub-image, but a new position calculated based on depth and the current viewpoint. This new position takes into account the parallax shift caused by depth, and therefore differs spatially from the original position. Based on the color of the intersection position on the target sub-image, the color information of each pixel in the 3D scene is determined. This color information can represent the visual effect of the pixel corresponding to the intersection in the 3D scene. By synthesizing the color information of all pixels into an image, a pseudo-3D sub-image can be obtained. The pseudo-3D sub-image has a visual 3D effect, but it is actually still constructed based on a 2D image and depth information.

[0077] In implementation, the color information of a pixel in a 3D scene can be determined in two ways: First, the color of the intersection point on the target sub-image can be directly determined as the pixel's color information in the 3D scene. Second, the surface normal of the object's surface at the intersection point in the target sub-image can be determined. This surface normal determines the direction of light reflection (such as diffuse reflection or specular reflection). Based on the color of the intersection point, the surface normal, and the lighting direction in the target sub-image, the color information of the pixel in the 3D scene can be calculated.

[0078] Then, the color information determined based on all intersection points can be used as the rendering strategy. According to the correspondence between intersection points and color information contained in the rendering strategy, the original pixels in the target sub-image associated with the intersection points are re-rendered so that the corresponding color information is written into the original pixel positions in the target sub-image, thus obtaining the pseudo-3D sub-image corresponding to the target sub-image.

[0079] In this embodiment, based on the light stepping technology, a two-dimensional to pseudo-three-dimensional conversion is achieved by combining a single depth map and a single RGB image. Its core is to simulate the propagation path of light in three-dimensional space and adjust the pixel values ​​by combining depth information, thereby generating an image with a sense of depth.

[0080] Based on this, in step S108, the pseudo-3D sub-images corresponding to the multi-layer target sub-images are superimposed according to the hierarchical order between the multi-layer target sub-images to obtain the target pseudo-3D image corresponding to the original image. There are various processing methods, and the following provides an optional implementation method, which may specifically include the following steps.

[0081] Step S1082: Obtain the depth value corresponding to each target sub-image. The magnitude of the depth value is used to characterize the hierarchical position of the target sub-image in the hierarchical order. Step S1084: Superimpose the pseudo-3D sub-images corresponding to the multi-layer target sub-images in order of increasing depth value, and associate the pivot points in the multi-layer target sub-images to obtain the target pseudo-3D image corresponding to the original image.

[0082] In this process, the depth values ​​of pixels in different target sub-images fall within different depth ranges. The average depth value of multiple pixels in each target sub-image layer can be determined as the depth value corresponding to that target sub-image. A larger depth value indicates that the pixel is closer to the observation point. The target sub-images can be sorted in ascending order of depth value, and the position of the target sub-image in the sorting result can be determined as its layer position in the hierarchical order. Therefore, the pseudo-3D sub-images corresponding to multiple target sub-images are superimposed according to the hierarchical order between multiple target sub-images, that is, the pseudo-3D sub-images corresponding to multiple target sub-images are superimposed in ascending order of depth value to synthesize the target pseudo-3D image corresponding to the original image.

[0083] Furthermore, it is necessary to associate the pivot points in the multi-layer target sub-images so that all pixels in the target pseudo-3D image are controlled by a single pivot point, thus avoiding pixel distortion when rotating the image.

[0084] The following describes in detail an image processing method provided by an embodiment of this specification, in conjunction with specific application scenarios. Figure 6 As shown, the execution entity of this method can be an image processing system, which includes a depth map generation module, a layer splitting module, an occlusion completion module, a ray stepping rendering module, and a compositing and output module. Specifically, this method may include the following steps: In step S602, the original image to be processed is obtained; In step S604, the depth estimation module is used to predict the depth of the original image to obtain the target depth map corresponding to the original image. In step S606, the layer splitting module is used to split the original image and the target depth map into layers based on the depth distribution of the target depth map, so as to obtain multi-layer sub-images and multi-layer depth sub-images. Step S608: Use the occlusion completion module to complete the content of each sub-image and each depth sub-image to obtain the completed multi-layer target sub-image and multi-layer target depth sub-image; Step S610: Determine the pivot point for each target sub-image layer, and based on the corresponding target depth sub-image, use the ray stepping rendering module to perform ray stepping rendering on the target sub-image to obtain the pseudo 3D sub-image corresponding to each target sub-image layer. Step S612: According to the order of depth values ​​corresponding to the multi-layer target sub-images from small to large, the pseudo-3D sub-images corresponding to the multi-layer target sub-images are superimposed, and the pivot points in the multi-layer target sub-images are associated to output the target pseudo-3D image corresponding to the original image.

[0085] This specification provides an image processing method that utilizes a combined approach of "layer splitting + RGB completion + depth completion + multi-layer independent ray stepping + layer compositing" to realistically restore occluded areas in a single image. Each layer possesses complete RGB and depth information, providing independent and complete terrain for subsequent ray stepping and avoiding depth breaks. Ray stepping is performed on all layers separately, calculating a rendering strategy that reflects true parallax. Based on multi-layer depth relationships, multiple re-rendered layers are accurately composited, effectively solving problems such as pixel stretching and holes caused by occlusion. It supports large parallax shifts without distortion, generating highly realistic pseudo-3D images.

[0086] The above describes the image processing method provided in the embodiments of this specification. Based on the same idea, the embodiments of this specification also provide an image processing apparatus, such as... Figure 7 As shown.

[0087] The image processing device includes: a layering module 701, a rendering module 702, and an overlay module 703, wherein: The layering module 701 is used to acquire the original image to be processed, which is a two-dimensional image; based on the spatial distance information of the pixels in the original image, it splits out multiple sub-images from the original image and performs content completion on the multiple sub-images to obtain multiple target sub-images. The spatial distance information of the pixels is used to characterize the relative distance between different pixels in the original image and the same reference surface. The rendering module 702 is used to determine the rendering strategy of the pixels in the target sub-image based on the spatial distance information of the pixels in the target sub-image, and re-render the pixels in the target sub-image according to the rendering strategy to obtain the pseudo-3D sub-image corresponding to the target sub-image. The overlay module 703 is used to overlay the pseudo-3D sub-images corresponding to the multi-layer target sub-images according to the hierarchical order between the multi-layer target sub-images to obtain the target pseudo-3D image corresponding to the original image.

[0088] In this embodiment of the specification, the layering module 701 includes a layering unit, which is used for: Determine the target depth map corresponding to the original image, the target depth map being used to characterize the spatial distance information of pixels within the original image; Based on the depth distribution of pixels in the target depth map, multiple sub-images are extracted from the original image.

[0089] In the embodiments of this specification, the layered unit is used for: Obtain a depth model for depth prediction of the image; The original image is input into the depth model, the depth of the original image is predicted by the depth model, and the depth map corresponding to the original image is output. The depth map is normalized and inverted to obtain the target depth map corresponding to the original image.

[0090] In the embodiments of this specification, the layered unit is used for: Based on the grayscale value corresponding to each pixel in the target depth map, the pixels in the original image are classified to obtain multiple pixel sets. Based on the multiple pixel sets and the original image, a mask is generated corresponding to each pixel set, and the pixel region composed of the pixel sets represents the effective region in the mask. Based on the masks corresponding to multiple sets of pixels, sub-images associated with each mask are extracted from the original image, and multi-layer sub-images are determined based on the images associated with multiple masks.

[0091] In the embodiments of this specification, the layered unit is further used for: Based on the masks corresponding to multiple pixel sets, a depth sub-map associated with each mask is extracted from the target depth map; Based on the depth sub-images associated with the multiple masks, a multi-layer depth sub-image associated with the target depth map is determined.

[0092] In this embodiment of the specification, the layering module 701 further includes a completion unit, which is used for: Based on the mask corresponding to each set of pixels, boundary detection and erosion processing are performed on the effective areas in the mask to obtain the processed mask. Based on the processed mask, the multi-layer depth sub-image is filled with content using a preset depth completion model to obtain a multi-layer target depth sub-image. Based on the processed mask, the multi-layer sub-image is filled in using a preset image completion model to obtain a multi-layer target sub-image.

[0093] In this embodiment of the specification, the rendering module 702 includes: Determine the pivot point in each target sub-image layer, and construct a step ray starting from the pivot point; Based on the target depth sub-image corresponding to the target sub-image, the stepping ray is controlled to perform a stepping operation on the target sub-image to determine the intersection point between the stepping ray and the target sub-image; Based on the intersection of the stepping ray and the target sub-image, the color information of the pixels in the target sub-image is calculated, and the pseudo-3D sub-image corresponding to the target sub-image is determined based on the color information of the pixels.

[0094] In the embodiments of this specification, the superposition module 703 includes: Obtain the depth value corresponding to each target sub-image layer. The magnitude of the depth value is used to characterize the hierarchical position of the target sub-image in the hierarchical order. The pseudo-3D sub-images corresponding to the multi-layer target sub-images are superimposed in order of increasing depth value, and the pivot points in the multi-layer target sub-images are associated to obtain the target pseudo-3D image corresponding to the original image.

[0095] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more embodiments of this specification, the functions of each module or unit can be implemented in one or more software and / or hardware components, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative; the division of each module and unit is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or modules can be combined or integrated into another system, or some features can be ignored or not executed, etc.

[0096] This specification provides an image processing apparatus that acquires an original image to be processed, splits multiple sub-images from the original image based on the spatial distance information of pixels within the original image, and performs content completion on the multiple sub-images to obtain multiple target sub-images. The spatial distance information of pixels can be used to characterize the relative distance between different pixels in the original image and the same reference surface. Then, based on the spatial distance information of pixels within the target sub-images, a rendering strategy for the pixels in the target sub-images is determined, and the pixels in the target sub-images are re-rendered according to the rendering strategy to obtain pseudo-3D sub-images corresponding to the target sub-images. Subsequently, the pseudo-3D sub-images corresponding to the multiple target sub-images are superimposed according to the hierarchical order between the multiple target sub-images to obtain the target pseudo-3D image corresponding to the original image. In this way, by splitting and completing the original image, each layer can possess complete image information, eliminating problems such as pixel stretching, tearing, and holes caused by occlusion. Then, pseudo-3D transformation is performed on each layer, and the multiple transformed pseudo-3D layers are accurately synthesized, which can effectively improve the quality of the pseudo-3D image.

[0097] The above are the image processing apparatuses provided in the embodiments of this specification. Based on the same idea, the embodiments of this specification also provide an image processing device, such as... Figure 8As shown.

[0098] The image processing device can provide a terminal device or server, etc., for the above embodiments.

[0099] Image processing devices can vary considerably depending on their configuration and performance. They may include a communication interface 802, a user interface 804, a processor 806, and a data storage 808. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 810. The communication interface 802 enables the image processing device 800 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 802 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 802 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 802 may also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 802 may also include multiple physical communication interfaces, such as a Wi-Fi interface, a Bluetooth interface, and a wide-area wireless interface.

[0100] User interface 804 includes receiving user input and providing output to the user. Therefore, user interface 804 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 804 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 804 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, image processing device 800 may support remote access from other devices via communication interface 802 or another physical interface (not shown). User interface 804 may be configured to receive user input, the position and movement of which may be indicated by an indicator or cursor described herein. User interface 804 may also be configured as a display device for rendering or displaying text fragments.

[0101] The processor 806 may contain one or more general-purpose processors and / or special-purpose processors.

[0102] Data storage 808 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 806. Data storage 808 may include removable and non-removable components.

[0103] Processor 806 is capable of executing program instructions 818 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 808 to perform the various functions described herein. Data storage 808 may contain a non-transitory computer-readable medium on which program instructions are stored, which, when executed by image processing device 800, enable image processing device 800 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Execution of program instructions 818 by processor 806 may result in processor 806 using data 812.

[0104] For example, program instructions 818 may include an operating system 822 (e.g., an operating system kernel, device drivers, and / or other modules) installed on the image processing device 800, and one or more application programs 820 (e.g., a browser, social application, or game application). Similarly, data 812 may include operating system data 816 and application data 814. Operating system data 816 is primarily accessible to the operating system 822, while application data 814 is primarily accessible to one or more application programs 820. Application data 814 may reside in a file system visible or hidden from the user of the image processing device 800.

[0105] Application 820 can communicate with operating system 812 through one or more application programming interfaces (APIs). These APIs help application 820 read and / or write application data 814, transmit or receive information via communication interface 802, receive or display information on user interface 804, etc.

[0106] In some terminology, application 820 may be simply referred to as "app". Furthermore, application 820 can be downloaded to image processing device 800 through one or more online app stores or app markets. However, the application can also be installed on image processing device 800 in other ways, such as through a web browser or a physical interface on image processing device 800 (e.g., a USB port).

[0107] Specifically, in this embodiment, the image processing device 800 includes a data storage 808 and one or more program instructions 818, wherein one or more program instructions 818 are stored in the data storage 808, and one or more program instructions 818 are configured to be executed by one or more processors. The one or more program instructions include computer-executable instructions for performing the following: Obtain the original image to be processed, wherein the original image is a two-dimensional image; Based on the spatial distance information of pixels in the original image, multiple sub-images are extracted from the original image, and the content of the multiple sub-images is completed to obtain multiple target sub-images. The spatial distance information of pixels is used to characterize the relative distance between different pixels in the original image and the same reference surface. Based on the spatial distance information of pixels in the target sub-image, the rendering strategy of pixels in the target sub-image is determined, and the pixels in the target sub-image are re-rendered according to the rendering strategy to obtain the pseudo-3D sub-image corresponding to the target sub-image. According to the hierarchical order between the multi-layer target sub-images, the pseudo-3D sub-images corresponding to the multi-layer target sub-images are superimposed to obtain the target pseudo-3D image corresponding to the original image.

[0108] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the image processing device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0109] This specification provides an image processing device that acquires an original image to be processed, splits multiple sub-images from the original image based on the spatial distance information of pixels within the original image, and performs content completion on the multiple sub-images to obtain multiple target sub-images. The spatial distance information of pixels can be used to characterize the relative distance between different pixels in the original image and the same reference surface. Then, based on the spatial distance information of pixels within the target sub-images, a rendering strategy for the pixels in the target sub-images is determined, and the pixels in the target sub-images are re-rendered according to the rendering strategy to obtain pseudo-3D sub-images corresponding to the target sub-images. Subsequently, the pseudo-3D sub-images corresponding to the multiple target sub-images are superimposed according to the hierarchical order between the multiple target sub-images to obtain the target pseudo-3D image corresponding to the original image. In this way, by splitting and completing the original image, each layer can possess complete image information, eliminating problems such as pixel stretching, tearing, and holes caused by occlusion. Then, pseudo-3D conversion is performed on each layer, and the multiple converted pseudo-3D layers are accurately synthesized, which can effectively improve the quality of the pseudo-3D image.

[0110] Furthermore, based on the above Figures 1 to 6This specification also provides a storage medium for storing computer-executable instruction information in one or more embodiments. In one specific embodiment, the storage medium may be a USB flash drive, optical disc, hard disk, etc. When the computer-executable instruction information stored in the storage medium is executed by a processor, it can realize the following process: Obtain the original image to be processed, wherein the original image is a two-dimensional image; Based on the spatial distance information of pixels in the original image, multiple sub-images are extracted from the original image, and the content of the multiple sub-images is completed to obtain multiple target sub-images. The spatial distance information of pixels is used to characterize the relative distance between different pixels in the original image and the same reference surface. Based on the spatial distance information of pixels in the target sub-image, the rendering strategy of pixels in the target sub-image is determined, and the pixels in the target sub-image are re-rendered according to the rendering strategy to obtain the pseudo-3D sub-image corresponding to the target sub-image. According to the hierarchical order between the multi-layer target sub-images, the pseudo-3D sub-images corresponding to the multi-layer target sub-images are superimposed to obtain the target pseudo-3D image corresponding to the original image.

[0111] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the above-described storage medium embodiment is basically similar to the method embodiment, so the description is relatively simple; relevant parts can be referred to the description of the method embodiment.

[0112] This specification provides a storage medium that acquires an original image to be processed, splits multiple sub-images from the original image based on the spatial distance information of pixels within the original image, and performs content completion on the multiple sub-images to obtain multiple target sub-images. The spatial distance information of pixels can be used to characterize the relative distance between different pixels in the original image and the same reference surface. Then, based on the spatial distance information of pixels within the target sub-images, a rendering strategy for the pixels in the target sub-images is determined, and the pixels in the target sub-images are re-rendered according to the rendering strategy to obtain pseudo-3D sub-images corresponding to the target sub-images. Subsequently, the pseudo-3D sub-images corresponding to the multiple target sub-images are superimposed according to the hierarchical order between the multiple target sub-images to obtain the target pseudo-3D image corresponding to the original image. In this way, by splitting and completing the original image, each layer can possess complete image information, eliminating problems such as pixel stretching, tearing, and holes caused by occlusion. Then, pseudo-3D transformation is performed on each layer, and the multiple transformed pseudo-3D layers are accurately synthesized, which can effectively improve the quality of the pseudo-3D image.

[0113] Furthermore, based on the above Figures 1 to 6 This specification also provides one or more embodiments of a computer program product, including a computer program, which, when executed by a processor, can perform the following processes: Obtain the original image to be processed, wherein the original image is a two-dimensional image; Based on the spatial distance information of pixels in the original image, multiple sub-images are extracted from the original image, and the content of the multiple sub-images is completed to obtain multiple target sub-images. The spatial distance information of pixels is used to characterize the relative distance between different pixels in the original image and the same reference surface. Based on the spatial distance information of pixels in the target sub-image, the rendering strategy of pixels in the target sub-image is determined, and the pixels in the target sub-image are re-rendered according to the rendering strategy to obtain the pseudo-3D sub-image corresponding to the target sub-image. According to the hierarchical order between the multi-layer target sub-images, the pseudo-3D sub-images corresponding to the multi-layer target sub-images are superimposed to obtain the target pseudo-3D image corresponding to the original image.

[0114] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the above-described embodiment of a computer program product is relatively simple in description because it is fundamentally similar to the method embodiment; relevant parts can be referred to the description of the method embodiment.

[0115] This specification provides a computer program product that acquires an original image to be processed, splits it into multiple sub-images based on the spatial distance information of pixels within the original image, and completes the content of these sub-images to obtain multiple target sub-images. The spatial distance information of pixels can be used to characterize the relative distance between different pixels in the original image and the same reference surface. Then, based on the spatial distance information of pixels within the target sub-images, a rendering strategy for the pixels in the target sub-images is determined, and the pixels in the target sub-images are re-rendered according to the rendering strategy to obtain pseudo-3D sub-images corresponding to the target sub-images. Subsequently, the pseudo-3D sub-images corresponding to the multiple target sub-images are superimposed according to the hierarchical order between the multiple target sub-images to obtain the target pseudo-3D image corresponding to the original image. In this way, by splitting and completing the original image, each layer can possess complete image information, eliminating problems such as pixel stretching, tearing, and holes caused by occlusion. Then, pseudo-3D transformation is performed on each layer, and the multiple pseudo-3D layers are accurately synthesized, effectively improving the quality of the pseudo-3D image.

[0116] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than those shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous. Moreover, although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is merely one possible execution order among many steps and does not represent the only execution order. Therefore, when method steps are involved in the claims, adjustments to the order of those steps, or parallelism between steps, are also within the scope of protection of the claims.

[0117] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0118] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0119] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0120] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0121] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] Embodiments in this specification are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable parallel device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable parallel device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0123] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable fraud device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0124] These computer program instructions can also be loaded onto a computer or other programmable device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0125] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0126] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0127] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0128] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical or equivalent elements in the process, method, article, or apparatus that includes said element. Furthermore, "a," "an," and "the" are not specifically singular and may include plural forms. Ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish objects. For example, "first server" and "second server" usually refer to two servers, described as "first server" and "second server" to differentiate them; however, sometimes these two servers may be the same server. Moreover, in this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can be indirect receiving and sending (i.e., receiving and sending indirectly through one or more entities). Similarly, in this specification, unless otherwise stated, the relationships between structures can be direct or indirect.

[0129] Furthermore, the specific terms used in this specification to describe embodiments, such as "an embodiment," "one embodiment," or "some embodiments," refer to a particular feature, structure, or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different locations in this specification do not necessarily refer to the same embodiment. Moreover, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples, without contradiction.

[0130] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0132] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0133] The above description is merely an embodiment of this specification and is not intended to limit this document. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims in this document.

Claims

1. An image processing method, the method comprising: Obtain the original image to be processed, wherein the original image is a two-dimensional image; Based on the spatial distance information of pixels in the original image, multiple sub-images are extracted from the original image, and the content of the multiple sub-images is completed to obtain multiple target sub-images. The spatial distance information of pixels is used to characterize the relative distance between different pixels in the original image and the same reference surface. Based on the spatial distance information of pixels in the target sub-image, the rendering strategy of pixels in the target sub-image is determined, and the pixels in the target sub-image are re-rendered according to the rendering strategy to obtain the pseudo-3D sub-image corresponding to the target sub-image. According to the hierarchical order between the multi-layer target sub-images, the pseudo-3D sub-images corresponding to the multi-layer target sub-images are superimposed to obtain the target pseudo-3D image corresponding to the original image.

2. The method according to claim 1, wherein splitting multiple sub-images from the original image based on the spatial distance information of pixels within the original image comprises: Determine the target depth map corresponding to the original image, the target depth map being used to characterize the spatial distance information of pixels within the original image; Based on the depth distribution of pixels in the target depth map, multiple sub-images are extracted from the original image.

3. The method according to claim 2, wherein determining the target depth map corresponding to the original image includes: Obtain a depth model for depth prediction of the image; The original image is input into the depth model, the depth of the original image is predicted by the depth model, and the depth map corresponding to the original image is output. The depth map is normalized and inverted to obtain the target depth map corresponding to the original image.

4. The method according to claim 2, wherein splitting multiple sub-images from the original image based on the depth distribution of pixels in the target depth map comprises: Based on the grayscale value corresponding to each pixel in the target depth map, the pixels in the original image are classified to obtain multiple pixel sets. Based on the multiple pixel sets and the original image, a mask is generated corresponding to each pixel set, and the pixel region composed of the pixel sets represents the effective region in the mask. Based on the masks corresponding to multiple sets of pixels, sub-images associated with each mask are extracted from the original image, and multi-layer sub-images are determined based on the images associated with multiple masks.

5. The method according to claim 4, further comprising: Based on the masks corresponding to multiple pixel sets, a depth sub-map associated with each mask is extracted from the target depth map; Based on the depth sub-images associated with the multiple masks, a multi-layer depth sub-image associated with the target depth map is determined.

6. The method according to claim 5, wherein the step of performing content completion on the multi-layer sub-image to obtain a multi-layer target sub-image includes: Based on the mask corresponding to each set of pixels, boundary detection and erosion processing are performed on the effective areas in the mask to obtain the processed mask. Based on the processed mask, the multi-layer depth sub-image is filled with content using a preset depth completion model to obtain a multi-layer target depth sub-image. Based on the processed mask, the multi-layer sub-image is filled in using a preset image completion model to obtain a multi-layer target sub-image.

7. The method according to claim 1, wherein determining the rendering strategy of pixels in the target sub-image based on the spatial distance information of pixels within the target sub-image, and re-rendering the pixels in the target sub-image according to the rendering strategy to obtain a pseudo-3D sub-image corresponding to the target sub-image, comprises: Determine the pivot point in each target sub-image layer, and construct a step ray starting from the pivot point; Based on the target depth sub-image corresponding to the target sub-image, the stepping ray is controlled to perform a stepping operation on the target sub-image to determine the intersection point between the stepping ray and the target sub-image; Based on the intersection of the stepping ray and the target sub-image, the rendering strategy of the pixels in the target sub-image is calculated, and the pixels in the target sub-image are re-rendered according to the rendering strategy to obtain the pseudo-3D sub-image corresponding to the target sub-image.

8. The method according to claim 7, wherein the step of superimposing the pseudo-3D sub-images corresponding to the multi-layer target sub-images according to the hierarchical order between the multi-layer target sub-images to obtain the target pseudo-3D image corresponding to the original image comprises: Obtain the depth value corresponding to each target sub-image layer. The magnitude of the depth value is used to characterize the hierarchical position of the target sub-image in the hierarchical order. The pseudo-3D sub-images corresponding to the multi-layer target sub-images are superimposed in order of increasing depth value, and the pivot points in the multi-layer target sub-images are associated to obtain the target pseudo-3D image corresponding to the original image.

9. An image processing apparatus, the image processing apparatus comprising: The layering module is used to acquire the original image to be processed, which is a two-dimensional image; based on the spatial distance information of the pixels in the original image, it splits out multiple sub-images from the original image and performs content completion on the multiple sub-images to obtain multiple target sub-images. The spatial distance information of the pixels is used to characterize the relative distance between different pixels in the original image and the same reference surface. The rendering module is used to determine the rendering strategy of the pixels in the target sub-image based on the spatial distance information of the pixels in the target sub-image, and to re-render the pixels in the target sub-image according to the rendering strategy to obtain the pseudo-3D sub-image corresponding to the target sub-image. The overlay module is used to overlay the pseudo-3D sub-images corresponding to the multi-layer target sub-images according to the hierarchical order between the multi-layer target sub-images to obtain the target pseudo-3D image corresponding to the original image.

10. An image processing apparatus, the image processing apparatus comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to: Obtain the original image to be processed, wherein the original image is a two-dimensional image; Based on the spatial distance information of pixels in the original image, multiple sub-images are extracted from the original image, and the content of the multiple sub-images is completed to obtain multiple target sub-images. The spatial distance information of pixels is used to characterize the relative distance between different pixels in the original image and the same reference surface. Based on the spatial distance information of pixels in the target sub-image, the rendering strategy of pixels in the target sub-image is determined, and the pixels in the target sub-image are re-rendered according to the rendering strategy to obtain the pseudo-3D sub-image corresponding to the target sub-image. According to the hierarchical order between the multi-layer target sub-images, the pseudo-3D sub-images corresponding to the multi-layer target sub-images are superimposed to obtain the target pseudo-3D image corresponding to the original image.

Images