Image processing method and device, storage medium, equipment and program product

Abstract

The invention discloses an image processing method and device, a storage medium, equipment and a program product, and the method comprises the steps: obtaining a cue word and a scene grid, the cue word being text information which is provided by a user and represents a scene style, and the scene grid being a three-dimensional grid which is generated based on real scene reconstruction and has real textures; generating a stylized panoramic texture map at the central position of the scene grid according to the prompt word; projecting the texture of the stylized panoramic texture map to a visible area of the scene grid to obtain a first stylized grid texture map; and performing spatial texture propagation processing on the first stylized grid texture map to fill an invisible area of the scene grid to obtain a second stylized grid texture map. According to the method, the stylized grid texture map can be automatically generated according to the prompt word provided by the user, meanwhile, the space consistency and the map correctness are ensured, and a complex geometric structure in a real scene can be completely covered.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, apparatus, storage medium, device and program product. Background Art

[0002] In the fields of virtual reality (VR) and mixed reality (MR), by transferring the style of three-dimensional scene expressions, it is possible to achieve personalized dressing of the user's real scene, such as changing the user's bed in the virtual scene screen displayed by the virtual reality device into a game-style bed, and changing the ceiling into a starry sky style, etc., so as to provide a rich and immersive experience. For example, a typical three-dimensional scene expression can be a mesh. However, current technology cannot automatically generate stylized mesh texture maps, and cannot guarantee spatial consistency and mapping correctness, and it is difficult to completely cover the complex geometric structures in the real scene. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, apparatus, storage medium, device and program product, which can automatically generate a stylized grid texture map based on prompt words provided by a user, while ensuring spatial consistency, mapping correctness and complete coverage of complex geometric structures in real scenes.

[0004] On the one hand, an embodiment of the present application provides an image processing method, which includes: obtaining a prompt word and a scene grid, wherein the prompt word is text information provided by a user to represent a scene style, and the scene grid is a three-dimensional grid with real texture generated based on real scene reconstruction; according to the prompt word, a stylized panoramic texture map is generated at the center position of the scene grid; the texture of the stylized panoramic texture map is projected to the visible area of ​​the scene grid to obtain a first stylized grid texture map; and the first stylized grid texture map is subjected to spatial texture propagation processing to fill the non-visible area of ​​the scene grid to obtain a second stylized grid texture map.

[0005] On the other hand, an embodiment of the present application provides an image processing device, the device comprising:

[0006] An acquisition unit, used to acquire a prompt word and a scene mesh, wherein the prompt word is text information representing the scene style provided by a user, and the scene mesh is a three-dimensional mesh with real texture generated based on the reconstruction of a real scene;

[0007] A first processing unit, configured to generate a stylized panoramic texture map at a center position of the scene grid according to the prompt word;

[0008] A second processing unit, configured to project the texture of the stylized panoramic texture map to a visible area of ​​the scene grid to obtain a first stylized grid texture map;

[0009] The third processing unit is configured to perform spatial texture propagation processing on the first stylized mesh texture map to fill the non-visible area of ​​the scene mesh and obtain a second stylized mesh texture map.

[0010] On the other hand, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the image processing method described in any of the above embodiments.

[0011] On the other hand, an embodiment of the present application provides a terminal device, which includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the image processing method described in any of the above embodiments by calling the computer program stored in the memory.

[0012] On the other hand, an embodiment of the present application provides a computer program product, including a computer program, which implements the image processing method described in any of the above embodiments when executed by a processor.

[0013] The embodiment of the present application obtains prompt words and scene grids, where the prompt words are text information provided by the user to represent the scene style, and the scene grid is a three-dimensional grid with real texture generated based on the reconstruction of the real scene; based on the prompt words, a stylized panoramic texture map is generated at the center of the scene grid; the texture of the stylized panoramic texture map is projected to the visible area of ​​the scene grid to obtain a first stylized grid texture map; the first stylized grid texture map is subjected to spatial texture propagation processing to fill the non-visible area of ​​the scene grid to obtain a second stylized grid texture map. The embodiment of the present application can automatically generate a stylized grid texture map based on the prompt words provided by the user, while ensuring spatial consistency, mapping correctness, and being able to completely cover complex geometric structures in real scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for describing the embodiments are briefly introduced below.

[0015] Figure 1 A first flow chart of the image processing method provided in an embodiment of the present application.

[0016] Figure 2 A schematic diagram of a first application scenario of the image processing method provided in an embodiment of the present application.

[0017] Figure 3A schematic diagram of a second application scenario of the image processing method provided in an embodiment of the present application.

[0018] Figure 4 A schematic diagram of a third application scenario of the image processing method provided in an embodiment of the present application.

[0019] Figure 5 A schematic diagram of a fourth application scenario of the image processing method provided in an embodiment of the present application.

[0020] Figure 6 A second flow chart of the image processing method provided in an embodiment of the present application.

[0021] Figure 7 A schematic diagram of a fifth application scenario of the image processing method provided in an embodiment of the present application.

[0022] Figure 8 A schematic diagram of a sixth application scenario of the image processing method provided in an embodiment of the present application.

[0023] Fig. 9 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application.

[0024] Fig.10 A first structural diagram of a terminal device provided in an embodiment of the present application.

[0025] Fig.11 A second structural diagram of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0027] The embodiments of the present application provide an image processing method, apparatus, computer-readable storage medium, terminal device, server, and computer program product. Specifically, the image processing method of the embodiments of the present application can be executed by a terminal device or a server.

[0028] The embodiments of the present application can be applied to various application scenarios such as extended reality (eXtended ed Reality, XR), virtual reality (Virtual Reality, VR), augmented reality (Augmented Reality, AR), and mixed reality (Mixed Reality, MR).

[0029] First, some nouns or terms that appear in the description of the embodiments of the present application are explained as follows:

[0030] A virtual scene is a virtual scene displayed (or provided) when an application is running on a terminal or server. Optionally, the virtual scene is a simulation of the real world, or a semi-simulation and semi-fictitious virtual environment, or a purely fictitious virtual environment. A virtual scene is any of a two-dimensional virtual scene and a three-dimensional virtual scene. The virtual environment can be the sky, land, ocean, etc., wherein the land includes environmental elements such as deserts and cities. Among them, a virtual scene is a scene with complete game logic of virtual objects such as user control.

[0031] Extended reality (XR) is a concept that includes virtual reality (VR), augmented reality (AR) and mixed reality (MR). It refers to the technology that creates an environment in which the virtual world is connected to the real world, and users can interact with the environment in real time.

[0032] Virtual Reality (VR) is a technology for creating and experiencing a virtual world. It generates a virtual environment by computation. It is a multi-source information (the virtual reality mentioned in this article includes at least visual perception, and can also include auditory perception, tactile perception, motion perception, and even taste perception, olfactory perception, etc.). It realizes the simulation of the fusion and interactive three-dimensional dynamic vision and entity behavior of the virtual environment, immersing users in a simulated virtual reality environment, and realizing its application in various virtual environments such as maps, games, videos, education, medical treatment, simulation, collaborative training, sales, assisted manufacturing, maintenance and repair.

[0033] Augmented Reality (AR) is a technology that calculates the camera's posture parameters in the real world (or three-dimensional world, real world) in real time during the process of the camera capturing images, and adds virtual elements to the images captured by the camera based on the camera posture parameters. Virtual elements include but are not limited to: images, videos and three-dimensional models. The goal of AR technology is to integrate the virtual world with the real world on the screen for interaction.

[0034] Mixed Reality (MR) is a simulated setting that integrates computer-created sensory input (e.g., virtual objects) with sensory input from a physical setting or its representation. In some MR settings, the computer-created sensory input can adapt to changes in sensory input from the physical setting. In addition, some electronic systems used to present MR settings can monitor the orientation and / or position relative to the physical setting so that virtual objects can interact with real objects (i.e., physical elements from the physical setting or their representations). For example, the system can monitor movement so that virtual plants appear stationary relative to physical buildings.

[0035] Augmented Virtuality (AV): An AV set is a simulated set that is a computer-created set or a virtual set that incorporates at least one sensory input from a physical set. The one or more sensory inputs from the physical set may be a representation of at least one feature of the physical set. For example, a virtual object may present the color of a physical element captured by one or more imaging sensors. As another example, a virtual object may present features consistent with actual weather conditions in the physical set, as identified via weather-related imaging sensors and / or online weather data. In another example, an augmented reality forest may have virtual trees and structures, but animals may have features accurately reproduced from images taken of the physical animals.

[0036] Virtual field of view refers to the area in the virtual environment that the user can perceive through the lens in the virtual reality device. The field of view (FOV) of the virtual field of view is used to represent the perceived area.

[0037] Virtual reality devices, terminals for realizing virtual reality effects, can usually be provided in the form of glasses, helmet-mounted displays (Head Mount Display, HMD), and contact lenses to realize visual perception and other forms of perception. Of course, the form of virtual reality devices is not limited to this, and can be further miniaturized or enlarged as needed.

[0038] The virtual reality devices described in the embodiments of the present application may include but are not limited to the following types:

[0039] Computer-based virtual reality (PCVR) devices use the PC to perform relevant calculations and data output for virtual reality functions. External computer-based virtual reality devices use the data output by the PC to achieve virtual reality effects.

[0040] Mobile virtual reality devices support the setting of mobile terminals (such as smart phones) in various ways (such as head-mounted displays with dedicated card slots). Through wired or wireless connection with the mobile terminal, the mobile terminal performs relevant calculations of virtual reality functions and outputs data to the mobile virtual reality device, such as watching virtual reality videos through the mobile terminal's APP.

[0041] The all-in-one virtual reality device has a processor for performing relevant calculations of virtual functions, and thus has independent virtual reality input and output functions. It does not need to be connected to a PC or mobile terminal and has a high degree of freedom in use.

[0042] UV is the abbreviation of texture mapping coordinate UVW (W coordinate is not commonly used), which defines the position coordinate information of each point on the map. The gap between points is processed by the software for image smoothing interpolation, and the relationship between the XYZ coordinates of the three-dimensional model determines the distribution position of the texture map; each three-dimensional model is composed of countless faces, and the process of tiling the three-dimensional faces on the two-dimensional canvas is called UV development. U and V represent the horizontal coordinate axis and the vertical coordinate axis respectively. The UV map is a two-dimensional texture image.

[0043] In the fields of virtual reality (VR) and mixed reality (MR), a user's real scene can be personalized by using a three-dimensional expression, such as changing the user's bed in a virtual scene displayed by a virtual reality device into a game-style bed, and changing the ceiling into a starry sky style, etc., to provide a rich and immersive experience. For example, the three-dimensional expression can be a mesh.

[0044] However, most of the current 3D personalization solutions rely on materials pre-made by artists, or use 3D modeling software to manually constrain the surface area of ​​the mesh for texture mapping (such as AI-based texture mapping generation tools), and cannot simply perform automated full-scene personalized texture generation based on user text input; in addition, current text-generated image solutions (such as the Vincent graph model) can only be applied to 2D images from a single camera perspective (such as the depth-based Vincent graph control model), and current text-driven automated Mesh texture generation solutions (such as first generating a multi-perspective 2D image corresponding to 3D, such as generating it by rotating around the center of the object, and then automatically pasting the 2D image texture onto the 3D Mesh) can only be applied to the object level, and cannot simultaneously ensure spatial consistency (such as 360° uniform-toned wallpaper) and mapping correctness (such as pasting the chair leg texture to the ground behind), and fully map it to the Mesh of the 3D scene.

[0045] For example, an AI texture map generation tool can be a plug-in that uses an AI model for texture generation and painting in free open source 3D graphics software, the principle of which is to project a map from the view angle.

[0046] For example, the text-to-image model can be a text-to-image model based on Latent Diffusion Models (LDMs). Its principle is that an image can be generated based on a text description as input, or an image and a text description can be used as input to modify the image.

[0047] For example, the text-to-image control model can be a neural network structure that controls the text-to-image model by adding additional conditions. It provides a method to enhance stable diffusion, and using input condition control in the text-to-image generation process can make the generated image closer to the input image.

[0048] The current image texture generation algorithm cannot be applied to the mesh at the 3D scene level. There are three main technical difficulties:

[0049] 1. Globally consistent high-definition texture generation: Current methods can only generate local textures or textures of a single object, but cannot generate coherent scene textures under a 360° perspective (such as ensuring that the wall wallpaper of the scene has a coherent color, or that the bed across multiple camera perspectives has a coherent texture, etc.), and the clarity is not high enough (for example, usually only 512*512 resolution).

[0050] 2. The generated texture is difficult to match with the geometry: Although the current geometric texture generation can use the textural graph control model for depth conditioning, in scene-level applications, due to the presence of many delicate and complex geometric structures in real scenes (such as chair legs, etc.), the generated texture often cannot correspond to these geometric structures, which will lead to misalignment during automatic texture projection (pasting 2D image texture onto 3D Mesh).

[0051] 3. Complex geometric structures in real scenes have complex occlusion relationships, which are difficult to fully cover using texture projection: for example, the gap between the bottom of the sofa and the ground is very small, and it is difficult to apply texture to the Mesh by re-projecting the texture from the camera's perspective.

[0052] Therefore, a solution is needed that can automatically generate stylized mesh texture maps based on user-provided prompts, while ensuring spatial consistency, mapping correctness, and full coverage of complex geometric structures in real scenes.

[0053] The embodiments of the present application propose an image processing method, apparatus, storage medium, device and program product, which can generate meaningful and beautiful stylized mesh texture maps of real scenes according to the prompt words provided by the user, while maintaining semantic consistency and spatial coherence (for example, the furniture still looks like the type of furniture, but is presented in different texture styles). In order to meet the requirements of global consistency, the embodiments of the present application propose to use a top-down approach to generate textures for the scene mesh in a panoramic texture space. Specifically, the embodiments of the present application first generate an initial stylized panoramic texture map at the center of the scene mesh during the panoramic texture generation process, and then project the texture of the stylized panoramic texture map to the visible area of ​​the scene mesh to obtain a first stylized mesh texture map, and then propagate the texture to the remaining non-visible areas of the scene to obtain a second stylized mesh texture map. The second stylized mesh texture map finally generated has been baked into the scene mesh through a two-dimensional texture (UV) map, and can be used to immersively experience stylized scenes on head-mounted display devices such as VR, XR, and AR.

[0054] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0055] Each embodiment of the present application provides an image processing method, which can be executed by a terminal or a server, or by a terminal and a server together; the embodiments of the present application are described by taking the execution of the image processing method by a terminal (terminal device) as an example.

[0056] See also Figures 1 to 8 , Figure 1 and Figure 6 A schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application, Figures 2 to 5 , Figures 7 and 8 All of them are schematic diagrams of application scenarios of the image processing method provided in the embodiments of the present application. The method can be applied to a terminal device, which may include any one of an extended reality device, a virtual reality device, an augmented reality device, and a mixed reality device. The method includes:

[0057] Step 110, obtaining prompt words and scene meshes, wherein the prompt words are text information representing the scene style provided by the user, and the scene mesh is a three-dimensional mesh with real texture generated based on the reconstruction of the real scene.

[0058] For example, a prompt word P provided by a user and a scene mesh (TexturedMesh) with real textures of a reality reconstruction are obtained. Specifically, the prompt word P provided by the user can be obtained through user input or automatic recognition technology, and the prompt word P can be text information. At the same time, a scene mesh (TexturedMesh) with real textures of a reality reconstruction is obtained through 3D scanning or image processing technology.

[0059] Step 120: Generate a stylized panoramic texture map at the center of the scene grid according to the prompt word.

[0060] In step 120, texture stylization may be performed based on the scene of the panoramic view to obtain a stylized panoramic texture map. Specifically, a stylized panoramic texture map with a high degree of scene consistency is achieved by utilizing personalized processing driven by text in the panoramic view space.

[0061] In some embodiments, the stylized panoramic texture map includes a style-priority panoramic map, and generating the stylized panoramic texture map at the center of the scene grid according to the prompt word includes:

[0062] Generate a style-priority panorama at the center of the scene grid according to the prompt word;

[0063] The stylized panoramic texture map is determined according to the style-priority panoramic map.

[0064] In some embodiments, generating a style-priority panorama at the center of the scene grid according to the prompt word includes:

[0065] Rendering a true color image and a depth image of the real scene at the center of the scene grid;

[0066] Using a pre-trained diffusion model to process the prompt word, the true color image, and the depth image to generate an initial style-priority panoramic image;

[0067] The initial style-priority panorama is magnified by a preset multiple based on a super-resolution magnification technology to obtain the style-priority panorama, wherein the resolution of the style-priority panorama is greater than the resolution of the initial style-priority panorama.

[0068] Among them, the pre-trained diffusion model is used to predict the prompt word P and the true color (RGB) image I P and the depth map D to generate a seamless and high-resolution style-priority panorama, and determine the style-priority panorama as a stylized panoramic texture map.

[0069] For example, the stylized panoramic texture map may include a style-priority panoramic map, and step 120 may be implemented by steps 121 to 123, specifically:

[0070] Step 121, rendering a true color image and a depth image of the real scene at the center position of the scene grid.

[0071] Among them, the true color (RGB) image of the real scene is rendered at the center of the scene grid. P And depth map D. Specifically, the RGB image I can be extracted from the scene mesh (Textured Mesh) P And depth map D. Among them, RGB map I P Represents the color information of each pixel on the surface of the scene grid; the depth map D represents the distance from each pixel on the surface of the scene grid to the camera center c.

[0072] Among them, true color images refer to images presented in real colors, usually images captured by color cameras or video cameras. These images contain detailed information such as the color and texture of objects in the scene, and can provide visual perception of the real scene. Depth map refers to an image with depth information, usually obtained by a depth sensor (such as a ToF camera, a structured light camera, or a stereo vision camera); in an embodiment of the present application, the depth map can be directly obtained by grid rendering when the scene grid is known. The depth map can provide distance information of objects in the scene from the camera, usually in the form of a grayscale image or color coding.

[0073] Step 122: Use a pre-trained diffusion model to process the prompt words, the true color image, and the depth image to generate an initial style-priority panoramic image.

[0074] The pre-trained diffusion model is obtained by pre-training using a panoramic image.

[0075] Specifically, in order to generate a high-resolution style-priority panorama that meets the requirements, the present application designs a coarse-to-fine generation scheme.

[0076] First, the RGB image I P and the depth map D as the input condition control of the diffusion model (using the Vincent map control model technology) to generate a low-resolution initial style-first panorama

[0077]

[0078] Among them, F C It is a diffusion model with multiple conditional controls, ε(I P )) is the soft edge (SoftEdge) map extracted by the pixel difference network (PiDiNet), P is the prompt word, and D is the depth map.

[0079] For example, this low-resolution initial style-first panorama The size can be 1024*512.

[0080] For example, an end-to-end neural network (Vincent image control model) can be used to control the RGB image and the depth image D as input conditions of the diffusion model.

[0081] Among them, the pixel difference network (PiDiNet) is used for effective edge detection.

[0082] Step 123 , based on the super-resolution upscaling technology, the initial style-priority panorama is enlarged by a preset multiple to obtain the style-priority panorama, wherein the resolution of the style-priority panorama is greater than the resolution of the initial style-priority panorama.

[0083] For example, using super-resolution upscaling technology, the low-resolution initial style priority panorama Zoom in by a preset factor (e.g. 3 times) to get a high-resolution style-first panorama

[0084] During inference, an asymmetric tiling strategy can be adopted to ensure that the left and right images of the panorama are continuous (for example, keeping the same tones and continuous patterns on both sides of the wall and furniture) by replacing the 2D convolution kernel filling pattern of the UNet structure in the diffusion model with ring filling.

[0085] Among them, the UNet structure is a convolutional neural network for image segmentation. The UNet network consists of a contraction path and an expansion path. There is a jump connection structure between the network layers corresponding to the contraction path and the expansion path.

[0086] Among them, the panoramic texture generation scheme described in step 120 can be widely used in various applications that require panoramic texture generation, such as game production, virtual reality, augmented reality and other scenarios. It not only takes into account the color and shape of the texture, but also combines the prompt words and depth information provided by the user. It can quickly and accurately generate high-quality panoramic textures, thereby improving the performance of related applications and user experience.

[0087] In some embodiments, the stylized panoramic texture map includes the style-priority panoramic map and the alignment-priority panoramic map, and determining the stylized panoramic texture map according to the style-priority panoramic map includes:

[0088] Generate an alignment-priority panoramic image according to the prompt word, the true color image and the depth image;

[0089] The style-priority panorama and the alignment-priority panorama are texture-mixed to generate the stylized panorama texture map.

[0090] In some embodiments, generating the alignment-priority panoramic image according to the prompt word, the true color image and the style-priority panoramic image includes:

[0091] Performing denoising on the true color image to obtain a denoised true color image;

[0092] Using the diffusion model, the prompt word, the denoised true color image and the style-priority panorama are processed to generate an initial alignment-priority panorama;

[0093] The initial alignment priority panoramic image is magnified by a preset multiple based on the super-resolution magnification technology to obtain the alignment priority panoramic image, wherein the resolution of the alignment priority panoramic image is greater than the resolution of the initial alignment priority panoramic image.

[0094] In some embodiments, the step of performing texture blending on the style-priority panorama and the alignment-priority panorama to generate the stylized panorama texture map includes:

[0095] Generating a first depth edge filtering mask according to an edge detection result at a transition point of the depth map, wherein the first depth edge filtering mask is used to characterize a pixel area of ​​a mixture of the style-priority panorama and the alignment-priority panorama;

[0096] According to the first depth edge filtering mask, a Poisson image editing algorithm is used to mix the textures of the style-first panorama and the alignment-first panorama to obtain the stylized panorama texture map.

[0097] Among them, although the RGB image and the depth map are used as input conditions of the diffusion model by using the text-generated image control model, the diffusion model can be effectively guided to generate a texture that is basically aligned with the scene grid. However, in the scene-level texture generation task, since the geometric shape of the real scene is usually more complex, the alignment of the initial generated texture often cannot meet the requirements. In addition, if the generated texture is forcibly required to be aligned with the real scene (such as using a diffusion model with a lower denoising strength to achieve the text-generated image), it will often cause the generated stylized panorama to be blurred or the style is not beautiful. Therefore, in step 120, a two-way texture alignment scheme can also be adopted, by generating two-way textures of the style-priority panorama and the alignment-priority panorama respectively, and then using the edge detection at the jump point of the depth map as a reference to mix the two-way textures to obtain a stylized panoramic texture map.

[0098] See also Figure 2 The first application scenario schematic diagram shown in FIG. Figure 2In the schematic diagram of two-way edge alignment shown in (a), a style-priority panorama is generated from a reference image through a panoramic texture generation step, and then an alignment texture is generated based on the reference image and the style information extracted from the style-priority panorama to obtain an alignment-priority panorama, and the textures of the style-priority panorama and the alignment-priority panorama are mixed based on a first depth edge filtering mask generated by depth edge detection to obtain a mixed panorama (stylized panoramic texture map). Figure 2 (b) shows a schematic diagram of the alignment result visualization. After two-way edge alignment, the aligned image can repair the misalignment and maintain the style compared to the image before alignment.

[0099] For example, the stylized panoramic texture map includes a style-priority panoramic map and an alignment-priority panoramic map, and step 120 can also be implemented through steps 121 to 127. Specifically, the style-priority panoramic map can be generated through the following steps 121 to 123, and then the alignment-priority panoramic map can be generated through the following steps 124 to 125, and then the stylized panoramic texture map can be obtained by mixing the two-way texture through steps 126 to 127. Specifically:

[0100] Step 121, rendering a true color image and a depth image of the real scene at the center position of the scene grid.

[0101] Step 122: Use a pre-trained diffusion model to process the prompt words, the true color image, and the depth image to generate an initial style-priority panoramic image.

[0102] Step 123 , based on the super-resolution upscaling technology, the initial style-priority panorama is enlarged by a preset multiple to obtain the style-priority panorama, wherein the resolution of the style-priority panorama is greater than the resolution of the initial style-priority panorama.

[0103] The specific implementation of steps 121 to 123 can refer to the specific description in the above embodiments, which will not be elaborated here.

[0104] Step 124: input the prompt word, the true color image, the depth image, and the style-priority panoramic image into the diffusion model for processing to generate an initial alignment-priority panoramic image.

[0105] Specifically, the prompt word, true color (RGB) image, depth map and style priority panorama are used as input, and the initial alignment priority panorama is generated through the diffusion model. In order to maintain the consistency of style, the style information T(I S ) as part of the conditional control.

[0106] To generate the initial alignment-first panorama This application uses edge detection algorithm to denoise the panoramic RGB image of the real world, and then uses the conditional control technology of the Wensheng image control model to generate the initial alignment priority panoramic image

[0107]

[0108] Among them, F C is a diffusion model with multiple conditional controls; P is the prompt word; C(I p ) is the edge detection algorithm for the true color image I p The edge detection result obtained after edge detection is used for the true color image I p Denoising; T(I S ) is the style information extracted from the style-priority panorama using the image-generated style control module to maintain the consistency of style.

[0109] The edge detection algorithm used in the embodiment of the present application is a method for detecting edges using a multi-level edge detection algorithm. It is used to detect edges or contours in an image. The basic idea of ​​this algorithm is to first find the obvious edges in the image, and then determine the position and direction of the edge by calculating the gradient of the pixels in the image.

[0110] The image style control module is a functional module used to control the generated image area to follow the style and semantics of the original image in large-scale image diffusion models. It is often used to control the newly generated details of the algorithm to follow the details of the original image in tasks such as image enlargement and detail restoration.

[0111] Step 125 , based on the super-resolution upscaling technology, the initial alignment priority panoramic image is enlarged by a preset multiple to obtain the alignment priority panoramic image, wherein the resolution of the alignment priority panoramic image is greater than the resolution of the initial alignment priority panoramic image.

[0112] Specifically, in order to make the initial alignment prioritize the panorama Panoramic with style priority The same size, using super-resolution enlargement algorithm, the initial alignment priority panorama is set according to the preset multiple (such as three times). Zoom into the panorama with style priority Same size, get alignment-first panorama

[0113] Step 126: Generate a first depth edge filtering mask according to the edge detection result at the transition point of the depth map, wherein the first depth edge filtering mask is used to characterize a pixel area of ​​a mixture of the style-priority panorama and the alignment-priority panorama.

[0114] Specifically, since alignment misalignment problems often occur where the scene depth changes significantly, the depth edge jump area can be detected from the depth map to generate a first depth edge filter mask, which can be used to characterize the pixel areas of the mixed style priority panorama and the alignment priority panorama. In order to ensure that the generated texture does not appear broken or blurred at the depth edge, the jump area is detected at the depth edge as a reference. Through the image processing algorithm, the first depth edge filter mask is generated according to the edge detection results at the jump of the depth map. The first depth edge filter mask can be used to indicate which pixel areas should be blended and which pixel areas should be kept as is.

[0115] Among them, this step mainly identifies depth edges by analyzing the changes in pixel values ​​in the depth map. Generally, depth edges correspond to the outlines or boundaries of objects in the image, and the pixel values ​​in these areas will change significantly. By setting an appropriate threshold, these depth edges can be detected and a first depth edge filtering mask can be generated.

[0116] Step 127 : According to the first depth edge filtering mask, a Poisson image editing algorithm is used to mix the textures of the style-priority panoramic image and the alignment-priority panoramic image to obtain the stylized panoramic texture image.

[0117] Specifically, the generated first depth edge filtering mask can be applied to the style-first panorama and the alignment-first panorama, and then the two-pass textures can be blended using the Poisson image editing algorithm. This can significantly alleviate the geometry-texture misalignment problem while keeping the desired stylized details unchanged.

[0118] Among them, once the depth edge is detected, the Poisson image editing algorithm can be used to blend the textures on both sides of the depth edge. The Poisson image editing method mainly satisfies certain boundary conditions so that the gradient field of the two-dimensional function to be solved in the region is as similar as possible. The boundary condition here means that the function value is equal to the value of the target image on the boundary. By solving the two-dimensional function that meets this boundary condition, a stylized panoramic texture image after mixing the style-first panorama and the alignment-first panorama can be obtained.

[0119] See also Figure 3 In the schematic diagram of the second application scenario shown in FIG. 1 , during the panoramic texture generation process in step 120, the prompt words "AAA...BBB...CCC..." described by the user are obtained; then a three-dimensional scene grid with real texture is generated based on the real scene reconstruction; then a style-priority panorama is generated at the center of the scene grid according to the prompt words; then a panoramic diffusion is performed to generate an alignment-priority panorama; then a two-way texture alignment is performed between the style-priority panorama and the alignment-priority panorama, thereby generating a high-definition stylized panoramic texture map.

[0120] Step 130 : Projecting the texture of the stylized panoramic texture map onto the visible area of ​​the scene mesh to obtain a first stylized mesh texture map.

[0121] After the stylized panoramic texture map is obtained in step 120, the texture of the stylized panoramic texture map can be projected to the visible area of ​​the three-dimensional scene grid through the UV map in the panoramic space to obtain the first stylized grid texture map.

[0122] In some embodiments, step 130 may be implemented through steps 131 to 135, specifically:

[0123] Step 131, obtaining a two-dimensional texture image of the scene grid and texture coordinates corresponding to each pixel point in the two-dimensional texture image.

[0124] The two-dimensional texture image is a UV map. In step 131, the UV map of the scene grid and the texture coordinates corresponding to each pixel in the UV map are obtained. This step can be implemented by software tools or algorithms. For example, a 3D modeling tool can directly generate a UV map and texture coordinates when creating a three-dimensional model.

[0125] Step 132: Calculate the spatial coordinates of each pixel point of the two-dimensional texture image in the three-dimensional space by interpolation according to the two-dimensional texture image and the texture coordinates.

[0126] For example, according to the UV map and the texture coordinates, the spatial coordinates of each pixel point of the UV map in the three-dimensional space are calculated by interpolation. The specific interpolation method may be a method such as barycentric interpolation.

[0127] For example, according to the texture coordinates of each valid pixel point p in the UV map, the spatial coordinate x of each pixel point p is calculated by interpolation:

[0128] x = Interp(MapTex(TexCoord(p), {T})), (3)

[0129] Where TexCoord(p) is the texture coordinate of each pixel p, {T} is the triangle of the scene mesh Mesh, MapTex(·) maps the texture coordinate to the triangle vertices with barycentric weights, and each spatial coordinate x is obtained by barycentric interpolation from the triangle vertices.

[0130] Step 133, according to the spatial coordinates of each pixel point of the two-dimensional texture image in the three-dimensional space, calculate the ray direction between each pixel point of the two-dimensional texture image and the camera, and map the ray direction to the stylized panoramic texture map through equirectangular projection.

[0131] Among them, for the spatial coordinates of each pixel point in three-dimensional space, the ray direction between each pixel point and the camera is calculated, and the ray direction is mapped to the stylized panoramic texture map through equirectangular projection.

[0132] In this step, according to the spatial coordinate x of each pixel point p and the position coordinate c of the center of the camera, the direction of the ray starting from the position coordinate c of the center of the camera and passing through the spatial coordinate x of the pixel point is calculated, where the ray direction d = cx / ||cx||; then, the ray direction is mapped to the stylized panoramic texture map through equirectangular projection. Equirectangular projection is a method of projecting an object in a three-dimensional space onto a two-dimensional plane, which can keep the shape and relative position of the object unchanged.

[0133] Step 134, determine the visibility information of each pixel point of the two-dimensional texture image under camera observation, and generate a first visibility mask according to the visibility information, wherein the visibility information is used to characterize whether the spatial coordinates of each pixel point of the two-dimensional texture image are visible when observed from the position coordinates of the center of the camera.

[0134] In this step, the visibility information of each pixel under camera observation is determined according to the distance between each pixel and the camera and the depth information of the rendered scene, and then a first visibility mask is generated according to the visibility information. The specific visibility test method may adopt a depth test method or the like.

[0135] For example, for the spatial coordinate x of each pixel point p, the observation distance of each pixel point p is compared with the depth information of the rendered scene to determine the visibility information of each pixel point under the camera observation, and the visibility information is used to indicate whether the spatial coordinate x of the pixel point p is visible when observed from the position coordinate c of the center of the camera. The observation distance of each pixel point p is the distance between the spatial coordinate x of each pixel point p and the position coordinate c of the center of the camera.

[0136] This method traverses all pixels of the UV map through visibility testing, obtains the visibility information of each pixel under camera observation, and forms a first visibility mask M in the UV space. init_vis , as shown below:

[0137]

[0138] The distance threshold ∈=0.01.

[0139] Step 135: assigning the panoramic color of the stylized panoramic texture map to a two-dimensional texture space according to the first visibility mask and the ray direction, so as to project the texture of the stylized panoramic texture map to the visible area of ​​the scene mesh, and obtain a first stylized mesh texture map.

[0140] The two-dimensional texture space is a UV space. In step 135, the panoramic color of the stylized panoramic texture image is assigned to the UV space according to the first visibility mask and the ray direction. In this step, the panoramic color is assigned to the UV space according to the first visibility mask and the ray direction, completing the operation of projecting the panoramic texture onto the three-dimensional grid. The specific assignment method can be designed according to actual needs.

[0141] In this step, according to the first visibility mask M init_vis And the corresponding ray direction d, assign the panoramic color of the stylized panoramic texture map to the UV space, complete the operation of projecting the texture of the stylized panoramic texture map to the visible area of ​​the three-dimensional scene grid to obtain the first stylized grid texture map, so as to achieve partial texture stylization of the scene and improve the effect and quality of texture mapping.

[0142] Step 140: Perform spatial texture propagation processing on the first stylized mesh texture map to fill the non-visible area of ​​the scene mesh to obtain a second stylized mesh texture map.

[0143] In step 140, a completed image can be obtained based on spatial texture propagation of multi-view panoramic view completion, specifically, the texture of the previously blocked area is filled through the panoramic view completion algorithm to obtain the completed image. Then, the non-visible area in the completed image is filled based on the spatial texture prediction network to obtain a second stylized grid texture map, specifically, the reasonable texture of the texture-uncovered area can be predicted by using the spatial texture prediction network to pre-learn from the current texture stylization map.

[0144] Through the above step 130, after the texture of the stylized panoramic texture map is projected to the visible area of ​​the three-dimensional scene grid to obtain the first stylized grid texture map, there are still some uncovered areas in the first stylized grid texture map that need to be filled. The present application proposes separate strategies for areas of different visibility. Specifically, in step 140, panoramic texture completion (Panoramic Texture Inpainting) can be performed on two credible unobstructed perspectives (areas without complex occlusions under the new perspective) of the first stylized grid texture map to obtain a completed image, and then the spatial texture prediction network is used to fill the remaining delicate geometry or complex occlusion areas in the completed image to obtain a second stylized grid texture map.

[0145] In some embodiments, step 140 may be implemented through steps 141 to 142, specifically:

[0146] Step 141 , performing panoramic perspective texture completion on the first stylized grid texture map to obtain a completed image.

[0147] In some embodiments, performing panoramic perspective texture completion on the first stylized grid texture map to obtain a completed image includes:

[0148] Performing dilation and blurring processing on the first stylized grid texture map to obtain a panorama completion mask;

[0149] Processing the prompt word, the first stylized grid texture map, the depth map, and the panoramic completion mask using an image completion model to generate an initial completed image;

[0150] Acquire a second depth edge filtering mask, a safety observation mask, and a second visibility mask, wherein the second depth edge filtering mask is used to filter areas where the depth in the initial completed image changes suddenly, the safety observation mask is used to filter areas where the incident viewing angle from the camera to the grid surface in the initial completed image is less than a preset angle and / or where the distance between the surface point of the corresponding grid surface in the initial completed image and the camera is greater than a preset distance, and the second visibility mask is used to filter visible pixel point areas in the initial completed image when observed from the position coordinates of the center of the camera at the completed position;

[0151] Determining a trusted mask according to an intersection of the depth edge filtering mask, the security observation mask, and the visibility mask;

[0152] The panoramic texture of the initial completed image is assigned to the two-dimensional texture of the first stylized grid texture map according to the credible mask to obtain a completed image, wherein the two-dimensional texture of the first stylized grid texture map is obtained by reusing the first visibility mask.

[0153] In step 141, the first stylized mesh texture map is subjected to panoramic view texture completion to obtain a completed image. In this step, a mesh with partial texture stylization filled in the first stylized mesh texture map is provided, and then the panoramic view texture completion is implemented in the panoramic space.

[0154] First, for each completed viewpoint, the panorama completion mask M is determined according to the new camera position inp Specifically, the first visibility mask M in the UV space can be init_vis It is reused as a UV texture and renders a panoramic image at the current viewpoint, then dilates and blurs the image to obtain a panoramic completion mask M. inp .

[0155] Then, use the current depth map D to control the image completion model F inp To generate the initial completion image As shown below:

[0156]

[0157] in, is a rendered panoramic image with a partially textured mesh; is the initial completed image, and is the image completion model F inp The completion result output after image completion; P is the prompt word; D is the depth map.

[0158] Then, when generating the initial completed image When completing the texture projection, we need to consider factors such as areas with sudden depth changes in the initial completed image, small incident angles (the incident angle from the camera to the mesh surface in the initial completed image is less than the preset angle), or surface points that are too far away (the distance between the surface points on the corresponding mesh surface in the initial completed image and the camera is greater than the preset distance), and the visibility of the camera angle at the completed position, so as to ensure reliable texture projection. Finally, the completed panoramic texture is assigned to the stylized UV texture to obtain the completed image, thereby further filling in part of the stylized scene.

[0159] Among them, the initial completion image It is not fully projected into the stylized UV texture, but only the credible areas are retained by filtering the visibility information in UV space. Optionally, three UV space masks can be designed for filtering to ensure credible texture projection. First, a second depth edge mask M can be used dep_edge Filtering the initial completion image In the area where the depth suddenly changes, the depth edge jump area can be detected from the depth map to generate the second depth edge mask M dep_edge . Second, we can filter the initial completion image Small and medium incident angles (the incident angle is less than the preset angle, such as the incident angle is less than 10°) or too far surface points (the distance between the surface point and the camera is greater than the preset distance, such as the distance between the surface point and the camera is greater than 2.5 meters) form a safety observation mask M safe_view Then, the second visibility mask M of the complement view can be constructed based on the visibility test of the camera view of the complement position. inp_vis , where the second visibility mask M can be constructed using the above formula (4): inp_vis This method combines all the above masks to realize the credible texture projection area in UV space, as shown in formula (6):

[0160] M conf =Mdep_edge ∩M safe_view ∩M inp_vis , (6)

[0161] Among them, M conf is the combined plausible mask representing the plausible texture projection area in UV space.

[0162] Since all masks are constructed in UV space rather than in a certain camera view or panoramic view, this design avoids the problem of view-based occlusion. Finally, using the trusted mask M conf The panoramic texture of the initial completed image is assigned to the UV texture of the stylized first stylized mesh texture map to obtain the completed image, and the partially stylized scene is further filled with more textures.

[0163] Step 142: fill the non-visible area in the completed image using a spatial texture prediction network to obtain a second stylized grid texture map, wherein the non-visible area includes at least one of the following: a thin geometric area and an occluded area.

[0164] In some embodiments, the using of a spatial texture prediction network to fill the non-visible area in the completed image to obtain a second stylized grid texture map includes:

[0165] Providing the spatial coordinates and real-world colors of all pixels in the two-dimensional texture image of the scene grid to the spatial texture prediction network to predict the target color;

[0166] Obtaining a third visibility mask by combining the first visibility mask and the second visibility mask;

[0167] The predicted target color is merged into the non-visible area in the completed image through the third visibility mask to obtain the second stylized grid texture map.

[0168] Among them, a spatial texture prediction network is used to fill in the remaining delicate geometries or areas with complex occlusion in the complement image to obtain a second stylized mesh texture map.

[0169] In the prediction stage, the spatial coordinates x and real-world colors C of all valid UV pixels are provided to the spatial texture prediction network. R To predict the color. After prediction, the accumulated visibility mask M accu The predicted colors are fused to the partially textured mesh, resulting in a fully stylized scene.

[0170] In order to further fill in the unobserved or undrawn areas in the scene-level mesh texture of the completed image, a spatial texture prediction network can be designed in step 142. The goal of texture prediction is to learn a style map from a partially stylized scene and then smoothly predict a reasonable texture for the invisible area.

[0171] First, the real-world color C R and the stylized color Cs is promoted from the corresponding UV texture to the spatial coordinate x.

[0172] In the training phase, a spatial texture prediction network F is learned Imit , the network inputs spatial coordinates x and real-world colors C from a partially textured scene R , and use the currently visible stylized color Cs to supervise the L2 loss function, as shown in the following formula (7):

[0173] Among them C S =F Imit (γ(x),C R ), (7)

[0174] Among them, γ(·) is based on NeRF position encoding technology, is the predicted color.

[0175] Then, in the prediction stage, the spatial texture prediction network is provided with the spatial coordinates x and real-world color C of all valid UV pixels. R To predict the target color

[0176] Finally, the third visibility mask M is accumulated accu The predicted target color The second stylized mesh texture map is fused into the partial texture grid of the completed image, and finally a fully stylized scene is generated.

[0177] Among them, the first visibility mask M can be combined init_vis and all second visibility masks M inp_vis Get the third visibility mask M accu .

[0178] See also Figure 4 The third application scenario schematic diagram shown in FIG. 1 shows, in step 142, a spatial texture prediction network is designed, such as Figure 4 In the texture prediction result diagram shown in (a), the reference image of the real scene provided has pillows and sheets. Before texture prediction, the black areas near the pillows and cabinets in the complementary image corresponding to the reference image are not covered with colors. After texture prediction, the original black areas in the image are covered with reasonable colors. Figure 4 In the schematic diagram of extracting color from UV texture shown in (b), the real world color C of the real UV texture is R The stylized color Cs of the stylized UV texture is mapped from the corresponding UV texture to the spatial coordinate x. Figure 4 In the schematic diagram of training on the visible stylized part shown in (c), during the training phase, the real world color C of the real scene is R Input the spatial texture prediction network, and use the currently visible stylized color Cs to supervise the L2 loss function to obtain a supervised stylized map. Figure 4 In the schematic diagram of the invisible part prediction shown in (d), in the prediction stage, the spatial coordinates x of all valid UV pixels and the real world color C of the real scene are provided to the spatial texture prediction network. R To predict the target color Output the predicted result map, in which the uncovered areas in the stylized scene of the completed image can be smoothly filled after being predicted by the spatial texture prediction network while maintaining good spatial consistency.

[0179] The embodiment of the present application can automatically and efficiently map textures to a three-dimensional scene grid by performing spatial texture propagation, and can handle various complex occlusions and delicate geometry situations. In addition, partial texture stylization of the scene can be achieved, thereby improving the effect and quality of texture mapping.

[0180] See also Figure 5 In the fourth application scenario shown in the figure, in the spatial texture propagation process of step 140, the texture of the stylized panoramic texture map is first projected to the visible area of ​​the scene grid to obtain a first stylized grid texture map, and then the spatial texture propagation is performed on the partially textured stylized scene based on the stylized UV texture. Among them, the panoramic view is completed for the credible unobstructed area. Specifically, for different completion viewpoints, the visibility of the completion view in the scene geometry is filtered based on the camera at different completion positions, and then the picture is completed to obtain the completed image. For the small occluded area, the spatial texture prediction is performed. Specifically, in the training stage, a spatial texture prediction network F is learned. Imit , the network inputs spatial coordinates x and real-world colors C from a partially textured scene R , and use the currently visible stylized color Cs to supervise the L2 loss function; then use the trained spatial texture prediction network to fill in the remaining slender geometry or complex occlusion areas in the complement image to obtain the second stylized mesh texture map.

[0181] In some embodiments, Figure 6 As shown, after step 140, the method may further include step 150:

[0182] Step 150: Pack the virtual window, the panoramic skybox, the second stylized grid texture map and the original scene grid to generate a target virtual scene, and display the target virtual scene.

[0183] See also Figure 7 As shown in the schematic diagram of the fifth application scenario, in step 150 of the embodiment of the present application, an immersive personalized application of the VR end based on the virtual window of the personalized scene and the panoramic sky box can be used to immersively browse the fully textured stylized target virtual scene on the head-mounted display device, which can increase the interactivity and immersion of the user with the virtual environment while ensuring the realism and visual effects, and significantly improve the VR end experience.

[0184] In some embodiments, step 150 may be implemented by following steps 151 to 154, specifically:

[0185] Step 151: Create a virtual environment space according to the second stylized mesh texture map and the original scene mesh.

[0186] For example, see Figure 8 The sixth application scenario schematic diagram shown in FIG. Figure 8 (a) shows a schematic diagram of the input material. For example, the stylized scene material (such as the stylized bedroom scene and transparent window) in the second stylized mesh texture map and the original scene mesh can be imported into the game engine to create a user-defined virtual environment space. It can also be operated according to the actual needs of the user to realize a personalized virtual environment space. The virtual environment space is uploaded to a head-mounted display (HMD) device for virtual tour.

[0187] The original scene mesh is a texture generated by an image acquisition device through acquisition of a real scene, and does not contain a real texture.

[0188] Step 152: Generate the virtual window in the virtual environment space.

[0189] For example, in order to further enhance the immersive experience, in the game engine, the baked UV image can be assigned a transparent alpha mask value according to the user-defined window creation requirements, and a transparent virtual window can be generated at the window position indicated in the window creation requirements. In the process of generating a virtual window, it is necessary to consider the window size, window shape and window position, as well as its visual effect in the scene. The alpha mask of the UV space is generated in a similar way to the previous completion mask. This can open a window in the virtual environment space, allowing the user to see the world outside the virtual environment space, enhancing the immersion of the virtual environment, and at the same time preventing the user from experiencing motion sickness in the immersive experience.

[0190] For example, Figure 8 The transparent window in the schematic diagram of the input material shown in (a) can be generated on a user-defined area after inputting a stylized bedroom scene.

[0191] Step 153, creating the panoramic sky box.

[0192] For example, a panoramic skybox can be generated through the skybox function in the game engine. The design of the panoramic skybox can be selected by the user, for example, it can be consistent with the scene style or form a sharp contrast. For example, the skybox function in the game engine can provide a variety of skybox materials, respond to the user's skybox material selection instruction, determine the target skybox, and render the target skybox into the virtual environment space to create a panoramic skybox. This can enhance the sense of space and immersion of the virtual environment.

[0193] For example, the target skybox is Figure 8 The skybox in the schematic diagram of the input material shown in (a) is a skybox with a starry sky theme. The target skybox is rendered into the virtual environment space to create a Figure 8 (b) The panoramic skybox shown.

[0194] Step 154: Pack and render the second stylized mesh texture map, the original scene mesh, the virtual window, and the panoramic sky box to generate the target virtual scene, and display the target virtual scene.

[0195] For example, in a game engine, the generated second stylized mesh texture map, the original scene mesh, the virtual window, and the panoramic skybox can be packaged and rendered to generate a target virtual scene. During the rendering process, the panoramic skybox is used as the background, and a virtual window with a transparent UV texture is opened at the same time, so that the panoramic skybox outside the virtual environment space can be seen through the transparent virtual window. The target virtual scene that is finally displayed will be a fully immersive virtual environment, allowing users to roam freely in a stylized space with a familiar scene structure but a completely different appearance.

[0196] like Figure 8 In the schematic diagram shown in (b), a VR device is used to immersively experience a stylized scene in a bedroom. In this stylized scene, the user can experience and observe the starry sky in the stylized bedroom scene.

[0197] The embodiments of the present application can convert real scenes into stylized scenes in a text-driven manner, and support 6DoF immersive roaming experience in VR devices.

[0198] All of the above technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described in detail here.

[0199] The embodiment of the present application obtains prompt words and scene grids, where the prompt words are text information provided by the user to represent the scene style, and the scene grid is a three-dimensional grid with real texture generated based on the reconstruction of the real scene; based on the prompt words, a stylized panoramic texture map is generated at the center of the scene grid; the texture of the stylized panoramic texture map is projected to the visible area of ​​the scene grid to obtain a first stylized grid texture map; the first stylized grid texture map is subjected to spatial texture propagation processing to fill the non-visible area of ​​the scene grid to obtain a second stylized grid texture map. The embodiment of the present application can automatically generate a stylized grid texture map based on the prompt words provided by the user, while ensuring spatial consistency, mapping correctness, and being able to completely cover complex geometric structures in real scenes.

[0200] In order to better implement the image processing method of the embodiment of the present application, the embodiment of the present application also provides an image processing device. Fig. 9 , Fig. 9 The following is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device 200 may include:

[0201] An acquisition unit 210 is used to acquire a prompt word and a scene mesh, wherein the prompt word is text information representing the scene style provided by the user, and the scene mesh is a three-dimensional mesh with real texture generated based on the reconstruction of the real scene;

[0202] A first processing unit 220, configured to generate a stylized panoramic texture map at a center position of the scene grid according to the prompt word;

[0203] A second processing unit 230 is used to project the texture of the stylized panoramic texture map to the visible area of ​​the scene grid to obtain a first stylized grid texture map;

[0204] The third processing unit 240 is configured to perform spatial texture propagation processing on the first stylized mesh texture map to fill the non-visible area of ​​the scene mesh to obtain a second stylized mesh texture map.

[0205] In some embodiments, the stylized panoramic texture map includes a style-priority panoramic map, and the first processing unit 220 can be used to: generate a style-priority panoramic map at the center of the scene grid according to the prompt word; and determine the stylized panoramic texture map according to the style-priority panoramic map.

[0206] In some embodiments, when the first processing unit 220 generates a style-priority panorama at the center of the scene grid according to the prompt word, it can be used to: render a true color image and a depth image of the real scene at the center of the scene grid; use a pre-trained diffusion model to process the prompt word, the true color image and the depth image to generate an initial style-priority panorama; based on super-resolution upscaling technology, enlarge the initial style-priority panorama by a preset multiple to obtain the style-priority panorama, wherein the resolution of the style-priority panorama is greater than the resolution of the initial style-priority panorama.

[0207] In some embodiments, the stylized panoramic texture map includes the style-priority panoramic map and the alignment-priority panoramic map, and the first processing unit 220 can also be used to: generate an alignment-priority panoramic map based on the prompt word, the true color map and the depth map; perform texture blending on the style-priority panoramic map and the alignment-priority panoramic map to generate the stylized panoramic texture map.

[0208] In some embodiments, when the first processing unit 220 generates an alignment-priority panorama according to the prompt word, the true color image and the style-priority panorama, it can be used to: input the prompt word, the true color image, the depth map and the style-priority panorama into the diffusion model for processing to generate an initial alignment-priority panorama; based on the super-resolution upscaling technology, enlarge the initial alignment-priority panorama by a preset multiple to obtain the alignment-priority panorama, wherein the resolution of the alignment-priority panorama is greater than the resolution of the initial alignment-priority panorama.

[0209] In some embodiments, when the first processing unit 220 performs texture blending on the style-priority panorama and the alignment-priority panorama to generate the stylized panoramic texture map, it can be used to: generate a first depth edge filtering mask according to the edge detection result at the jump point of the depth map, and the first depth edge filtering mask is used to characterize the pixel area of ​​the blended style-priority panorama and the alignment-priority panorama; according to the first depth edge filtering mask, use a Poisson image editing algorithm to blend the textures of the style-priority panorama and the alignment-priority panorama to obtain the stylized panoramic texture map.

[0210] In some embodiments, the second processing unit 230 can be used to: obtain the two-dimensional texture image of the scene grid and the texture coordinates corresponding to each pixel in the two-dimensional texture image; calculate the spatial coordinates of each pixel of the two-dimensional texture image in the three-dimensional space by interpolation according to the two-dimensional texture image and the texture coordinates; calculate the ray direction between each pixel of the two-dimensional texture image and the camera according to the spatial coordinates of each pixel of the two-dimensional texture image in the three-dimensional space, and map the ray direction to the stylized panoramic texture map through equirectangular projection; determine the visibility information of each pixel of the two-dimensional texture image under camera observation, so as to generate a first visibility mask according to the visibility information, and the visibility information is used to characterize whether the spatial coordinates of each pixel of the two-dimensional texture image are visible when observed from the position coordinates of the center of the camera; assign the panoramic color of the stylized panoramic texture map to the two-dimensional texture space according to the first visibility mask and the ray direction, so as to realize the projection of the texture of the stylized panoramic texture map to the visible area of ​​the scene grid, and obtain the first stylized grid texture map.

[0211] In some embodiments, the third processing unit 240 can be used to: perform panoramic perspective texture completion on the first stylized grid texture map to obtain a completed image; use a spatial texture prediction network to fill the non-visible area in the completed image to obtain a second stylized grid texture map, wherein the non-visible area includes at least one of the following: a slender geometric area and an occluded area.

[0212] In some embodiments, when the third processing unit 240 performs panoramic perspective texture completion on the first stylized grid texture map to obtain a completed image, it can be used to: perform expansion processing and blur processing on the first stylized grid texture map to obtain a panoramic completion mask; use the image completion model to process the prompt word, the first stylized grid texture map, the depth map and the panoramic completion mask to generate an initial completed image; obtain a second depth edge filtering mask, a safety observation mask and a second visibility mask, wherein the second depth edge filtering mask is used to filter the area with sudden depth changes in the initial completed image, and the safety observation mask is used to filter the area with sudden depth changes in the initial completed image. The second visibility mask is used to filter the visible pixel point area in the initial completed image when observed from the position coordinates of the center of the camera at the completed position; a credible mask is determined according to the intersection of the depth edge filtering mask, the security observation mask and the visibility mask; and the panoramic texture of the initial completed image is assigned to the two-dimensional texture of the first stylized grid texture map according to the credible mask to obtain a completed image, wherein the two-dimensional texture of the first stylized grid texture map is obtained by reusing the first visibility mask.

[0213] In some embodiments, when the third processing unit 240 uses the spatial texture prediction network to fill the non-visible area in the completed image to obtain the second stylized mesh texture map, it can be used to: provide the spatial texture prediction network with the spatial coordinates and real-world colors of all pixels in the two-dimensional texture image of the scene grid to predict the target color; obtain a third visibility mask by combining the first visibility mask and the second visibility mask; and fuse the predicted target color into the non-visible area in the completed image through the third visibility mask to obtain the second stylized mesh texture map.

[0214] In some embodiments, the image processing apparatus 200 further includes:

[0215] The fourth processing unit is used to package the virtual window, the panoramic sky box, the second stylized grid texture map and the original scene grid to generate a target virtual scene and display the target virtual scene.

[0216] In some embodiments, the fourth processing unit is used to: create a virtual environment space based on the second stylized mesh texture map and the original scene mesh; generate the virtual window in the virtual environment space; create the panoramic sky box; package and render the second stylized mesh texture map, the original scene mesh, the virtual window and the panoramic sky box to generate the target virtual scene, and display the target virtual scene.

[0217] Each unit in the above-mentioned image processing device 200 can be implemented in whole or in part by software, hardware or a combination thereof. Each unit can be embedded in or independent of a processor in a terminal device in the form of hardware, or can be stored in a memory in a terminal device in the form of software, so that the processor can call and execute the corresponding operations of each unit.

[0218] The image processing device 200 may be integrated in a terminal or a server that has a storage device and a processor and has computing capabilities, or the image processing device 200 may be the terminal or the server.

[0219] In some embodiments, the present application further provides a terminal device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0220] like Fig.10 As shown, Fig.10 This is a schematic diagram of the structure of the terminal device provided in the embodiment of the present application. The terminal device 300 can generally be provided in the form of glasses, a helmet-mounted display (Head Mount Display, HMD), or contact lenses to achieve visual perception and other forms of perception. Of course, the form of the terminal device is not limited thereto and can be further miniaturized or enlarged as needed. The terminal device 300 may include but is not limited to the following components:

[0221] Detection module 301: Use various sensors to detect the user's operation commands and act on the virtual environment, such as continuously updating the image displayed on the display screen following the user's line of sight, to achieve user interaction with the virtual and scene, for example, continuously updating the real content based on the detected rotation direction of the user's head.

[0222] Feedback module 302: receiving data from sensors and providing real-time feedback to users; wherein the feedback module 302 may be used to display a graphical user interface, such as displaying a virtual environment on the graphical user interface. For example, the feedback module 302 may include a display screen, etc.

[0223] Sensor 303: On the one hand, it receives operation commands from the user and applies them to the virtual environment; on the other hand, it provides the results of the operation to the user in the form of various feedback.

[0224] Control module 304: controls sensors and various input / output devices, including obtaining user data (such as movements, voice) and outputting perception data, such as images, vibrations, temperature, and sounds, etc., to affect users, virtual environments, and the real world.

[0225] Modeling module 305: constructs a three-dimensional model of the virtual environment, and may also include various feedback mechanisms such as sound and touch in the three-dimensional model.

[0226] In the embodiment of the present application, a virtual scene in a three-dimensional scene grid can be constructed by a modeling module 305; a prompt word and a scene grid are obtained by a detection module 301, the prompt word is text information representing the scene style provided by the user, and the scene grid is a three-dimensional grid with real texture generated based on the reconstruction of the real scene; a stylized panoramic texture map is generated at the center of the scene grid according to the prompt word by a control module 304; the texture of the stylized panoramic texture map is projected to the visible area of ​​the scene grid to obtain a first stylized grid texture map; the first stylized grid texture map is subjected to spatial texture propagation processing to fill the non-visible area of ​​the scene grid to obtain a second stylized grid texture map; a virtual window and a panoramic skybox can also be rendered to the second stylized grid texture map to generate a target virtual scene. A graphical user interface is displayed by a feedback module 302, and the graphical user interface may include a second stylized grid texture map and may also include a target virtual scene.

[0227] In some embodiments, Fig.11 As shown, Fig.11 Another structural diagram of a terminal device provided in an embodiment of the present application, the terminal device 300 also includes a processor 310 having one or more processing cores, a memory 320 having one or more computer-readable storage media, and a computer program stored in the memory 320 and executable on the processor. The processor 310 is electrically connected to the memory 320. Those skilled in the art will appreciate that the terminal device structure shown in the figure does not constitute a limitation on the terminal device, and may include more or fewer components than shown, or combine certain components, or arrange components differently.

[0228] The processor 310 is the control center of the terminal device 300. It uses various interfaces and lines to connect the various parts of the entire terminal device 300, executes various functions of the terminal device 300 and processes data by running or loading software programs and / or modules stored in the memory 320, and calling data stored in the memory 320, thereby monitoring the terminal device 300 as a whole.

[0229] In the embodiment of the present application, the processor 310 in the terminal device 300 loads instructions corresponding to the processes of one or more application programs into the memory 320 according to the following steps, and the processor 310 runs the application programs stored in the memory 320 to implement various functions:

[0230] Obtain a prompt word and a scene grid, wherein the prompt word is text information provided by a user to represent the scene style, and the scene grid is a three-dimensional grid with real texture generated based on real scene reconstruction; generate a stylized panoramic texture map at the center of the scene grid according to the prompt word; project the texture of the stylized panoramic texture map to the visible area of ​​the scene grid to obtain a first stylized grid texture map; perform spatial texture propagation processing on the first stylized grid texture map to fill the non-visible area of ​​the scene grid to obtain a second stylized grid texture map.

[0231] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0232] In some embodiments, the processor 310 may include a detection module 301 , a control module 304 , and a modeling module 305 .

[0233] In some embodiments, Fig.11 As shown, the terminal device 300 further includes: a radio frequency circuit 306, an audio circuit 307 and a power supply 308. The processor 310 is electrically connected to the memory 320, the feedback module 302, the sensor 303, the radio frequency circuit 306, the audio circuit 307 and the power supply 308 respectively. Those skilled in the art can understand that Fig.10 or Fig.11 The terminal device structure shown in the figure does not constitute a limitation on the terminal device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0234] The radio frequency circuit 306 may be used to send and receive radio frequency signals, so as to establish wireless communication with a network device or other terminal devices through wireless communication, and to send and receive signals between the network device or other terminal devices.

[0235] The audio circuit 307 can be used to provide an audio interface between the user and the terminal device through a speaker and a microphone. The audio circuit 307 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 307 and converted into audio data. The audio data is then processed by the output processor 301 and sent to, for example, another terminal device through the radio frequency circuit 306, or the audio data is output to a memory for further processing. The audio circuit 307 may also include an earplug jack to provide communication between an external headset and the terminal device.

[0236] The power supply 308 is used to supply power to various components of the terminal device 300 .

[0237] although Fig.10 or Fig.11 Not shown, the terminal device 300 may also include a camera, a wireless fidelity module, a Bluetooth module, an input module, etc., which will not be described in detail here.

[0238] In some embodiments, the present application further provides a computer-readable storage medium for storing a computer program. The computer-readable storage medium can be applied to a terminal device or a server, and the computer program enables the terminal device or the server to execute the corresponding process in the image processing method in the embodiment of the present application, which will not be described here for the sake of brevity.

[0239] In some embodiments, the present application further provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of the terminal device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the terminal device executes the corresponding process in the image processing method in the embodiment of the present application, which will not be described here for the sake of brevity.

[0240] The present application also provides a computer program, which includes a computer program, and the computer program is stored in a computer-readable storage medium. The processor of the terminal device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the terminal device executes the corresponding process in the image processing method in the embodiment of the present application, which will not be repeated here for the sake of brevity.

[0241] It should be understood that the processor of the embodiment of the present application may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and performed. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, and other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0242] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0243] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0244] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0245] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0246] In addition, each functional unit in the embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0247] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a terminal device (which can be a personal computer or a server) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0248] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Obtaining a prompt word and a scene mesh, wherein the prompt word is text information provided by a user to represent the scene style, and the scene mesh is a three-dimensional mesh with real texture generated based on real scene reconstruction; Generate a stylized panoramic texture map at the center of the scene grid according to the prompt word; Projecting the texture of the stylized panoramic texture map onto the visible area of ​​the scene grid to obtain a first stylized grid texture map; The first stylized mesh texture map is subjected to spatial texture propagation processing to fill the non-visible area of ​​the scene mesh, thereby obtaining a second stylized mesh texture map.

2. The image processing method according to claim 1, characterized in that: The stylized panoramic texture map includes a style-priority panoramic map, and generating the stylized panoramic texture map at the center of the scene grid according to the prompt word includes: Generate a style-priority panorama at the center of the scene grid according to the prompt word; The stylized panoramic texture map is determined according to the style-priority panoramic map.

3. The image processing method according to claim 2, characterized in that: Generating a style-priority panorama at the center of the scene grid according to the prompt word includes: Rendering a true color image and a depth image of the real scene at the center of the scene grid; Using a pre-trained diffusion model to process the prompt word, the true color image, and the depth image to generate an initial style-priority panoramic image; The initial style-priority panorama is magnified by a preset multiple based on a super-resolution magnification technology to obtain the style-priority panorama, wherein the resolution of the style-priority panorama is greater than the resolution of the initial style-priority panorama.

4. The image processing method according to claim 3, characterized in that: The stylized panoramic texture map includes the style-priority panoramic map and the alignment-priority panoramic map, and determining the stylized panoramic texture map according to the style-priority panoramic map includes: Generate an alignment-priority panoramic image according to the prompt word, the true color image and the depth image; The style-priority panorama and the alignment-priority panorama are texture-mixed to generate the stylized panorama texture map.

5. The image processing method according to claim 4, characterized in that: The step of generating an alignment-priority panoramic image according to the prompt word, the true color image and the style-priority panoramic image comprises: Inputting the prompt word, the true color image, the depth image and the style-priority panoramic image into the diffusion model for processing to generate an initial alignment-priority panoramic image; The initial alignment priority panoramic image is magnified by a preset multiple based on the super-resolution magnification technology to obtain the alignment priority panoramic image, wherein the resolution of the alignment priority panoramic image is greater than the resolution of the initial alignment priority panoramic image.

6. The image processing method according to claim 4, characterized in that: The step of mixing the style-priority panorama and the alignment-priority panorama to generate the stylized panorama texture map includes: Generating a first depth edge filtering mask according to an edge detection result at a transition point of the depth map, wherein the first depth edge filtering mask is used to characterize a pixel area of ​​a mixture of the style-priority panorama and the alignment-priority panorama; According to the first depth edge filtering mask, a Poisson image editing algorithm is used to mix the textures of the style-first panorama and the alignment-first panorama to obtain the stylized panorama texture map.

7. The image processing method according to any one of claims 3 to 6, characterized in that: The step of projecting the texture of the stylized panoramic texture map onto the visible area of ​​the scene mesh to obtain a first stylized mesh texture map comprises: Obtaining a two-dimensional texture image of the scene grid and texture coordinates corresponding to each pixel point in the two-dimensional texture image; Calculating the spatial coordinates of each pixel point of the two-dimensional texture image in the three-dimensional space by interpolation according to the two-dimensional texture image and the texture coordinates; According to the spatial coordinates of each pixel point of the two-dimensional texture image in the three-dimensional space, calculating the ray direction between each pixel point of the two-dimensional texture image and the camera, and mapping the ray direction to the stylized panoramic texture image through equirectangular projection; Determine visibility information of each pixel point of the two-dimensional texture image under camera observation, so as to generate a first visibility mask according to the visibility information, wherein the visibility information is used to indicate whether the spatial coordinates of each pixel point of the two-dimensional texture image are visible when observed from the position coordinates of the center of the camera; According to the first visibility mask and the ray direction, the panoramic color of the stylized panoramic texture map is assigned to the two-dimensional texture space to realize the projection of the texture of the stylized panoramic texture map to the visible area of ​​the scene mesh, thereby obtaining a first stylized mesh texture map.

8. The image processing method according to claim 7, characterized in that: The performing spatial texture propagation processing on the first stylized mesh texture map to fill the non-visible area of ​​the scene mesh to obtain a second stylized mesh texture map includes: Performing panoramic perspective texture completion on the first stylized grid texture map to obtain a completed image; A spatial texture prediction network is used to fill in non-visible areas in the completed image to obtain a second stylized grid texture map, wherein the non-visible areas include at least one of the following: a slender geometric area and an occluded area.

9. The image processing method according to claim 8, characterized in that: The performing panoramic perspective texture completion on the first stylized grid texture map to obtain a completed image includes: Performing dilation and blurring processing on the first stylized grid texture map to obtain a panorama completion mask; Processing the prompt word, the first stylized grid texture map, the depth map, and the panoramic completion mask using an image completion model to generate an initial completed image; Acquire a second depth edge filtering mask, a safety observation mask, and a second visibility mask, wherein the second depth edge filtering mask is used to filter areas where the depth in the initial completed image changes suddenly, the safety observation mask is used to filter areas where the incident viewing angle from the camera to the grid surface in the initial completed image is less than a preset angle and / or where the distance between the surface point of the corresponding grid surface in the initial completed image and the camera is greater than a preset distance, and the second visibility mask is used to filter visible pixel point areas in the initial completed image when observed from the position coordinates of the center of the camera at the completed position; Determining a trusted mask according to an intersection of the depth edge filtering mask, the security observation mask, and the visibility mask; The panoramic texture of the initial completed image is assigned to the two-dimensional texture of the first stylized grid texture map according to the credible mask to obtain a completed image, wherein the two-dimensional texture of the first stylized grid texture map is obtained by reusing the first visibility mask.

10. The image processing method according to claim 9, characterized in that: The method of using a spatial texture prediction network to fill the non-visible area in the completed image to obtain a second stylized grid texture map includes: Providing the spatial coordinates and real-world colors of all pixels in the two-dimensional texture image of the scene grid to the spatial texture prediction network to predict the target color; Obtaining a third visibility mask by combining the first visibility mask and the second visibility mask; The predicted target color is merged into the non-visible area in the completed image through the third visibility mask to obtain the second stylized grid texture map.

11. The image processing method according to claim 1, wherein: The method further comprises: The virtual window, the panoramic skybox, the second stylized grid texture map and the original scene grid are packaged to generate a target virtual scene, and the target virtual scene is displayed.

12. The image processing method according to claim 11, characterized in that: The step of packaging the virtual window, the panoramic skybox, the second stylized grid texture map and the original scene grid to generate a target virtual scene and displaying the target virtual scene includes: Creating a virtual environment space according to the second stylized mesh texture map and the original scene mesh; generating the virtual window in the virtual environment space; Creating the panoramic skybox; The second stylized mesh texture map, the original scene mesh, the virtual window, and the panoramic sky box are packaged and rendered to generate the target virtual scene, and the target virtual scene is displayed.

13. An image processing device, characterized in that: The device comprises: An acquisition unit, used to acquire a prompt word and a scene mesh, wherein the prompt word is text information representing the scene style provided by a user, and the scene mesh is a three-dimensional mesh with real texture generated based on the reconstruction of a real scene; A first processing unit, configured to generate a stylized panoramic texture map at a center position of the scene grid according to the prompt word; A second processing unit, configured to project the texture of the stylized panoramic texture map to a visible area of ​​the scene grid to obtain a first stylized grid texture map; The third processing unit is configured to perform spatial texture propagation processing on the first stylized mesh texture map to fill the non-visible area of ​​the scene mesh to obtain a second stylized mesh texture map.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the image processing method according to any one of claims 1 to 12.

15. A terminal device, characterized in that: The terminal device includes a processor and a memory, wherein a computer program is stored in the memory, and the processor is used to execute the image processing method according to any one of claims 1 to 12 by calling the computer program stored in the memory.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 12 is implemented.

Images