Method and device for mapping figure in image based on depth-of-field model

Abstract

The invention relates to a method and device for mapping a figure in an image based on a depth-of-field model, and the method comprises the steps: inputting an original image containing the figure into the depth-of-field model, and obtaining a figure depth map of the original image; according to the figure depth map, constructing a figure 3D model, and obtaining three-dimensional coordinates of all vertexes in the figure 3D model; carrying out UV expansion on the figure 3D model to obtain UV coordinates of all vertexes mapped to a two-dimensional plane in the figure 3D model; loading the 2D map texture on the two-dimensional plane to obtain a pixel color value of a normalized coordinate of the 2D map texture; according to the pixel color value of the normalized coordinate of the original image, the pixel color value of the normalized coordinate of the 2D chartlet texture and a preset color mixing mode, calculating and outputting the pixel color value of the normalized coordinate of the original image after chartlet; wherein the normalized coordinates of the original image and the normalized coordinates of the 2D mapping texture conform to the mapping relation when all vertexes in the 3D figure model are mapped to a two-dimensional plane.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for mapping a person in an image based on a depth of field model. Background Art

[0002] There are currently two methods for mapping characters in images:

[0003] 1. Two-dimensional planar method, which usually applies stickers or patterns to the surface of the character without considering the three-dimensional characteristics of the character's shape. This means that no matter how the character's posture and angle change, the sticker effect cannot adapt to these changes in real time, which results in a significant visual deviation between the sticker and the actual outline and shape of the character. The overall effect appears flat and lacks three-dimensionality, which is difficult to meet users' expectations for the visual effect of pattern fitting.

[0004] 2. Three-dimensional mapping method, which requires users to rely on careful manual editing in order to achieve the ideal mapping effect. This process requires users to manually identify and analyze the three-dimensional structure of the character, observe the shape, posture and angle of the character, so as to reasonably apply stickers to the surface of the character. Specifically, users need to manually adjust the vertex position of the sticker grid, stretch, scale or rotate the stickers so that the stickers can better fit the curves of the character and present a three-dimensional sense visually. This manual adjustment method is not only complicated to operate, but also requires users to have certain three-dimensional modeling knowledge and artistic aesthetic ability to make appropriate judgments and decisions during the adjustment process. This makes it difficult for many ordinary users to face this technology, thus limiting the widespread application and popularization of three-dimensional mapping methods in the mass market. Summary of the invention

[0005] In order to at least to some extent overcome the problem that the three-dimensional mapping method in the related art relies on the user's sophisticated manual editing and cannot be widely used and popularized, the present application provides a method and device for mapping characters in an image based on a depth of field model.

[0006] The scheme of this application is as follows:

[0007] According to a first aspect of an embodiment of the present application, a method for mapping a person in an image based on a depth of field model is provided, comprising:

[0008] Inputting an original image containing a person into a depth model to obtain a depth map of the person in the original image;

[0009] Constructing a 3D model of the character according to the character depth map, and obtaining the three-dimensional coordinates of all vertices in the 3D model of the character;

[0010] Perform UV unfolding on the character 3D model to obtain UV coordinates of all vertices in the character 3D model mapped to a two-dimensional plane;

[0011] Load the 2D map texture on the two-dimensional plane and obtain the pixel color value of the normalized coordinates of the 2D map texture;

[0012] According to the pixel color value of the normalized coordinates of the original image and the pixel color value of the normalized coordinates of the 2D map texture, and a preset color mixing method, the pixel color value of the normalized coordinates of the original image after mapping is calculated and output;

[0013] The normalized coordinates of the original image and the normalized coordinates of the 2D map texture conform to the mapping relationship when all vertices in the character 3D model are mapped to a two-dimensional plane.

[0014] Preferably, constructing a 3D model of the character according to the character depth map, and obtaining the three-dimensional coordinates of all vertices in the 3D model of the character includes:

[0015] Define the size of the character depth map as M×N, and the maximum height of the character 3D model as H max ;

[0016] Construct a three-dimensional grid containing m×N vertices; the three-dimensional coordinates of any vertex V(i,j) in the three-dimensional grid are expressed as:

[0017]

[0018] Where i∈{0,1,...,m-1} and j∈{0,1,...,N-1}; Depth(i,j) represents the depth value of the pixel in the i-th column and j-th row of the character depth map.

[0019] Preferably, constructing a 3D model of a character further includes:

[0020] Constructing the triangular faces of the character 3D model.

[0021] Preferably, constructing the triangular surface of the character 3D model includes:

[0022] Consider four adjacent vertices in the upper left, upper right, lower left and lower right of the three-dimensional grid as a group;

[0023] Cut the quadrilateral surface composed of a group of vertices from the upper left vertex to the lower right vertex to obtain a triangular surface composed of the upper left, lower left and lower right vertices and a triangular surface composed of the upper left, lower right and upper right vertices.

[0024] Preferably, after loading the 2D map texture on the two-dimensional plane, the method further comprises:

[0025] Generates a mipmap texture for a 2D map.

[0026] Preferably, the output pixel color value of any normalized coordinate (i, j) after the original image is mapped is expressed as:

[0027] OutputColor i,j =blend(InputColor i,j ,Tex u,v );

[0028] Among them, OutputColor i,j Represents the output pixel color value of any normalized coordinate (i, j) in the original image; blend represents the preset color mixing method; InputColor i,j Represents the pixel color value of any normalized coordinate (i, j) in the original image; Tex u,v Represents the pixel color value at the normalized coordinate (u,v) of a 2D map texture.

[0029] Preferably, the method further comprises:

[0030] Inputting the original image into the portrait segmentation model to obtain a portrait mask image of the original image;

[0031] According to the pixel color values ​​of the normalized coordinates of the original image, the pixel color values ​​of the normalized coordinates of the 2D map texture and the pixel color values ​​of the normalized coordinates of the portrait mask map, and the preset color mixing method, the pixel color values ​​of the normalized coordinates of the original image combined with the portrait mask map for mapping are calculated and output.

[0032] Preferably, the output pixel color value of any normalized coordinate (i, j) of the original image combined with the portrait mask image for mapping is expressed as:

[0033] OutputColor i,j =blend(InputColor i,j ,Tex u,v InBody i,j );

[0034] Among them, InBody i,j InBody represents the pixel color value of the normalized coordinate (i, j) of the portrait mask image. i,j ∈[0,1], a value of 0 means it is not in the portrait area, and a value of 1 means it is in the portrait area.

[0035] According to a second aspect of an embodiment of the present application, a device for mapping a person in an image based on a depth of field model is provided, comprising:

[0036] Processor and memory;

[0037] The processor and the memory are connected via a communication bus:

[0038] Wherein, the processor is used to call and execute the program stored in the memory;

[0039] The memory is used to store a program, and the program is used to execute at least one of the methods described above for mapping a person in an image based on a depth of field model.

[0040] The technical solution provided by this application may have the following beneficial effects:

[0041] The method for mapping a person in an image based on a depth of field model in the present application includes: inputting an original image containing a person into a depth of field model to obtain a depth map of the person in the original image; constructing a 3D model of the person according to the depth map of the person, and obtaining three-dimensional coordinates of all vertices in the 3D model of the person; UV unfolding the 3D model of the person to obtain UV coordinates of all vertices in the 3D model of the person mapped to a two-dimensional plane; loading a 2D mapping texture on the two-dimensional plane to obtain pixel color values ​​of normalized coordinates of the 2D mapping texture; calculating and outputting pixel color values ​​of normalized coordinates of the original image after mapping according to the pixel color values ​​of the normalized coordinates of the original image and the pixel color values ​​of the normalized coordinates of the 2D mapping texture, as well as a preset color mixing method; wherein the normalized coordinates of the original image and the normalized coordinates of the 2D mapping texture conform to the mapping relationship when all vertices in the 3D model of the person are mapped to a two-dimensional plane.

[0042] This technical solution combines the depth of field model and automatically constructs a 3D model of the character based on the character depth map output by the depth of field model. When mapping, the user only needs to simply move the position of the 2D map, and the system will adjust the deformation of the 2D map texture in real time according to the character's 3D model information to ensure that the 2D map texture can accurately fit the character's posture and angle to achieve a refined 3D mapping effect. The automated processing of this process eliminates the tedious steps that users previously needed to manually adjust sticker meshes, vertices, or perform complex stretching, compression, and rotation operations, greatly reducing the complexity of the operation and the user's learning cost. This technical solution enhances the realism and three-dimensionality of the mapping effect, and also provides users with a simpler operating experience.

[0043] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] Figure 1 It is a flowchart of a method for mapping a person in an image based on a depth of field model provided by an embodiment of the present application;

[0046] Figure 2 A character depth map obtained according to an original image provided by an embodiment of the present application;

[0047] Figure 3 is a front view of a 3D model of a character constructed according to a depth map of the character provided by an embodiment of the present application;

[0048] Figure 4 is a side view of a 3D model of a character constructed according to a depth map of the character provided by an embodiment of the present application;

[0049] Figure 5 It is a front view of a 3D character model after UV unfolding provided by an embodiment of the present application;

[0050] Figure 6 It is a side view of a 3D character model after UV unfolding provided by an embodiment of the present application;

[0051] Figure 7 It is a structural schematic diagram of a device for mapping characters in an image based on a depth of field model provided by an embodiment of the present application.

[0052] Reference numerals: processor-21; memory-22. DETAILED DESCRIPTION

[0053] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0054] Embodiment 1

[0055] Figure 1 is a flow chart of a method for mapping a person in an image based on a depth of field model provided by an embodiment of the present application, with reference to Figure 1 , a method for mapping a person in an image based on a depth of field model, comprising:

[0056] S11: inputting the original image containing the person into the depth model to obtain a depth map of the person in the original image;

[0057] The depth map of the original image is obtained by using the depth model. Figure 2As shown, the depth of field model is an existing mature technology and will not be described in detail here. Figure 2 In the image, the depth value of each pixel ranges from 0 to 1. The larger the value, the greater the distance from the camera position, that is, 0 represents a close view and 1 represents a distant view. Usually, the characters in the image are close views, and the background content is a distant view.

[0058] S12: constructing a 3D model of the character according to the character depth map, and obtaining the three-dimensional coordinates of all vertices in the 3D model of the character;

[0059] Specifically, the size of the character depth map is defined as M×N, and the maximum height of the character 3D model is H max ;

[0060] Construct a three-dimensional grid containing M×N vertices; the three-dimensional coordinates of any vertex V(i,j) in the three-dimensional grid are expressed as:

[0061]

[0062] Where i∈{0,1,...,M-1} and j∈{0,1,...,N-1}; Depth(i,j) represents the depth value of the pixel in the i-th column and j-th row of the character depth map.

[0063] In this way, the three-dimensional coordinates of all vertices of the character's 3D model can be obtained, where the x and y coordinates correspond to the normalized coordinates of the pixels in the original image, and the z coordinate represents the height of the corresponding pixel in the three-dimensional space.

[0064] Furthermore, the triangular surface of the character 3D model is constructed, including:

[0065] The four adjacent vertices at the upper left, upper right, lower left, and lower right in the three-dimensional grid are considered as a group;

[0066] Cut the quadrilateral surface composed of a group of vertices from the upper left vertex to the lower right vertex to obtain a triangular surface composed of the upper left, lower left and lower right vertices and a triangular surface composed of the upper left, lower right and upper right vertices.

[0067] For example:

[0068] For each group of four adjacent vertices, tl upper left, tr upper right, bl lower left, and br lower right, cut them into two counterclockwise triangular faces, T(tl,bl,br) and T(tl,br,tr).

[0069] The front view of the character 3D model constructed based on the character depth map is as follows Figure 3 As shown, the side view of the character 3D model constructed according to the character depth map is as follows Figure 4 shown.

[0070] S13: UV unfolding the character 3D model to obtain UV coordinates of all vertices in the character 3D model mapped to a two-dimensional plane;

[0071] UV unwrapping is the process of mapping the mesh surface of a 3D model to a 2D plane. In this process, any vertex P of the 3D mesh i There exists a point on a two-dimensional plane One-to-one mapping, and the grid on the two-dimensional plane is topologically isomorphic to the original three-dimensional grid. Through UV unfolding, the corresponding points between the two-dimensional image and the surface of the three-dimensional object can be found, thereby realizing texture mapping. There are many methods to realize UV ​​unfolding, and the more commonly used ones are: barycentric mapping or Tutte embedding; angle based flattening (ABF); least squares conformal maps (LSCM); most isometric parameterization of surfaces (MIPS), etc. Preferably, the barycentric mapping method is used in the present technical solution to realize UV ​​unfolding. The core principle is to use barycentric coordinates to optimize the UV mapping process, which can effectively maintain the topological relationship between surfaces and reduce texture stretching and distortion.

[0072] The front view of the character 3D model after UV unfolding is as follows Figure 5 As shown, the side view of the character 3D model after UV unfolding is as follows Figure 6 shown.

[0073] S14: loading a 2D map texture on a two-dimensional plane, and obtaining a pixel color value of a normalized coordinate of the 2D map texture;

[0074] In practice, the OpenGL image rendering API is used to texture the 3D model. OpenGL is an open graphics library, a cross-language, cross-platform application programming interface for rendering 2D and 3D vector graphics. This interface consists of nearly 350 different function calls, which are used to draw simple graphics to more complex three-dimensional scenes. Another programming interface system is Direct3D, which is only used on Microsoft Windows. OpenGL is often used in CAD, virtual reality, scientific visualization programs and video game development.

[0075] Preferably, after the 2D map texture is loaded on the two-dimensional plane, a mipmap texture of the 2D map is also generated. The mipmap texture can improve the jagged feeling of the map.

[0076] S15: Calculate and output the pixel color value of the normalized coordinate of the original image after mapping according to the pixel color value of the normalized coordinate of the original image and the pixel color value of the normalized coordinate of the 2D map texture, as well as a preset color mixing method;

[0077] In practice, the mesh vertices and triangles of the 3D model, as well as the UV coordinates of the mesh vertices mapped to the two-dimensional plane, can be input into OpenGL. Then, based on the pixel color values ​​of the normalized coordinates of the original image and the pixel color values ​​of the normalized coordinates of the 2D map texture, as well as the preset color mixing method, the pixel color values ​​of the normalized coordinates of the original image after mapping are calculated and output:

[0078] OutputColor i,j =blend(InputColor i,j ,Tex u,v );

[0079] Among them, OutputColor i,j Represents the output pixel color value of any normalized coordinate (i, j) in the original image; blend represents the preset color mixing method; InputColor i,j Represents the pixel color value of any normalized coordinate (i, j) in the original image; Tex u,v Represents the pixel color value at the normalized coordinate (u,v) of a 2D map texture.

[0080] It should be noted that if you want to perform texture mapping only on the person, you can combine the portrait segmentation model to infer the portrait mask map and then perform mapping.

[0081] Specifically, the method further includes:

[0082] Input the original image into the portrait segmentation model to obtain the portrait mask image of the original image;

[0083] According to the pixel color values ​​of the normalized coordinates of the original image, the pixel color values ​​of the normalized coordinates of the 2D map texture and the pixel color values ​​of the normalized coordinates of the portrait mask map, and the preset color mixing method, the pixel color values ​​of the normalized coordinates of the original image combined with the portrait mask map for mapping are calculated and output.

[0084] The output pixel color value of any normalized coordinate (i, j) of the original image combined with the portrait mask map is expressed as:

[0085] OutputColor i,j =blend(InputColor i,j ,Tex u,v InBody i,j );

[0086] Among them, InBody i,j InBody represents the pixel color value of the normalized coordinate (i, j) of the portrait mask image. i,j ∈[0,1], a value of 0 means it is not in the portrait area, and a value of 1 means it is in the portrait area.

[0087] This technical solution combines the depth of field model and automatically constructs a 3D model of the character based on the character depth map output by the depth of field model. When mapping, the user only needs to simply move the position of the 2D map, and the system will adjust the deformation of the 2D map texture in real time according to the character's 3D model information to ensure that the 2D map texture can accurately fit the character's posture and angle to achieve a refined 3D mapping effect. The automated processing of this process eliminates the tedious steps that users previously needed to manually adjust sticker meshes, vertices, or perform complex stretching, compression, and rotation operations, greatly reducing the complexity of the operation and the user's learning cost. This technical solution enhances the realism and three-dimensionality of the mapping effect, and also provides users with a simpler operating experience.

[0088] Embodiment 2

[0089] A device for mapping characters in an image based on a depth of field model, referring to Figure 7 ,include:

[0090] Processor 21 and memory 22;

[0091] The processor 21 and the memory 22 are connected via a communication bus:

[0092] The processor 21 is used to call and execute the program stored in the memory 22;

[0093] The memory 22 is used to store a program, and the program is used to execute at least a method for mapping a person in an image based on a depth of field model as described in the above embodiment.

[0094] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0095] It should be noted that, in the description of this application, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0096] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0097] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0098] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0099] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0100] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0101] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0102] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for mapping a person in an image based on a depth of field model, characterized in that: include: Inputting an original image containing a person into a depth model to obtain a depth map of the person in the original image; Constructing a 3D model of the character according to the character depth map, and obtaining the three-dimensional coordinates of all vertices in the 3D model of the character; Perform UV unfolding on the character 3D model to obtain UV coordinates of all vertices in the character 3D model mapped to a two-dimensional plane; Load the 2D map texture on the two-dimensional plane and obtain the pixel color value of the normalized coordinates of the 2D map texture; According to the pixel color value of the normalized coordinates of the original image and the pixel color value of the normalized coordinates of the 2D map texture, and a preset color mixing method, the pixel color value of the normalized coordinates of the original image after mapping is calculated and output; The normalized coordinates of the original image and the normalized coordinates of the 2D map texture conform to the mapping relationship when all vertices in the character 3D model are mapped to a two-dimensional plane.

2. The method according to claim 1, characterized in that According to the character depth map, a character 3D model is constructed to obtain the three-dimensional coordinates of all vertices in the character 3D model, including: Define the size of the character depth map as M×N, and the maximum height of the character 3D model as H max ; Construct a three-dimensional grid containing M×N vertices; the three-dimensional coordinates of any vertex V(i,j) in the three-dimensional grid are expressed as: Where i∈{0,1,...,M-1} and j∈{0,1,...,N-1}; Depth(i,j) represents the depth value of the pixel in the i-th column and j-th row of the character depth map.

3. The method according to claim 1, characterized in that Building a 3D character model also includes: Constructing the triangular faces of the character 3D model.

4. The method according to claim 2, characterized in that: Constructing the triangular surface of the character 3D model includes: Consider four adjacent vertices in the upper left, upper right, lower left and lower right of the three-dimensional grid as a group; Cut the quadrilateral surface composed of a group of vertices from the upper left vertex to the lower right vertex to obtain a triangular surface composed of the upper left, lower left and lower right vertices and a triangular surface composed of the upper left, lower right and upper right vertices.

5. The method according to claim 1, characterized in that After the 2D map texture is loaded on the two-dimensional plane, the method further includes: Generates a mipmap texture for a 2D map.

6. The method according to claim 1, characterized in that The output pixel color value of any normalized coordinate (i, j) after the original image is mapped is expressed as: OutputColor i,j =blend(InputColor u,j ,Tex u,v ); Among them, OutputColori i,j Represents the output pixel color value of any normalized coordinate (i, j) in the original image; blend represents the preset color mixing method; InputColor i,j Represents the pixel color value of any normalized coordinate (i, j) in the original image; Tex u,v Represents the pixel color value at the normalized coordinate (u,v) of a 2D map texture.

7. The method according to claim 1, characterized in that The method further comprises: Inputting the original image into the portrait segmentation model to obtain a portrait mask image of the original image; According to the pixel color values ​​of the normalized coordinates of the original image, the pixel color values ​​of the normalized coordinates of the 2D map texture and the pixel color values ​​of the normalized coordinates of the portrait mask map, and the preset color mixing method, the pixel color values ​​of the normalized coordinates of the original image combined with the portrait mask map for mapping are calculated and output.

8. The method according to claim 7, characterized in that The output pixel color value of any normalized coordinate (i, j) of the original image combined with the portrait mask map is expressed as: OutputColor i,j =blend(InputColor i,j ,Tex u,v ·InBody i,j ); Among them, InBody i,j InBody represents the pixel color value of the normalized coordinate (i, j) of the portrait mask image. i,j ∈[0,1], a value of 0 means it is not in the portrait area, and a value of 1 means it is in the portrait area.

9. A device for mapping a person in an image based on a depth of field model, characterized in that: include: Processor and memory; The processor and the memory are connected via a communication bus: Wherein, the processor is used to call and execute the program stored in the memory; The memory is used to store a program, and the program is used to execute at least a method for mapping a person in an image based on a depth of field model as described in any one of claims 1 to 8.

Images