Texture image mapping method, device, electronic device and readable storage medium

Abstract

The present application discloses a texture image mapping method, a texture image mapping device, an electronic device and a readable storage medium, and belongs to the field of image data processing technology. The texture image mapping method provided by the present application includes: obtaining multiple texture images; determining the first semantics of a first face patch and a second face patch, the second face patch being adjacent to the first face patch in a three-dimensional model; determining a target texture image from the multiple texture images according to the depth information of the first face patch and the multiple texture images, and the first semantics of the first face patch and the second face patch; and mapping the first face patch using the target texture image.

Description

Technical Field

[0001] The present application belongs to the technical field of image data processing, and specifically relates to a texture image mapping method, a texture image mapping device, an electronic device and a readable storage medium. Background Art

[0002] In the prior art, the internal environment of a scene can be viewed through the three-dimensional model of the scene. Texture mapping is the most intuitive display method in the three-dimensional model of the scene. However, due to problems such as inaccurate mapping posture and calibration errors, it is difficult to obtain high-quality texture mapping. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a texture image mapping method, a texture image mapping device, an electronic device and a readable storage medium, which realizes the use of depth information to judge the accuracy of texture mapping, and uses semantic information to ensure the smoothness of texture mapping selection, thereby effectively ensuring the validity and integrity of texture mapping selection.

[0004] In a first aspect, an embodiment of the present application provides a method for mapping a texture image, the method for mapping a texture image comprising: acquiring multiple texture images; determining first semantics of a first face and a second face, the second face being adjacent to the first face in a three-dimensional model; determining a target texture image from the multiple texture images based on depth information of the first face and the multiple texture images, and the first semantics of the first face and the second face; and using the target texture image to map the first face.

[0005] In a second aspect, an embodiment of the present application provides a texture image mapping device, the texture image mapping device comprising: an acquisition unit, for acquiring multiple texture images; a processing unit, for determining first semantics of a first face patch and a second face patch, the second face patch being adjacent to the first face patch in a three-dimensional model; the processing unit, further for determining a target texture image from the multiple texture images based on depth information of the first face patch and the multiple texture images, and the first semantics of the first face patch and the second face patch; the processing unit, further for using the target texture image to perform mapping processing on the first face patch.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a program or instruction stored in the memory and executable on the processor, which, when executed by the processor, implements the steps of the texture image mapping method of the first aspect.

[0007] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the texture image mapping method of the first aspect are implemented.

[0008] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the texture image mapping method of the first aspect.

[0009] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the texture image mapping method of the first aspect.

[0010] In the present application, in the process of texture mapping of a three-dimensional model of a target scene, among multiple texture images, a target texture image is determined by the semantics of the facets in the three-dimensional model and the depth information of the texture image, and the target texture image is used for mapping in the three-dimensional model. The texture image mapping method provided by the present application realizes the accuracy of judging the position and posture of the texture image by using the depth information of the facets and the texture image, and the smoothness of the texture image texture mapping selection is ensured by using the semantic information of the facets, thereby ensuring the validity and integrity of the texture image texture mapping, and improving the display effect of the three-dimensional model of the target scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 One of the flow diagrams of the texture image mapping method provided in the embodiment of the present application is shown;

[0012] Figure 2 The second flowchart of the texture image mapping method provided in the embodiment of the present application is shown;

[0013] Figure 3 The third flowchart of the texture image mapping method provided in the embodiment of the present application is shown;

[0014] Figure 4 A fourth flow chart of a texture image mapping method provided in an embodiment of the present application is shown;

[0015] Figure 5 A structural block diagram of a texture image mapping device provided in an embodiment of the present application is shown;

[0016] Figure 6 A structural block diagram of an electronic device provided in an embodiment of the present application is shown;

[0017] Figure 7 The fifth flowchart of the texture image mapping method provided in the embodiment of the present application is shown;

[0018] Figure 8 A schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0020] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0021] For ease of understanding, some technical terms involved in this application are briefly introduced below:

[0022] Three-dimensional model: A three-dimensional model is a polygonal representation of an object, usually displayed by a computer or other video device. The displayed object can be a real-world entity or a fictional object. Anything that exists in the physical nature can be represented by a three-dimensional model. In the embodiment of the present application, the three-dimensional model of the object is used to indicate the three-dimensional structure and size information of the target scene (such as a house). There are many forms of data storage for three-dimensional models, such as representation in the form of three-dimensional point clouds, grids, or voxels, which are not specifically limited here.

[0023] Facets: Facets are the smallest plane units in a 3D mesh model. Usually in rendering, the model in space needs to be divided into countless tiny planes. These planes are also called patches, which can be any polygons, and triangles and quadrilaterals are commonly used. The intersection points of the edges of these patches are the vertices of each patch. Patches can be randomly divided according to information such as the material or color of the model.

[0024] The following is combined with Figures 1 to 8 , the texture image mapping method, texture image mapping device, electronic device and readable storage medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0025] In an embodiment of the present application, a method for mapping a texture image is provided. Figure 1 FIG. 1 shows one of the flow diagrams of the texture image mapping method provided in the embodiment of the present application, such as Figure 1 As shown, the mapping method of the texture image includes:

[0026] Step 102: Acquire multiple texture images.

[0027] It should be noted that the above-mentioned multiple texture images may be multiple texture images captured by the depth camera in the target scene.

[0028] Furthermore, the texture image can be used to map the faces in the three-dimensional model that require texture mapping.

[0029] Exemplarily, the target scene may be a house, and a camera is used to collect images inside the house to obtain a plurality of texture images of the interior of the house. These texture images can be used to map the three-dimensional model of the house.

[0030] Step 104: Determine the first semantics of the first face patch and the second face patch;

[0031] The first face patch and the second face patch are face patches in the three-dimensional model, and the second face patch is adjacent to the first face patch.

[0032] Specifically, a patch is a plane component unit of a three-dimensional model, and a plurality of patches are included in the three-dimensional model. The first patch and the second patch may be patches in the three-dimensional model, the first patch and the second patch have an adjacent positional relationship, the first patch may be a target patch, the second patch may be a patch adjacent to the target patch, and the first semantics may be the three-dimensional semantics of the patch in the three-dimensional model.

[0033] For example, in the three-dimensional model of a house, the three-dimensional semantics of all faces are confirmed. The face with the three-dimensional semantics of "table" can be set as the first face, and the face adjacent to the first face can be set as the second face. The three-dimensional semantics of the second face can also be "table".

[0034] Step 106: Determine a target texture image from the multiple texture images according to the depth information of the first patch and the multiple texture images, and the first semantics of the first patch and the second patch.

[0035] Specifically, the depth information may include distance information from a first face in the three-dimensional model to a texture image, and a depth value of a depth map corresponding to any one of the multiple texture images.

[0036] Furthermore, according to the depth information of the first patch mapped to the texture image and the depth value of the depth map corresponding to the texture image, combined with the "table" semantics corresponding to the first patch in the above example, a target texture image is selected from multiple texture images and the target texture image is used as a map.

[0037] Step 108: Use the target texture image to perform mapping on the first face.

[0038] In the embodiment of the present application, a plurality of texture images are first obtained, and the first semantics of a first face patch and a second face patch adjacent thereto in the three-dimensional model are determined. Through the obtained plurality of texture images, a target texture image corresponding to the first face patch is selected from the plurality of texture images according to the depth information of the first face patch and the plurality of texture images and the first semantics of the first face patch and the second face patch, and then the first face patch is mapped using the target texture image, thereby completing the texture mapping process of the three-dimensional model.

[0039] Specifically, in the process of texture mapping of the three-dimensional model of the target scene, multiple texture images captured by the device camera are obtained, and then the target face (first face) among the multiple faces in the three-dimensional model is selected to determine the three-dimensional semantics of the first face, and the faces adjacent to the first face are also determined to have three-dimensional semantics. The depth information of the first face mapped to each texture image and the depth information (depth value) of the depth map corresponding to the texture image are determined, and the target texture image is selected from the multiple texture images based on the three-dimensional semantics of the first face and the second face adjacent to it and the above depth information, and the target texture image is used for mapping processing for the first face.

[0040] The texture image mapping method provided in the embodiment of the present application determines the target texture image from multiple texture images through the three-dimensional semantics and depth information of the target face in the three-dimensional model, and then uses the target texture image to map the target face. This effectively ensures the effectiveness and integrity of the texture image selection and improves the display effect of the three-dimensional model of the target scene.

[0041] In some embodiments of the present application, Figure 2 The second flow chart of the texture image mapping method provided in the embodiment of the present application is shown as follows: Figure 2 As shown, the mapping method of the above texture image specifically includes:

[0042] Step 202: Acquire multiple texture images.

[0043] Step 204: Determine a second semantics of any one of the plurality of texture images;

[0044] Specifically, the second semantics may be two-dimensional semantics of a texture image.

[0045] Specifically, a plurality of texture images of the target scene are obtained, and image recognition processing is performed on the texture images to obtain the second semantics of each texture image, that is, to obtain the two-dimensional semantics of each texture image. Generally, the two-dimensional semantics of the texture image of the target scene may include: "table", "chair", "bed", "wall", etc.

[0046] Step 206: Project the center of the first face patch to any one of the multiple texture images to obtain all second semantics of the first face patch.

[0047] Specifically, in the three-dimensional model of the target scene, the center of the first patch is mapped to each texture image to obtain all the two-dimensional semantics of the first patch, which is also the two-dimensional semantics of the texture image where its projection is located.

[0048] Furthermore, all the above two-dimensional semantics may be a set of two-dimensional semantics obtained by projecting the first face patch onto a plurality of texture images.

[0049] It should be noted that the first patch can be projected onto multiple texture images to obtain multiple second semantics. Generally speaking, the frequency of occurrence of each of the multiple second semantics is not the same. The second semantics of the first patch can be confirmed based on the frequency of occurrence of each second semantic. In addition, the probability that the second semantics of two adjacent patches are the same is high, so it is necessary to constrain the second semantics of the first patch and the second patch adjacent to it to be as consistent as possible.

[0050] Step 208: Determine the first semantics of the first face patch and the second face patch according to all the second semantics of the first face patch and all the second semantics of the second face patch.

[0051] Specifically, after the two-dimensional semantics of the first patch and the second patch are determined, the first semantics (three-dimensional semantics) of the first patch and the second patch may be determined using a Markov random field.

[0052] Step 210: Determine a target texture image from the multiple texture images according to the depth information of the first patch and the multiple texture images, and the first semantics of the first patch and the second patch.

[0053] Step 212: Use the target texture image to perform mapping on the first face.

[0054] In the embodiment of the present application, the second semantics of each texture image is first determined, and the center of the first facet in the three-dimensional model is projected onto each texture image to obtain all the second semantics of the first facet (two-dimensional semantics). The center of the second facet is projected onto each texture image to obtain all the second semantics of the second facet. According to all the second semantics of the first facet and all the second semantics of the second facet, the first semantics (three-dimensional semantics) of the first facet and the second facet are respectively determined.

[0055] The texture image mapping method provided in the embodiment of the present application determines the three-dimensional semantics of the facets through multiple two-dimensional semantics of the facets in the three-dimensional model, and then improves the accuracy of identifying the facet semantics through the three-dimensional semantics of the facets.

[0056] In some embodiments of the present application, the above step 206 specifically includes:

[0057] Step 206a: taking the occurrence frequency of any semantics in all the second semantics of the first face patch as a data item, and taking whether the second semantics of the first face patch and the second face patch are consistent as a smoothing item;

[0058] Step 206b: Determine the first semantics of the first patch and the second patch by using a Markov random field.

[0059] In an embodiment of the present application, the Markov random field needs to set data items and smoothing items. Specifically, the occurrence frequency of any one of the second semantics of the first face patch is set as a data item, and is set as a smoothing item based on whether the second semantics of the first face patch and the second face patch are consistent.

[0060] Furthermore, the data item and the smoothing item are used as input items of the Markov random field, and the first semantics of the first patch and the second patch are determined through the Markov random field.

[0061] Specifically, the calculation formula for the above data items is:

[0062]

[0063] Among them, F i is the first patch, l k is any two-dimensional semantic of the first face, for l k The frequency of occurrence.

[0064] The calculation formula of the above smoothing term is:

[0065]

[0066] Among them, F i is the first patch, F j is the second patch, l k is any two-dimensional semantic of the first face, l p is any two-dimensional semantics of the second patch. If the semantics of the first patch and the second patch are the same, the smoothness term is set to 1, otherwise, the smoothness term is set to 0.

[0067] The calculation formula of Markov random field is:

[0068]

[0069] According to the data item and the smoothing item, the three-dimensional semantics E(l) of the first face patch and the second face patch are calculated.

[0070] The texture image mapping method provided in the embodiment of the present application sets the frequency of occurrence of any one of the second semantics of the first face patch as a data item, sets it to a smooth phase according to whether the second semantics of the first face patch and the second face patch are consistent, and calculates the three-dimensional semantics of the first face patch according to the Markov random field, thereby improving the accuracy and convenience of converting the two-dimensional semantics of the face patch into three-dimensional semantics.

[0071] In some embodiments of the present application, Figure 3 The third flow chart of the texture image mapping method provided in the embodiment of the present application is shown as follows: Figure 3 As shown, the mapping method of the above texture image specifically includes:

[0072] Step 302: Acquire multiple texture images;

[0073] Step 304: Determine a second semantics of any one of the plurality of texture images;

[0074] Step 306: Project the center of the first face patch to any one of the multiple texture images to obtain all second semantics of the first face patch;

[0075] Step 308: Determine the first semantics of the first face patch and the second semantics of the second face patch according to all the second semantics of the first face patch and the second semantics of the second face patch;

[0076] Step 310: taking a depth parameter of a depth map corresponding to a texture image among the multiple texture images as a first depth parameter, and taking a depth parameter of the first patch projected onto any texture image among the multiple texture images as a second depth parameter;

[0077] Step 312: determining a target texture image from a plurality of texture images according to the first depth parameter, the second depth parameter, and the first semantics of the first patch and the second patch;

[0078] Step 314: Use the target texture image to perform mapping on the first face.

[0079] In an embodiment of the present application, the process of determining the target texture image needs to set the first depth parameter and the second depth parameter respectively. Specifically, the depth value of the depth map corresponding to a texture image among the multiple texture images is used as the first depth parameter, and the depth parameter of the first patch projected on any one of the multiple texture images is set as the second depth parameter.

[0080] Furthermore, according to the first depth parameter, the second depth parameter, and the first semantics of the first patch and the second patch, a target texture image is selected from the plurality of texture images, and the target texture image is the texture image with the best display effect corresponding to the first patch.

[0081] Specifically, the depth map corresponding to the texture image can be generated and acquired based on the camera device and the three-dimensional model. Any texture image among the multiple texture images can have a corresponding depth map, and the depth parameter of the depth map of the texture image is set as the first depth parameter.

[0082] Specifically, a depth parameter of the first patch projected onto the texture image is determined and set as the second depth parameter.

[0083] It is understandable that when texture mapping is performed on a 3D model, there are usually a large number of texture images to choose from. In order to obtain a clearer and more realistic mapping effect, a texture image with a normal viewing angle and accurate posture is usually selected as the target texture image for mapping for the first face. Therefore, the depth parameter of the first face projected onto the texture image can be used as a measure of the accuracy of the texture image posture.

[0084] It should be noted that the pose accuracy of the texture map is one factor in optimizing the texture map, and the smoothness of the texture image transition is another factor in optimizing the texture map. Therefore, when selecting the target texture map, it is necessary to refer to the semantics of the first face and the adjacent second face to ensure that the optimal target texture image is obtained.

[0085] The texture image mapping method provided in the embodiment of the present application selects a target texture image from multiple texture images by referring to the three-dimensional semantics of a first face patch and an adjacent second face patch, as well as a first depth parameter and a second depth parameter, thereby ensuring the accuracy and smoothness of texture mapping.

[0086] In some embodiments of the present application, the first depth parameter in the above texture image mapping method includes: a first depth value of a depth map corresponding to a texture image among multiple texture images; the second depth parameter includes: a projection area of ​​the first patch projected onto any texture image and a second depth value. The above step 312 specifically includes:

[0087] Step 312a: taking the projection area and the joint item of the difference between the first depth value and the second depth value as the data item, and taking whether the first semantics of the first face patch and the second face patch are consistent as the smoothness item;

[0088] Step 312b: Determine a target texture image from multiple texture images using a Markov random field.

[0089] In the above embodiment, when selecting the target texture map, it is necessary to refer to the semantics of the first face and the adjacent second face to ensure that the optimal target texture image is obtained. Therefore, it is necessary for the first face and the adjacent second face to have the same semantics as much as possible. When the semantics of the two are the same, the smoothness of the texture map transition can be guaranteed. Therefore, the semantic consistency of the first face and the second face is used as a measure of the smoothness of the texture image.

[0090] Furthermore, the larger the area of ​​the first patch projected onto the texture image, the more appropriate the shooting angle of the camera device can be considered. At the same time, the difference between the depth value of the texture image itself (the first depth value) and the depth of the first patch projected onto the texture image (the second depth value) is used as a measure of the accuracy of the texture image posture, thereby further optimizing the accuracy of the texture image posture.

[0091] In the embodiment of the present application, a target texture image among multiple texture images can be determined by a Markov random field, and the Markov random field requires setting data items and smoothing items.

[0092] Specifically, the projection area and the difference between the first depth value and the second depth value are set as a data item, and whether the second semantics of the first patch and the second patch are consistent is set as a smoothing item. The data item and the smoothing item are used as input items, and a target texture image is calculated from multiple texture images through a Markov random field.

[0093] In a possible implementation, the process of calculating the target texture image from multiple texture images is regarded as a Markov random field calculation process. Markov random field calculation requires setting data items and smoothing items. The calculation formula of the data item is:

[0094] Data(F,v)=S / |dt-d|;

[0095] Among them, F is the first patch, v is the selected texture image, S is the area of ​​the first patch projected onto the texture image, dt is the second depth value of the first patch projected onto the texture image, and d is the first depth value in the depth map corresponding to the texture image.

[0096] The calculation formula of the smoothing term is:

[0097]

[0098] Among them, l k is the first semantics (3D semantics) of the first face, l p is the first semantics of the second patch. If the semantics of the first patch and the second patch are the same, the smoothness item is set to 1, otherwise, the smoothness item is set to 0.

[0099] The calculation formula of Markov random field is:

[0100]

[0101] Among them, F i is the first patch, F j For the second patch, a target texture image E(v) is calculated from multiple texture images according to the data item and the smoothing item.

[0102] The texture image mapping method provided in the embodiment of the present application calculates the target texture image in the texture image according to the Markov random field by setting the data item and the smoothing item respectively, thereby ensuring the accuracy of determining the target texture image among multiple texture images.

[0103] In some embodiments of the present application, Figure 4 FIG. 4 shows a fourth flow chart of a texture image mapping method provided in an embodiment of the present application. Figure 4 As shown, the mapping method of the texture image includes:

[0104] Step 402: Acquire multiple texture images;

[0105] Step 404: Determine the first semantics of the first face patch and the second face patch;

[0106] Step 406: determining a target texture image from the multiple texture images according to the depth information of the first patch and the multiple texture images, and the first semantics of the first patch and the second patch;

[0107] Step 408: Use the target texture image to perform mapping on the first face;

[0108] Step 410: determining a third facet corresponding to a boundary of an invisible area in the three-dimensional model, and filling the third facet with a texture color of a fourth facet, where the fourth facet is a facet adjacent to the third facet and outside the boundary of the invisible area;

[0109] Specifically, the invisible area in the three-dimensional model is an area where texture information is missing inside the three-dimensional model. The third patch is a patch corresponding to the boundary of the texture missing area in the three-dimensional model. The fourth patch is a patch adjacent to the third patch, and the three-dimensional model area corresponding to the fourth patch is outside the boundary of the invisible area.

[0110] Step 412: Iteratively fill the patches corresponding to the boundaries of the invisible area until the invisible area is completely filled.

[0111] In the above embodiment, the third facet corresponding to the boundary of the invisible area in the three-dimensional model is determined, and the fourth facet adjacent to the third facet is determined, the fourth facet is located outside the boundary of the invisible area, the texture color of the fourth facet is filled into the third facet, and the texture color is filled in the faces corresponding to the boundary of the invisible area from the outside to the inside in sequence until the invisible area is completely filled.

[0112] Specifically, a traversal algorithm can be used to determine all the hole areas with missing textures in the three-dimensional model. The hole refers to a connected domain formed by patches without textures, and the hole area is the invisible area mentioned above.

[0113] Specifically, for each patch in the hole area (the third patch), the color of the patch with texture (the fourth patch) is used to diffuse inward in circles from the outside to the inside.

[0114] Exemplarily, a texture patch (image block) may be generated for each fourth patch and the third patch, and the pacth of the fourth patch adjacent to and corresponding to the third patch may be assigned to the patch of the third patch.

[0115] Furthermore, after the above step 412, the texture image mapping method provided by the present application may further include:

[0116] Step 414: Perform color balancing processing on all filled third facets.

[0117] It is understandable that due to differences in exposure and white balance of the camera, there are differences in color between different texture images. In order to balance the color difference between texture patches, the filled color can be smoothed to balance the color difference between the hole-filled patch and the patch captured by the camera.

[0118] Furthermore, after the above step 414, the texture image mapping method provided by the present application may further include:

[0119] Step 416: classify all third face patches, synthesize third face patches of the same type into a virtual perspective, and project the third face patches of the same type to the virtual perspective to obtain a first texture image;

[0120] Step 418: synthesize the first texture image into a first texture patch, and perform a balancing process on the first texture patch.

[0121] Specifically, after the color balancing process, there may still be differences in the invisible area. To this end, the normal vectors of the third facets may be used to classify the facets, synthesize the facets belonging to the same category into a virtual perspective, and then project all the facets belonging to the category to the virtual perspective to obtain a new texture image (first texture image).

[0122] Furthermore, a new texture patch is synthesized from the first texture image, and then the texture patch is smoothed to make the hole filling more natural.

[0123] In the embodiment provided by the present application, in the three-dimensional model of the target scene, there will be an area with missing texture, and the area with missing texture may correspond to multiple patches. From the boundary of the area with missing texture to the inside, the patches with missing color are set as patches to be filled (third patches) in sequence, and the patches to be filled are filled with color according to the color of the patches (fourth patches) adjacent to the patches to be filled and with complete texture color, until the texture colors of all patches to be filled are fully filled. After all the filling is completed, color smoothing can be performed between the filled patches to ensure the color display effect.

[0124] The texture image mapping method provided in the embodiment of the present application fills the facets inside the invisible area with texture colors according to the facets outside the invisible area of ​​the three-dimensional model, thereby ensuring the integrity of the texture color of the three-dimensional model.

[0125] In some embodiments of the present application, the texture image mapping method provided by the present application, after step 412 or step 418, or before or after step 108, the texture image mapping method further includes:

[0126] Step 420: Adjust the color difference between multiple target texture images in the three-dimensional model.

[0127] In a possible implementation, after texture mapping is performed on the three-dimensional model, multiple target texture images that have been mapped in the three-dimensional model may have color differences. In order to obtain a better visual effect, the color differences between the target texture images need to be adjusted.

[0128] In the above embodiment, the color difference of the vertices of a patch is set to g v , assuming that the patch is located at the seam of two texture maps, split the vertex into two vertices V left and V right , the colors of the two tops in the two texture images are f vleft and f vrihgt Therefore, it is necessary to optimize the difference between two vertices plus the color difference, the color difference g of adjacent vertices v To be as consistent as possible, color balancing is performed by optimizing the following energy equation.

[0129] The function used to adjust the color difference is:

[0130]

[0131] The texture image mapping method provided in the embodiment of the present application adjusts the color difference between multiple target texture images in the three-dimensional model, balances the color of the texture mapping, and improves the display effect of the three-dimensional model.

[0132] In some embodiments of the present application, a texture image mapping device is provided. Figure 5 The structural block diagram of the texture image mapping device provided by the embodiment of the present application is shown as follows: Figure 5 As shown, the texture image mapping device 500 includes:

[0133] An acquisition unit 502, used to acquire a plurality of texture images;

[0134] A processing unit 504 is configured to determine a first semantics of a first face patch and a second face patch, wherein the second face patch is adjacent to the first face patch in the three-dimensional model;

[0135] The processing unit 504 is further configured to determine a target texture image from the multiple texture images according to the depth information of the first patch and the multiple texture images, and the first semantics of the first patch and the second patch;

[0136] The processing unit 504 is further configured to perform mapping processing on the first surface using the target texture image.

[0137] In the embodiment of the present application, the acquisition unit 502 first acquires multiple texture images, and the processing unit 504 then determines the first semantics of the first face patch and the second face patch adjacent to it in the three-dimensional model; the processing unit 504 selects a target texture image based on the depth information of the first face patch and the multiple texture images and the first semantics of the first face patch and the second face patch for the multiple texture images acquired; the processing unit 504 uses the target texture image to perform mapping processing on the first face patch.

[0138] The texture image mapping device provided in the embodiment of the present application, in the process of texture mapping of a three-dimensional model of a target scene, among multiple texture images, the mapping device determines the target texture image through the semantics of the facets in the three-dimensional model and the depth information of the texture image, and uses the target texture image to perform mapping processing in the three-dimensional model. In this way, the accuracy of the texture image posture is judged by using the depth information of the facets and the texture image, and the smoothness of the texture image texture mapping selection is ensured by using the semantic information of the facets, thereby ensuring the effectiveness and integrity of the texture image texture mapping, and improving the display effect of the three-dimensional model of the target scene.

[0139] In some embodiments of the present application, the processing unit 504 is also used to determine the second semantics of any one of a plurality of texture images; project the center of the first face patch to any one of a plurality of texture images to obtain all the second semantics of the first face patch; and determine the first semantics of the first face patch and the second semantics of the second face patch based on all the second semantics of the first face patch and the second semantics of the second face patch.

[0140] The texture image mapping device provided in the embodiment of the present application determines the three-dimensional semantics of the facets in the three-dimensional model according to multiple two-dimensional semantics of the facets through a processing unit, and performs semantic segmentation according to the three-dimensional semantics of the facets to improve the accuracy of semantic segmentation.

[0141] In some embodiments of the present application, the processing unit 504 is also used to take the occurrence frequency of any semantics of all the second semantics of the first patch as a data item, and whether the second semantics of the first patch and the second patch are consistent as a smoothing item, and determine the first semantics of the first patch and the second patch through a Markov random field.

[0142] The texture image mapping device provided in the embodiment of the present application sets data items and smoothing items respectively through a processing unit, calculates the three-dimensional semantics of the target face according to the Markov random field, and realizes semantic segmentation of the face in the three-dimensional model according to the three-dimensional semantics.

[0143] In some embodiments of the present application, the processing unit 504 is also used to use the depth parameter of the depth map corresponding to a texture image among multiple texture images as the first depth parameter, and use the depth parameter of the first patch projected onto any texture image among the multiple texture images as the second depth parameter; the processing unit 504 is also used to determine the target texture image from the multiple texture images based on the first depth parameter, the second depth parameter, and the first semantics of the first patch and the second patch.

[0144] The texture image mapping device provided in the embodiment of the present application selects a target texture image from multiple texture images according to the three-dimensional semantics of the target face and the adjacent face and the first depth parameter and the second depth parameter through a processing unit, and selects the target texture image for mapping to ensure the accuracy of texture mapping.

[0145] In some embodiments of the present application, the first depth parameter includes: a first depth value of a depth map corresponding to a texture image among multiple texture images; the second depth parameter includes: a projection area of ​​a first patch projected onto any texture image and a second depth value; the processing unit 504 is also used to take the projection area and a joint item of the difference between the first depth value and the second depth value as a data item, and whether the first semantics of the first patch and the second patch are consistent as a smoothing item, and determine a target texture image from multiple texture images through a Markov random field.

[0146] The texture image mapping device provided in the embodiment of the present application sets the data item and the smoothing item respectively through the processing unit, and calculates the target texture image in the texture image according to the Markov random field, thereby ensuring the accuracy of the target texture image.

[0147] In some embodiments of the present application, the processing unit 504 is further used to determine a third facet corresponding to a boundary of an invisible area in the three-dimensional model, and fill the third facet with a texture color of a fourth facet, where the fourth facet is a facet adjacent to the third facet and located outside the boundary of the invisible area; the processing unit 504 is further used to alternately fill the facet corresponding to the boundary of the invisible area until the invisible area is completely filled;

[0148] The texture image mapping device provided in the embodiment of the present application uses a processing unit to fill the faces inside the invisible area with texture colors based on the faces outside the invisible area of ​​the three-dimensional model, thereby ensuring the integrity of the texture color of the three-dimensional model.

[0149] In some embodiments of the present application, the processing unit 504 is further configured to adjust the color difference between multiple target texture images in the three-dimensional model.

[0150] The texture image mapping device provided in the embodiment of the present application adjusts the color difference between multiple target texture images in the three-dimensional model through a processing unit, balances the color of the texture mapping, and improves the display effect of the three-dimensional model.

[0151] The texture image mapping device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than the terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0152] The texture image mapping device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0153] The texture image mapping device provided in the embodiment of the present application can implement each process implemented in the above method embodiment, and will not be described again here to avoid repetition.

[0154] Optionally, an embodiment of the present application further provides an electronic device, Figure 6 A structural block diagram of an electronic device according to an embodiment of the present application is shown. Figure 6 As shown, the electronic device 600 includes a processor 602 and a memory 604. The memory 604 stores programs or instructions that can be executed on the processor 602. When the program or instruction is executed by the processor 602, the various steps of the above-mentioned method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0155] Further illustratively, if Figure 7 As shown, the processor 602 may also implement the following process in executing the texture image mapping method:

[0156] In an embodiment of the present application, multiple two-dimensional semantics of the facets in the three-dimensional model are determined based on the RGB Depth map in the three-dimensional model, and then the three-dimensional semantics of the facets are further determined based on the multiple two-dimensional semantics of the facets. Combined with the depth information in the RGB Depth map and the three-dimensional semantics of the facets, a target texture view with better effect is selected from multiple views. The hollow areas in the three-dimensional model are filled with color according to the texture colors of the target view and other views, and the filled colors are balanced to make the colors more natural. Then, the target view is mapped to the three-dimensional model to ensure the integrity of the three-dimensional model.

[0157] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0158] Figure 8 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0159] The electronic device 800 includes but is not limited to: a radio frequency unit 801, a network module 802, an audio output unit 803, an input unit 804, a sensor 805, a display unit 806, a user input unit 807, an interface unit 808, a memory 809, a processor 810 and other components.

[0160] Those skilled in the art will appreciate that the electronic device 800 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 810 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 8 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0161] Among them, processor 810 is used to obtain multiple texture images; determine the first semantics of a first face and a second face, where the second face is adjacent to the first face in a three-dimensional model; determine a target texture image from the multiple texture images based on depth information of the first face and the multiple texture images, and the first semantics of the first face and the second face; and use the target texture image to map the first face.

[0162] In the electronic device provided by the embodiment of the present application, in the process of texture mapping of a three-dimensional model of a target scene, among multiple texture images, a mapping device determines a target texture image through the semantics of the facets in the three-dimensional model and the depth information of the texture image, and uses the target texture image to perform mapping processing in the three-dimensional model. In this way, the accuracy of the texture image posture is judged by using the depth information of the facets and the texture image, and the smoothness of the texture image texture mapping selection is ensured by using the semantic information of the facets, thereby ensuring the effectiveness and integrity of the texture image texture mapping, and improving the display effect of the three-dimensional model of the target scene.

[0163] Furthermore, the processor 810 is also used to determine the second semantics of any one of a plurality of texture images; project the center of the first face patch to any one of a plurality of texture images to obtain all second semantics of the first face patch; and determine the first semantics of the first face patch and the second semantics of the second face patch based on all second semantics of the first face patch and the second semantics of the second face patch.

[0164] The electronic device provided in the embodiment of the present application determines the three-dimensional semantics of a facet through multiple two-dimensional semantics of the facet in the three-dimensional model, and performs semantic segmentation according to the three-dimensional semantics of the facet to improve the accuracy of semantic segmentation.

[0165] Furthermore, processor 810 is also used to take the occurrence frequency of any semantics of all second semantics of the first face patch as a data item, and whether the second semantics of the first face patch and the second face patch are consistent as a smoothing item, and determine the first semantics of the first face patch and the second face patch through a Markov random field.

[0166] The electronic device provided in the embodiment of the present application calculates the three-dimensional semantics of the target face according to the Markov random field by setting the data item and the smoothing item respectively, thereby realizing semantic segmentation of the face in the three-dimensional model according to the three-dimensional semantics.

[0167] Furthermore, the processor 810 is also used to use a depth parameter of a depth map corresponding to a texture image among multiple texture images as a first depth parameter, and a depth parameter of a first patch projected onto any one of the multiple texture images as a second depth parameter; determine a target texture image from the multiple texture images based on the first depth parameter, the second depth parameter, and the first semantics of the first patch and the second patch.

[0168] The electronic device provided in the embodiment of the present application selects a target texture image from multiple texture images through the three-dimensional semantics of the target face and the adjacent face and the first depth parameter and the second depth parameter, and selects the target texture image for mapping to ensure the accuracy of texture mapping.

[0169] Furthermore, the first depth parameter includes: a first depth value of a depth map corresponding to a texture image among a plurality of texture images; the second depth parameter includes: a projection area of ​​a first patch projected onto any texture image and a second depth value; the processor 810 is also used to use a joint item of the projection area and a difference between the first depth value and the second depth value as a data item, and whether the first semantics of the first patch and the second patch are consistent as a smoothing item, and determine a target texture image from a plurality of texture images through a Markov random field.

[0170] The electronic device provided in the embodiment of the present application calculates the target texture image in the texture image according to the Markov random field by respectively setting the data item and the smoothing item, thereby ensuring the accuracy of the target texture image.

[0171] Furthermore, the processor 810 is also used to determine the third facet corresponding to the boundary of the invisible area in the three-dimensional model, fill the texture color of the fourth facet into the third facet, the fourth facet is a facet adjacent to the third facet and located outside the boundary of the invisible area; iteratively fill the facets corresponding to the boundary of the invisible area until the invisible area is completely filled.

[0172] The electronic device provided in the embodiment of the present application fills the facets inside the invisible area with texture colors according to the facets outside the invisible area in the three-dimensional model, thereby ensuring the integrity of the texture colors of the three-dimensional model.

[0173] Furthermore, the processor 810 is further configured to adjust the color difference between multiple target texture images in the three-dimensional model.

[0174] The electronic device provided in the embodiment of the present application adjusts the color difference between multiple target texture images in a three-dimensional model, balances the colors of the texture map, and improves the display effect of the three-dimensional model.

[0175] It should be understood that in the embodiment of the present application, the input unit 804 may include a graphics processor (Graphics Processing Unit, GPU) 8041 and a microphone 8042, and the graphics processor 8041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 806 may include a display panel 8061, and the display panel 8061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 807 includes a touch panel 8071 and at least one of other input devices 8072. The touch panel 8071 is also called a touch screen. The touch panel 8071 may include two parts: a touch detection device and a touch controller. Other input devices 8072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0176] The memory 809 can be used to store software programs and various data. The memory 809 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 809 may include a volatile memory or a non-volatile memory, or the memory 809 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 809 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0177] The processor 810 may include one or more processing units; optionally, the processor 810 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 810.

[0178] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned texture image mapping method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0179] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0180] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned texture image mapping method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0181] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0182] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned texture image mapping method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0183] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0184] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0185] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A texture image mapping method, characterized in that: The mapping method of the texture image comprises: Get multiple texture images; Determine first semantics of a first face patch and a second face patch, wherein the first face patch and the second face patch are faces in a three-dimensional model, and the second face patch is adjacent to the first face patch; Determine a target texture image from the multiple texture images according to the depth information of the first patch and the multiple texture images, and the first semantics of the first patch and the second patch; Using the target texture image to perform mapping on the first surface; The step of determining a target texture image from the multiple texture images according to the depth information of the first patch and the multiple texture images, and the first semantics of the first patch and the second patch comprises: The target texture image is determined from the multiple texture images by using a Markov random field. The Markov random field needs to set a data item and a smoothing item. Whether the first semantics of the first face patch and the second face patch are consistent is used as a smoothing item. The calculation formula of the data item is: Data(F, v) = S / |dt-d|; Wherein, F is the first patch, v is the selected texture image, S is the area of ​​the first patch projected onto the texture image, dt is the second depth value of the first patch projected onto the texture image, and d is the first depth value in the depth map corresponding to the texture image; The determining the first semantics of the first face patch and the second face patch specifically includes: Determine the first semantics of the first face patch and the second face patch according to all the second semantics of the first face patch and all the second semantics of the second face patch; Among them, the first semantics is three-dimensional semantics, and the second semantics is two-dimensional semantics. The first semantics of the first patch and the second patch are determined by a Markov random field. The Markov random field needs to set data items and smoothing items. The data item is the inverse of any two-dimensional semantics of the first patch. Whether the first semantics of the first patch and the second patch are consistent is used as the smoothing item.

2. The texture image mapping method according to claim 1, characterized in that: The determining of the first semantics of the first face patch and the second face patch further includes: determining a second semantic meaning of any one of the plurality of texture images; Project the center of the first face patch to any one of the multiple texture images to obtain all second semantics of the first face patch.

3. The texture image mapping method according to claim 2, characterized in that: The determining, according to all the second semantics of the first face patch and the second semantics of the second face patch, the first semantics of the first face patch and the second face patch specifically includes: The occurrence frequency of any semantics of all second semantics of the first patch is taken as a data item, and whether the second semantics of the first patch and the second patch are consistent is taken as a smoothing item, and the first semantics of the first patch and the second patch are determined through a Markov random field.

4. The texture image mapping method according to any one of claims 1 to 3, characterized in that: Determining a target texture image from the multiple texture images according to the depth information of the first patch and the multiple texture images, and the first semantics of the first patch and the second patch, specifically includes: Using a depth parameter of a depth map corresponding to one of the multiple texture images as a first depth parameter, and using a depth parameter of the first patch projected onto any one of the multiple texture images as a second depth parameter; A target texture image is determined from the multiple texture images according to the first depth parameter, the second depth parameter, and first semantics of the first patch and the second patch.

5. The texture image mapping method according to claim 4, characterized in that: The first depth parameter includes: a first depth value of a depth map corresponding to one of the multiple texture images; The second depth parameter includes: a projection area of ​​the first patch projected onto any one of the texture images and a second depth value; The determining a target texture image from the plurality of texture images according to the first depth parameter, the second depth parameter, and the first semantics of the first patch and the second patch specifically includes: The projection area and the joint item of the difference between the first depth value and the second depth value are taken as the data item, whether the first semantics of the first patch and the second patch are consistent is taken as the smoothing item, and the target texture image is determined from the multiple texture images through a Markov random field.

6. The texture image mapping method according to any one of claims 1 to 3, characterized in that: The mapping method further comprises: Determine a third facet corresponding to a boundary of the invisible area in the three-dimensional model, and fill the third facet with a texture color of a fourth facet, wherein the fourth facet is a facet adjacent to the third facet and located outside the boundary of the invisible area; The patches corresponding to the boundaries of the invisible area are filled iteratively until the invisible area is completely filled.

7. The texture image mapping method according to any one of claims 1 to 3, characterized in that: The mapping method further comprises: The color difference between the plurality of target texture images in the three-dimensional model is adjusted.

8. A texture image mapping device, characterized in that: The mapping device of the texture image comprises: An acquisition unit, used for acquiring a plurality of texture images; A processing unit, configured to determine first semantics of a first face patch and a second face patch, wherein the first face patch and the second face patch are faces in a three-dimensional model, and the second face patch is adjacent to the first face patch; The processing unit is further configured to determine a target texture image from the multiple texture images according to the depth information of the first patch and the multiple texture images, and the first semantics of the first patch and the second patch; The processing unit is further used to perform mapping processing on the first surface patch using the target texture image; The processing unit is specifically used to determine the target texture image from the multiple texture images through a Markov random field. The Markov random field needs to set a data item and a smoothing item. Whether the first semantics of the first face patch and the second face patch are consistent is used as a smoothing item. The calculation formula of the data item is: Data(F, v) = S / |dt-d|; Wherein, F is the first patch, v is the selected texture image, S is the area of ​​the first patch projected onto the texture image, dt is the second depth value of the first patch projected onto the texture image, and d is the first depth value in the depth map corresponding to the texture image; The processing unit is further configured to determine the first semantics of the first face patch and the second face patch according to all second semantics of the first face patch and all second semantics of the second face patch; Among them, the first semantics is three-dimensional semantics, and the second semantics is two-dimensional semantics. The first semantics of the first patch and the second patch are determined by a Markov random field. The Markov random field needs to set data items and smoothing items. The data item is the inverse of any two-dimensional semantics of the first patch. Whether the first semantics of the first patch and the second patch are consistent is used as the smoothing item.

9. An electronic device, characterized in that: include: A memory on which programs or instructions are stored; A processor, configured to implement the steps of the texture image mapping method as described in any one of claims 1 to 7 when executing the program or instruction.

10. A readable storage medium having a program or instruction stored thereon, characterized in that: When the program or instruction is executed by a processor, the steps of the texture image mapping method according to any one of claims 1 to 7 are implemented.

Images