Material migration method and device, data processing method

Abstract

The present application discloses a material migration method and device, and a data processing method. The method includes: extracting a first material map of a first three-dimensional object from a first image; receiving a mapping instruction for mapping a second three-dimensional object; responding to the mapping instruction, using a material migration module to migrate the first material map to obtain a second material map of a second three-dimensional object, wherein the material migration module constructs a semantic migration relationship between three-dimensional objects of different styles; and rendering the second material map onto the second three-dimensional object. The present invention solves the technical problem that in the prior art, in the process of designing a three-dimensional image of an object, the material of the two-dimensional image can be used to generate a corresponding three-dimensional model map through a deep learning network, and then the three-dimensional model map is re-rendered back to the three-dimensional model of the object. Since the migration of the above-mentioned material is limited to the migration between objects of the same style, the migration process of the material is inefficient.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a material migration method and device, and a data processing method. Background Art

[0002] In the field of clothing design, designers usually spend a lot of energy on the design of clothing textures. A common design idea is to collect clothing-related show pictures, street photos, product pictures, etc., and extract popular texture elements from them for design. For traditional clothing texture design, its verification idea is usually based on two-dimensional images, and original texture materials are required. But in most cases, it is difficult for designers to obtain the original texture materials, and it takes a lot of effort. Therefore, if designers want to verify the texture effect of a certain popular clothing, they need to manually process the materials from the corresponding show pictures, street photos, etc. to obtain the clothing texture and verify the design plan, which increases the design difficulty and design cost.

[0003] There are some solutions that can automatically generate the corresponding 3D model UV map from the front and back textures of the clothing image through a deep learning network, and then re-render the UV map back to the 3D clothing model. However, the disadvantage of this solution is that it can only achieve the migration of clothing textures of the same style. For example, if the user wants to migrate the texture pattern on a T-shirt to a 3D model, the corresponding 3D model must be a T-shirt 3D model, and it is impossible to migrate the T-shirt texture to a dress 3D model. Therefore, the scope of use is very limited.

[0004] In the process of designing a three-dimensional image of an object in the prior art, the material of the two-dimensional image can be used to generate a corresponding three-dimensional model map through a deep learning network, and then the three-dimensional model map is re-rendered back into the three-dimensional model of the object. Since the migration of the above-mentioned materials is limited to the migration between objects of the same style, the migration process of the materials is inefficient, and no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a material migration method and device, and a data processing method, so as to at least solve the technical problem in the prior art that, in the process of designing a three-dimensional image of an object, the material of the two-dimensional image can be used to generate a corresponding three-dimensional model map through a deep learning network, and then the three-dimensional model map is re-rendered back into the three-dimensional model of the object. Since the migration of the above-mentioned materials is limited to the migration between objects of the same style, the efficiency of the material migration process is low.

[0006] According to one aspect of an embodiment of the present invention, a material migration method is provided, comprising: extracting a first material map of a first three-dimensional object from a first image; receiving a mapping instruction for mapping a second three-dimensional object, wherein the mapping instruction is used to indicate that the material in the first material map needs to be replaced onto the surface of the second three-dimensional object, and the first three-dimensional object and the second three-dimensional object are of different styles; in response to the mapping instruction, using a material migration module to migrate the first material map to obtain a second material map of the second three-dimensional object, wherein the material migration module constructs a semantic migration relationship between three-dimensional objects of different styles; and rendering the second material map onto the second three-dimensional object.

[0007] According to another aspect of an embodiment of the present invention, a material migration method is also provided, including: displaying a first image and a second image in a display interface, wherein the object displayed in the first image is a first three-dimensional object, and the object displayed in the second image is a second three-dimensional object, and the first three-dimensional object and the second three-dimensional object are of different styles; selecting the first image, and extracting the material of the first three-dimensional object from the first image to obtain a first material map; receiving a mapping instruction for mapping the second three-dimensional object, wherein the mapping instruction is used to indicate that the material in the first material map needs to be replaced onto the surface of the second three-dimensional object; in response to the mapping instruction, using a material migration module to migrate the first material map to obtain a second material map of the second three-dimensional object, wherein the material migration module constructs a semantic migration relationship between three-dimensional objects of different styles; and displaying a second image in which the second material map is rendered onto the second three-dimensional object.

[0008] According to another aspect of an embodiment of the present invention, a material migration device is also provided, including: an extraction module, used to extract a first material map of a first three-dimensional object from a first image; a trigger module, used to receive a mapping instruction for mapping a second three-dimensional object, wherein the mapping instruction is used to indicate that the material in the first material map needs to be replaced on the surface of the second three-dimensional object, and the styles of the first three-dimensional object and the second three-dimensional object are different; a response module, used to respond to the mapping instruction, and use a material migration module to migrate the first material map to obtain a second material map of the second three-dimensional object, wherein the material migration module constructs a semantic migration relationship between three-dimensional objects of different styles; a rendering module, used to render the second material map onto the second three-dimensional object.

[0009] According to another aspect of an embodiment of the present invention, a material migration device is also provided, including: a first display module, used to display a first image and a second image in a display interface, wherein the object displayed in the first image is a first three-dimensional object, and the object displayed in the second image is a second three-dimensional object, and the first three-dimensional object and the second three-dimensional object are of different styles; an extraction module, used to select the first image and extract the material of the first three-dimensional object from the first image to obtain a first material map; receive a mapping instruction for mapping the second three-dimensional object, wherein the mapping instruction is used to indicate that the material in the first material map needs to be replaced on the surface of the second three-dimensional object; a response module, used to respond to the mapping instruction, use the material migration module to migrate the first material map to obtain a second material map of the second three-dimensional object, wherein the material migration module constructs a semantic migration relationship between three-dimensional objects of different styles; a second display module, used to display a second image in which the second material map is rendered on the second three-dimensional object.

[0010] According to another aspect of an embodiment of the present invention, a storage medium is provided. The storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to perform the migration of the above-mentioned material.

[0011] According to another aspect of an embodiment of the present invention, a processor is further provided, and the processor is used to run a program, wherein the migration of the above-mentioned materials is performed when the program is running.

[0012] According to another aspect of an embodiment of the present invention, a material migration system is also provided, including: a processor; and a memory connected to the processor, for providing the processor with instructions for processing the following processing steps: extracting a first material map of a first three-dimensional object from a first image; receiving a mapping instruction for mapping a second three-dimensional object, wherein the mapping instruction is used to indicate that the material in the first material map needs to be replaced onto the surface of the second three-dimensional object, and the first three-dimensional object and the second three-dimensional object are of different styles; in response to the mapping instruction, using a material migration module to migrate the first material map to obtain a second material map of the second three-dimensional object, wherein the material migration module constructs a semantic migration relationship between three-dimensional objects of different styles; and rendering the second material map onto the second three-dimensional object.

[0013] According to another aspect of an embodiment of the present invention, a data processing method is also provided, including: acquiring a first image; extracting a first material map of a first three-dimensional object from the first image; and rendering the first material map onto the second three-dimensional object based on a geometric transformation relationship between the first three-dimensional object and the second three-dimensional object.

[0014] According to another aspect of an embodiment of the present invention, a data processing method is also provided, including: displaying a first image and a second image; receiving a material extraction instruction, and extracting a first material map of a first three-dimensional object from the first image according to the material extraction instruction; receiving a material migration instruction, and rendering the first material map onto the second three-dimensional object based on a geometric transformation relationship between the first three-dimensional object and the second three-dimensional object in the second image according to the material migration instruction.

[0015] In an embodiment of the present application, a first material map of a first three-dimensional object is extracted from a first image; a mapping instruction for mapping a second three-dimensional object is received, wherein the mapping instruction is used to indicate that a material in the first material map needs to be replaced onto a surface of the second three-dimensional object, and the first three-dimensional object and the second three-dimensional object are of different styles; in response to the mapping instruction, the first material map is migrated using a material migration module to obtain a second material map of the second three-dimensional object, wherein the material migration module constructs a semantic migration relationship between three-dimensional objects of different styles; and the second material map is rendered onto the second three-dimensional object. The above scheme obtains the texture of the object from the image, restores the material map of the three-dimensional model under the same style, and then sends it to the material migration module to obtain the material map suitable for objects of other styles, and then renders based on the newly generated material map, so as to complete the migration of materials between objects of different styles, thereby realizing the migration of textures between objects of different styles, and solves the problem in the prior art that in the process of designing the three-dimensional image of an object, the material of the two-dimensional image can be used to generate the corresponding three-dimensional model map through a deep learning network, and then the three-dimensional model map is re-rendered back into the three-dimensional model of the object. Since the above-mentioned material migration is limited to the migration between objects of the same style, the technical problem of low efficiency of the material migration process is caused. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a computing device (or mobile device) for implementing a material migration method is shown;

[0018] Figure 2 This is a process of a material migration method according to Embodiment 1 of the present application;

[0019] Figure 3 is a schematic diagram of extracting a material map according to Embodiment 1 of the present application;

[0020] Figure 4is a schematic diagram of material migration according to Example 1 of the present application;

[0021] Figure 5 is a flow chart of another material migration method according to Embodiment 2 of the present application;

[0022] Figure 6 is a schematic diagram of a material migration device according to Embodiment 3 of the present application;

[0023] Figure 7 is a schematic diagram of a material migration device according to Embodiment 4 of the present application;

[0024] Figure 8 is a structural block diagram of a computing device according to an embodiment of the present invention;

[0025] Fig. 9 is a flow chart of a data processing method according to Embodiment 7 of the present application;

[0026] Fig.10 is a flow chart of another data processing method according to Embodiment 8 of the present application;

[0027] Fig.11 is a schematic diagram of a data processing device according to Embodiment 9 of the present application;

[0028] Fig.12 It is a schematic diagram of a data processing device according to embodiment 10 of the present application. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0032] 3D Cloth: It is used to represent 3D clothing models synthesized based on 3D modeling, physical simulation, manual modeling and other schemes, and has mesh and texture information.

[0033] UV Map: UV is the abbreviation of texture mapping coordinates. It defines the position of each point on the image. These points are interconnected with the 3D model to determine the position of the surface texture map. Just like a flat piece of cloth is cut and made into a 3D piece of clothing. UV is to accurately match each point on the image to the surface of the model object. The gaps between points are processed by the software to smoothly interpolate the image. This is called UV mapping.

[0034] Texture Transfer: Transfer the texture of an object in the original image to a target object, which can exist in a 3D model or a 2D image. This innovative solution specifically refers to transferring the texture of clothing in a single image to a target 3D clothing model.

[0035] Texture Inpaint: How to use other information to fill in the missing part of the texture image, and make it impossible for the human eye to distinguish the filled part. The texture in this innovative solution specifically refers to the UV map.

[0036] Intrinsic Decomposition: Decompose an image into two parts, the reflection map and the illumination map. The reflection map refers to the part of the image that can remain unchanged under changing lighting conditions. It is an image that reflects the lighting conditions of the original image. This innovative solution specifically refers to decomposing the clothing lighting and wrinkle information in a single image.

[0037] Texture Super Resolution: A technique for increasing the resolution of texture images.

[0038] Example 1

[0039] According to an embodiment of the present invention, an embodiment of a material migration method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computing device or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computing device (or mobile device) for implementing a material migration method. Figure 1 As shown, the computing device 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0041] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computing device 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0042] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the material migration method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned material migration method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computing device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0043] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computing device 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0044] The display may be, for example, a touch screen liquid crystal display (LCD) that may enable a user to interact with a user interface of computing device 10 (or mobile device).

[0045] It should be noted that, in some optional embodiments, the above Figure 1 The computer device (or mobile device) shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the above-described computer device (or mobile device).

[0046] Under the above operating environment, this application provides Figure 2 How to migrate the material shown. Figure 2 This is a flowchart of a material migration method according to Example 1 of the present application.

[0047] Step S21: extracting a first material map of a first three-dimensional object from a first image.

[0048] Specifically, the first material map may be a texture map on a first three-dimensional object. The first three-dimensional object may be an object from which material is to be extracted. The first image may be an image containing the first three-dimensional object.

[0049] The first image may be a two-dimensional image including the first three-dimensional object. In an optional embodiment, taking clothing design as an example, the first image may be a show picture or a street shot picture. After the designer gets design inspiration from the show or street shot, the show picture or street shot picture may be used as the first image, and the material texture of the clothing may be extracted from the show picture or street shot picture.

[0050] The first image may also be a three-dimensional image including the first three-dimensional object. In an optional embodiment, taking a three-dimensional dress-up game as an example, the first image may be a three-dimensional model of a garment that already exists in the game, and when the game player needs to use the material of the garment to dress up, the material texture of the garment may be extracted from the first image.

[0051] It should be noted that, currently in order to carry out related designs based on the same material map, designers need to obtain the source file of the material map, while based on the solution of the present application, only the first image containing the first three-dimensional object needs to be obtained, such as show photos and street photos, thereby greatly reducing the difficulty of obtaining materials and improving the efficiency of finding materials.

[0052] Step S23, receiving a mapping instruction for mapping the second three-dimensional object, wherein the mapping instruction is used to indicate that the material in the first material map needs to be replaced on the surface of the second three-dimensional object, and the first three-dimensional object and the second three-dimensional object are of different styles.

[0053] Specifically, the above-mentioned mapping instruction can be to trigger the device processor to execute the step of replacing the first material texture on the first three-dimensional object with the second three-dimensional object. For designers, the mapping instruction can be to extract the first material texture from the first image, move it to the model of the second three-dimensional object displayed on the interface by dragging, and the device processor receives the mapping instruction. For players of the dress-up game, after selecting the first material texture of the first image and the second three-dimensional object, select the dress-up control, and the above-mentioned mapping instruction can be sent to the device processor.

[0054] The first three-dimensional object and the second three-dimensional object are of different styles. Taking clothing as an example, the first three-dimensional object may be short-sleeved, and the second three-dimensional object may be long-sleeved, a skirt, or any other style of clothing; taking a vase as an example, the first three-dimensional object may be a cylindrical vase, and the second three-dimensional object may be a cone, a sphere, or any other vase different from the first three-dimensional object.

[0055] Step S25, in response to the mapping instruction, the first material map is migrated by using the material migration module to obtain a second material map of the second three-dimensional object, wherein the material migration module constructs a semantic migration relationship between three-dimensional objects of different styles.

[0056] Since the first three-dimensional object and the second three-dimensional object are objects of different styles, it is difficult to directly migrate the first material map to the second three-dimensional object. The above scheme constructs a semantic migration relationship between the first three-dimensional object and the second three-dimensional object to obtain a material migration model, and then uses the material migration model to migrate the first material map to obtain a second material map suitable for the second migration model.

[0057] In an optional embodiment, the first three-dimensional object and the second three-dimensional object may be structurally analyzed to obtain a rigid body transformation matrix between the first three-dimensional object and the second three-dimensional object, and then the material migration module may be obtained based on the rigid body transformation matrix between the two.

[0058] Step S27, rendering the second material map onto the second three-dimensional object.

[0059] In the above steps, by rendering the second material texture onto the second three-dimensional object, a similar design with the same material texture as the first three-dimensional object but different styles can be obtained. Still in the scene of clothing design, by rendering the second material texture onto the second three-dimensional object, new product designs of different styles can be obtained. For example, a designer can get inspiration from a show picture, extract the first material texture from a short-sleeved shirt in the show picture, migrate the first material texture through the material migration module to obtain the second material texture, and render the second material texture onto a long-sleeved shirt to obtain a design of a long-sleeved shirt of the same series as the short-sleeved shirt. Using a similar method, skirts, jackets, etc. with the same material texture can also be obtained, which greatly improves the efficiency of design. Based on the above scheme, users can semi-automatically specify the semantic migration relationship between object components of different styles to automatically migrate the material texture corresponding to the input object image to the material texture of another three-dimensional object model.

[0060] The above steps use the acquired second material map and combine it with a three-dimensional rendering engine to render the final texture migration result. In addition to the static model effect display, it is also possible to combine physical simulation, cloth animation and other technologies to render a digital fashion show video, allowing designers to experience the effect of texture design in all directions. Therefore, this solution can be applied to three-dimensional clothing design and virtual try-on scenes, especially in the field of interactive games. Users can migrate real clothing textures to the clothing models of three-dimensional virtual characters to generate customized clothing effects, which greatly enhances interactivity and fun.

[0061] In the above embodiment of the present application, a first material map of a first three-dimensional object is extracted from a first image; a mapping instruction for mapping a second three-dimensional object is received, wherein the mapping instruction is used to indicate that a material in the first material map needs to be replaced on the surface of the second three-dimensional object, and the first three-dimensional object and the second three-dimensional object are of different styles; in response to the mapping instruction, the first material map is migrated using a material migration module to obtain a second material map of the second three-dimensional object, wherein the material migration module constructs a semantic migration relationship between three-dimensional objects of different styles; and the second material map is rendered onto the second three-dimensional object. The above scheme obtains the texture of the object from the image, restores the material map of the three-dimensional model under the same style, and then sends it to the material migration module to obtain the material map suitable for objects of other styles, and then renders based on the newly generated material map, so as to complete the migration of materials between objects of different styles, thereby realizing the migration of textures between objects of different styles, and solves the problem in the prior art that in the process of designing the three-dimensional image of an object, the material of the two-dimensional image can be used to generate the corresponding three-dimensional model map through a deep learning network, and then the three-dimensional model map is re-rendered back into the three-dimensional model of the object. Since the above-mentioned material migration is limited to the migration between objects of the same style, the technical problem of low efficiency of the material migration process is caused.

[0062] As an optional embodiment, the semantic transfer relationship is used to represent the rigid geometric transformation between image regions located on three-dimensional objects of different styles.

[0063] Specifically, the above migration model can be obtained by constructing semantic migration relationships between objects of different styles and obtaining rigid transformation matrices between objects of different styles. This relationship can be converted into rigid geometric transformations between different image regions in the form of material maps.

[0064] For the migration of different styles of material maps, we can first use a computer graphics rendering algorithm to generate two clothing style material maps with the same texture, and then send them to the image migration network for training, so that we can get the material texture migration network model between the two clothing models, thus forming an end-to-end migration solution. However, due to the large shape differences between different materials and the complexity and diversity of clothing textures, it is very difficult to directly train such an image migration network, and the effect is not satisfactory. The above solution is based on the semantically specified model migration model, which can achieve better migration results.

[0065] As an optional embodiment, extracting a first material map of a first three-dimensional object from a first image includes: segmenting the first image to obtain an object image and a segmentation mask of the first three-dimensional object in the first image; processing the segmentation mask using a material migration network model to generate a material pixel coordinate map on the first three-dimensional object; based on the material pixel coordinate map on the first three-dimensional object, collecting the material of the first three-dimensional object to generate a first material map, wherein the shape of the material pixel coordinate map is the same as the shape of the first material map.

[0066] Figure 3 This is a schematic diagram of extracting a material map according to Embodiment 1 of the present application. Figure 3 , detailed description of extracting the first material map:

[0067] The first image may be segmented using an image segmentation model to segment an object image and a segmentation mask of the first three-dimensional object from the first image. In an optional embodiment, a large number of clothing images segmented from images containing clothing and segmentation masks may be used as training samples to train UNet to obtain the above segmentation model. Figure 3 As shown, the clothing is segmented from a single clothing image I through the image segmentation model to obtain the segmented clothing I mask And segmentation mask Mask I .

[0068] The material migration network model can be a texture migration network based on UNet+ResNet. Based on the material migration network model, a material pixel coordinate map on the first three-dimensional object can be generated according to the segmentation mask. Figure 3 As shown in Figure 2, a texture migration network based on UNet+ResNet can be constructed to transform Mask coord Send it into the network to generate the corresponding UV pixel coordinate map G_UV coord .

[0069] Since the shape of the material pixel coordinate map is the same as the shape of the first material map, the material pixel coordinate map of the first three-dimensional object can be extracted from the first graphic based on the material pixel coordinate map. Figure 3 In the example, G_UV coord The shape of is the UV map shape corresponding to the clothing 3D model, where the coordinate value stored in each pixel is I mask The pixel normalized coordinates in G_UV. That is, coord From I mask Perform sampling to obtain the corresponding UV map G_UV pix .

[0070] As an optional embodiment, before using the material migration network model to process the segmentation mask, the above method also includes: normalizing the coordinates of each pixel on the first image to obtain the image coordinates of the first image; and filling the image coordinates of the first image into the segmentation mask of the object image.

[0071] In an optional embodiment, after obtaining the object image and the segmentation mask, Figure 3 As shown, the segmentation mask Mask can be I Fill in the coordinate values ​​to get Mask coord For example, the image coordinates can be normalized and filled into the segmentation mask part. The above coordinate values ​​can be the coordinate values ​​of two channels, but even if the third channel is pre-filled, it does not affect the network training.

[0072] As an optional embodiment, before using the material migration network model to process the segmentation mask to generate a material pixel coordinate map on the first three-dimensional object, the above method also includes: constructing a migration network model, which step includes: obtaining training data, wherein the training data is three-dimensional image samples and material map samples obtained by processing based on a computer graphics rendering algorithm; using a neural network model to train the training data to generate a material migration model.

[0073] Specifically, the training data can be obtained through a computer graphics rendering algorithm, and still combined with Figure 3 As shown, according to the acquired clothing 3D model and the corresponding original clothing texture material, an offline rendering algorithm (such as ray tracing) is used to generate a realistic clothing image and a corresponding UV map, including the corresponding G_UV coord and G_UV pix Real data Real_UV coord and Real_UV pix . Where G_UV coord and Real_UV coord The L2-loss function (least square error loss function) can be used between G_UV pix and Real_UV pix The L1-loss function (minimum absolute deviation loss function) and the perceptual loss function can be used for training. By using the above training data for training, a mature texture migration module can be obtained.

[0074] The existing technology may use a solution that directly fits the material map from the product image, but this solution has the problem of low material resolution and long data generation time. The above solution uses a computer graphics rendering algorithm to generate a large amount of clothing training data through clothing 3D models and original texture materials, including corresponding UV maps, synthesized product images, etc. Using the rendering synthesis solution, clearer training data can be quickly generated.

[0075] As an optional embodiment, after generating the first material map, the method further includes: performing material completion on the first material map.

[0076] After the first material map is obtained through the material migration model, the first material map needs to be completed because the training data itself may have a texture missing problem.

[0077] In an optional embodiment of the above, still in combination with Figure 3 As shown, the UV map G_UV is generated through the texture migration module pix Finally, since the training data itself has texture missing problems, the texture completion module can be used to complete the final G_UV pix The texture completion module can be implemented by networks such as Image Inpainting, without the need to retrain and optimize the clothing training data, which improves the semantics of texture completion. Finally, the completed UV texture map G_UV is obtained. pix_comple .

[0078] The above scheme extracts the clothing texture UV map on the first three-dimensional object through clothing segmentation, texture migration, and texture completion modules. The difference between this scheme and the pix2surf scheme is that this scheme generates corresponding training data through computer graphics rendering algorithms, and adds a texture completion module in the last step to complete the missing parts of the UV map, reducing the texture loss caused by data generation and network training, making the final result more complete.

[0079] As an optional embodiment, before using the material migration module to migrate the first material map to obtain the second material map of the second three-dimensional object, the above method also includes: performing semantic segmentation on the first three-dimensional object and the second three-dimensional object respectively to obtain a first component set of the first three-dimensional object and a second component set of the second three-dimensional object, wherein the component set is composed of block bounding boxes of the three-dimensional objects; obtaining a rigid body enclosing matrix between the first component set and the second component set, the rigid body enclosing matrix records the semantic migration relationship between the block bounding boxes in the first component set and the block bounding boxes in the second component set; based on the rigid body enclosing matrix between the first component set and the second component set, generating a material migration module, wherein the material migration module records the semantic migration relationship matrix between each component in the first component set and the second component set.

[0080] In the above scheme, semantic segmentation is performed on the first three-dimensional object and the second three-dimensional object respectively, and different semantic parts of the two three-dimensional objects can be segmented. Taking clothing as an example, different parts of clothing can be segmented, such as the left sleeve, the right sleeve, the collar, the body, etc. The block bounding box can be used to enclose 0BB (oriented bounding box) of different semantic areas. By specifying the corresponding relationship between the block bounding boxes of two three-dimensional objects, a rigid body bounding matrix between the two objects can be established.

[0081] Based on the migration model generated in this step, by giving two different styles of three-dimensional clothing models, the semantic migration relationship of their material maps can be semi-automatically specified, and the material map G_UV generated in the above embodiment can be pix_comple , after migration, we get the material maps Trans_UV of different styles of clothing pix_final .

[0082] Figure 4 is a schematic diagram of a material migration according to Example 1 of the present application, combined with Figure 4 As shown in the example, still taking clothing and UV mapping as an example, there are two clothing 3D models. The two clothing 3D models are styled respectively to obtain the information of each part of the clothing. For example, sleeves, collars, etc. all have corresponding model mesh material information. The first 3D object is recorded as M src , and the second three-dimensional object is recorded as M dst In order to src The corresponding UV map is migrated to M dst On, M src and M dst Two UV maps UV are obtained through computer graphics rendering baking algorithm src and UV dst , where different blocks represent different clothing semantic segmentation information. For example, UVsrc The leftmost rectangular area in the figure represents the right sleeve. Through a simple image segmentation algorithm, the OBB of different semantic areas in the image is obtained, and the UV src Corresponding to OBB src_left_sleeve 、OBB src_front_body 、OBB src_right_sleeve ...UV dst Corresponding to OBB dstleft_sleeve 、OBB dst_frontbody 、OBB dst_right_sleeve ....... Tongno specifies the semantic migration relationship between OBBs between two models, such as UV src OBB src_leftsleeve Migrate to UV dst OBB dst_leftsleeve , so that the semantic migration relationship between different clothing components can be converted into the rigid body transformation matrix between two OBBs.

[0083] According to the semantic migration relationship between the specified OBBs, the corresponding rigid body transformation matrix can be calculated as follows: src_left_sleeve Migrate to OBB dst_left_sleeve , the corresponding translation, rotation and scaling matrix M can be calculated left_sleeve , where the scaling part can be scaled by width or height according to the semantic information. For example, if a short sleeve becomes a long sleeve, the height should be scaled. According to the above process, the transformation matrix M of the semantic transfer relationship specified by all clothing components is determined trans According to the transformation matrix M trans , you can add UV src Transform to the UV in the image above src_trans Similarly, for the G_UV generated in the first step pix_comple , can be migrated to Trans_UV through the above transformation scheme pix .

[0084] Since the UV map may have texture missing after migration, it is necessary to add a texture completion module to complete it. The network structure of this module can be consistent with the previous texture completion. Only the clothing data needs to be retrained to obtain the completed migration UV map Trans_UV pix_comple .

[0085] As an optional embodiment, before rendering the second material map onto the second three-dimensional object, the method further includes: removing light and shadow wrinkle information on the second material map.

[0086] Since the original input clothing image contains information such as light, shadow, and wrinkles, the second material map will also retain this information. However, for the final three-dimensional rendering, the light, shadow, and wrinkle information of the second material map will have a certain impact on the three-dimensional rendering effect, that is, the second material map should not contain this information.

[0087] The above solution adds an intrinsic image decomposition module to remove the light and shadow wrinkle information on the UV map. Figure 4 As shown, this scheme can use the traditional or learning-based intrinsic image decomposition module to obtain a new UV map Trans_UV pix_decomp Since the UV textures generated by the network generally have low resolution, a super-resolution module can be added at the end for super-resolution.

[0088] Due to the current limitations of deep learning networks, the generated images have low resolution, while the UV maps required for 3D rendering have high requirements for resolution and clarity, otherwise the final result will be very blurry. Here, we can use the common learning-based image super-resolution module and generate clothing UV map training data for retraining. Through this step, we can get the final high-definition UV map Trans_UV pix_final .

[0089] The above scheme of this embodiment addresses the defect that the texture migration scheme based on two-dimensional images requires original materials, and proposes an idea of ​​obtaining the texture of clothes from a single show picture, street shot picture, and product picture, and then combines the intrinsic image decomposition module to remove various redundant light and shadow wrinkle information in the image. The restrictions on user use are greatly reduced, and the design ideas of clothing designers are broadened, so that as long as there is a corresponding clothing picture, the corresponding texture migration can be performed. As long as there is a corresponding three-dimensional clothing model and the semantic information after segmentation, the user can arbitrarily specify the semantic migration relationship between clothing components, such as migrating the texture on the sleeve to the collar.

[0090] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0091] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention.

[0092] Example 2

[0093] According to an embodiment of the present invention, another embodiment of a material migration method is also provided. Figure 5 is a flow chart of another material migration method according to Embodiment 2 of the present application, combined with Figure 5 , the method comprising:

[0094] Step S51, displaying a first image and a second image in a display interface, wherein the object displayed in the first image is a first three-dimensional object, the object displayed in the second image is a second three-dimensional object, and the first three-dimensional object and the second three-dimensional object are of different styles.

[0095] The display interface may be an application interface for material migration. For example, when a designer migrates clothing materials, the display interface may be a clothing design interface. When a player migrates clothing materials in a dress-up game, the display interface may be a game interface.

[0096] The first three-dimensional object and the second three-dimensional object are of different styles. Taking clothing as an example, the first three-dimensional object may be short-sleeved, and the second three-dimensional object may be long-sleeved, a skirt, or any other style of clothing; taking a vase as an example, the first three-dimensional object may be a cylindrical vase, and the second three-dimensional object may be a cone, a sphere, or any other vase different from the first three-dimensional object.

[0097] The first image may be a two-dimensional image including the first three-dimensional object. In an optional embodiment, taking clothing design as an example, the first image may be a show picture or a street shot picture. After the designer gets design inspiration from the show or street shot, the show picture or street shot picture may be used as the first image, and the material texture of the clothing may be extracted from the show picture or street shot picture.

[0098] The first image may also be a three-dimensional image including the first three-dimensional object. In an optional embodiment, taking a three-dimensional dress-up game as an example, the first image may be a three-dimensional model of a garment that already exists in the game, and when the game player needs to use the material of the garment to dress up, the material texture of the garment may be extracted from the first image.

[0099] Step S53: Select the first image, and extract the material of the first three-dimensional object from the first image to obtain a first material map.

[0100] Specifically, extracting the first material map from the first image may be obtained by segmenting the first image.

[0101] Step S55, receiving a mapping instruction for mapping the second three-dimensional object, wherein the mapping instruction is used to indicate that the material in the first material map needs to be replaced onto the surface of the second three-dimensional object.

[0102] Specifically, the above-mentioned mapping instruction can be to trigger the device processor to execute the step of replacing the first material texture on the first three-dimensional object with the second three-dimensional object. For designers, the mapping instruction can be to extract the first material texture from the first image, move it to the model of the second three-dimensional object displayed on the interface by dragging, and the device processor receives the mapping instruction. For players of the dress-up game, after selecting the first material texture of the first image and the second three-dimensional object, select the dress-up control, and the above-mentioned mapping instruction can be sent to the device processor.

[0103] It should be noted that, currently in order to carry out related designs based on the same material map, designers need to obtain the source file of the material map, while based on the solution of the present application, only the first image containing the first three-dimensional object needs to be obtained, such as show photos and street photos, thereby greatly reducing the difficulty of obtaining materials and improving the efficiency of finding materials.

[0104] Step S57, responding to the mapping instruction, using the material migration module to migrate the first material map to obtain a second material map of the second three-dimensional object, wherein the material migration module constructs a semantic migration relationship between three-dimensional objects of different styles.

[0105] Since the first three-dimensional object and the second three-dimensional object are objects of different styles, it is difficult to directly migrate the first material map to the second three-dimensional object. The above scheme constructs a semantic migration relationship between the first three-dimensional object and the second three-dimensional object to obtain a material migration model, and then uses the material migration model to migrate the first material map to obtain a second material map suitable for the second migration model.

[0106] In an optional embodiment, the first three-dimensional object and the second three-dimensional object may be structurally analyzed to obtain a rigid body transformation matrix between the first three-dimensional object and the second three-dimensional object, and then the material migration module may be obtained based on the rigid body transformation matrix between the two.

[0107] Step S59: displaying a second image rendered by the second material map onto the second three-dimensional object.

[0108] In the above steps, by rendering the second material texture onto the second three-dimensional object, a similar design with the same material texture as the first three-dimensional object but different styles can be obtained. Still in the scene of clothing design, by rendering the second material texture onto the second three-dimensional object, new product designs of different styles can be obtained. For example, a designer can get inspiration from a show picture, extract the first material texture from a short-sleeved shirt in the show picture, migrate the first material texture through the material migration module to obtain the second material texture, and render the second material texture onto a long-sleeved shirt to obtain a design of a long-sleeved shirt of the same series as the short-sleeved shirt. Using a similar method, skirts, jackets, etc. with the same material texture can also be obtained, which greatly improves the efficiency of design. Based on the above scheme, users can semi-automatically specify the semantic migration relationship between object components of different styles to automatically migrate the material texture corresponding to the input object image to the material texture of another three-dimensional object model.

[0109] The above steps use the acquired second material map and combine it with a three-dimensional rendering engine to render the final texture migration result. In addition to the static model effect display, it is also possible to combine physical simulation, cloth animation and other technologies to render a digital fashion show video, allowing designers to experience the effect of texture design in all directions. Therefore, this solution can be applied to three-dimensional clothing design and virtual try-on scenes, especially in the field of interactive games. Users can migrate real clothing textures to the clothing models of three-dimensional virtual characters to generate customized clothing effects, which greatly enhances interactivity and fun.

[0110] In the above embodiment of the present application, a first image and a second image are displayed in a display interface, wherein the object displayed in the first image is a first three-dimensional object, and the object displayed in the second image is a second three-dimensional object, and the first three-dimensional object and the second three-dimensional object are of different styles; the first image is selected, and the material of the first three-dimensional object is extracted from the first image to obtain a first material map; a mapping instruction for mapping the second three-dimensional object is received, wherein the mapping instruction is used to indicate that the material in the first material map needs to be replaced on the surface of the second three-dimensional object; in response to the mapping instruction, the first material map is migrated using a material migration module to obtain a second material map of the second three-dimensional object, wherein the material migration module constructs a semantic migration relationship between three-dimensional objects of different styles; and a second image in which the second material map is rendered on the second three-dimensional object is displayed. The above scheme obtains the texture of the object from the image, restores the material map of the three-dimensional model under the same style, and then sends it to the material migration module to obtain the material map suitable for objects of other styles, and then renders based on the newly generated material map, so as to complete the migration of materials between objects of different styles, thereby realizing the migration of textures between objects of different styles, and solves the problem in the prior art that in the process of designing the three-dimensional image of an object, the material of the two-dimensional image can be used to generate the corresponding three-dimensional model map through a deep learning network, and then the three-dimensional model map is re-rendered back into the three-dimensional model of the object. Since the above-mentioned material migration is limited to the migration between objects of the same style, the technical problem of low efficiency of the material migration process is caused.

[0111] Example 3

[0112] According to an embodiment of the present application, a material migration device for implementing the material migration method in the above-mentioned embodiment 1 is also provided. Figure 6 is a schematic diagram of a material migration device according to Embodiment 3 of the present application, such as Figure 6 As shown, the device 600 includes:

[0113] An extraction module 602 is used to extract a first material map of a first three-dimensional object from a first image;

[0114] A trigger module 604 is used to receive a mapping instruction for mapping the second three-dimensional object, wherein the mapping instruction is used to indicate that the material in the first material map needs to be replaced on the surface of the second three-dimensional object, and the first three-dimensional object and the second three-dimensional object are different in style;

[0115] A response module 606 is used to respond to the mapping instruction and use the material migration module to migrate the first material map to obtain a second material map of the second three-dimensional object, wherein the material migration module constructs a semantic migration relationship between three-dimensional objects of different styles;

[0116] The rendering module 608 is used to render the second material map onto the second three-dimensional object.

[0117] It should be noted that the above extraction module 602, trigger module 604, response module 606 and rendering module 608 correspond to steps S21 to S27 in Example 1, and the four modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the contents disclosed in the above Example 1. It should be noted that the above modules, as part of the apparatus, can be run in the computing device 10 provided in Example 1.

[0118] As an optional embodiment, the semantic transfer relationship is used to represent the rigid geometric transformation between image regions located on three-dimensional objects of different styles.

[0119] As an optional embodiment, the extraction module includes: a segmentation submodule, used to segment the first image, obtain the object image and segmentation mask of the first three-dimensional object in the first image; a generation submodule, used to process the segmentation mask using a material migration network model, and generate a material pixel coordinate map on the first three-dimensional object; a collection submodule, used to collect the material of the first three-dimensional object based on the material pixel coordinate map on the first three-dimensional object, and generate a first material map, wherein the shape of the material pixel coordinate map is the same as the shape of the first material map.

[0120] As an optional embodiment, the above-mentioned device also includes: a normalization module, which is used to normalize the coordinates of each pixel on the first image before using the material migration network model to process the segmentation mask to obtain the image coordinates of the first image; and a filling module, which is used to fill the image coordinates of the first image into the segmentation mask of the object image.

[0121] As an optional embodiment, the above-mentioned device also includes: a construction module, which is used to construct a migration network model before using the material migration network model to process the segmentation mask and generate a material pixel coordinate map on the first three-dimensional object; a first acquisition module, which is used to obtain training data, wherein the training data is a three-dimensional image sample and a material map sample obtained by processing based on a computer graphics rendering algorithm; and a first generation module, which is used to train the training data using a neural network model to generate a material migration model.

[0122] As an optional embodiment, the above-mentioned device further includes: a completion module, which is used to complete the first material map after generating the first material map.

[0123] As an optional embodiment, the above-mentioned device also includes: a semantic segmentation module, which is used to perform semantic segmentation on the first three-dimensional object and the second three-dimensional object respectively before using the material migration module to migrate the first material map to obtain the second material map of the second three-dimensional object, so as to obtain a first component set of the first three-dimensional object and a second component set of the second three-dimensional object, wherein the component set is composed of block bounding boxes of the three-dimensional objects; a second acquisition module, which is used to obtain a rigid body enclosing matrix between the first component set and the second component set, and the rigid body enclosing matrix records the semantic migration relationship between the block bounding boxes in the first component set and the block bounding boxes in the second component set; a second generation module, which is used to generate a material migration module based on the rigid body enclosing matrix between the first component set and the second component set, wherein the material migration module records the semantic migration relationship matrix between each component in the first component set and the second component set.

[0124] As an optional embodiment, the above-mentioned device also includes: a removal module, which is used to remove light and shadow wrinkle information on the second material map before rendering the second material map onto the second three-dimensional object.

[0125] Example 4

[0126] According to an embodiment of the present invention, there is also provided a material migration device for implementing the material migration method in the above-mentioned embodiment 2. Figure 7 is a schematic diagram of a material migration device according to Embodiment 4 of the present application, such as Figure 7 As shown, the device 700 includes:

[0127] A first display module 702, configured to display a first image and a second image in a display interface, wherein the object displayed in the first image is a first three-dimensional object, and the object displayed in the second image is a second three-dimensional object, and the first three-dimensional object and the second three-dimensional object are of different styles;

[0128] An extraction module 704 is used to select the first image and extract the material of the first three-dimensional object from the first image to obtain a first material map;

[0129] A trigger module 706 is used to receive a mapping instruction for mapping the second three-dimensional object, wherein the mapping instruction is used to indicate that the material in the first material map needs to be replaced on the surface of the second three-dimensional object;

[0130] A response module 708 is used to respond to the mapping instruction and use the material migration module to migrate the first material map to obtain a second material map of the second three-dimensional object, wherein the material migration module constructs a semantic migration relationship between three-dimensional objects of different styles;

[0131] The second display module 7010 is used to display a second image rendered by a second material map onto a second three-dimensional object.

[0132] It should be noted that the first display module 702, the extraction module 704, the trigger module 706, the response module 708 and the second display module 7010 correspond to steps S51 to S59 in Example 2, and the five modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned embodiment 1. It should be noted that the above-mentioned modules as part of the apparatus can be run in the computing device 10 provided in the embodiment 1.

[0133] Example 5

[0134] The embodiment of the present invention may provide a computing device, which may be any computing device in a computing device group. Optionally, in this embodiment, the computing device may also be replaced by a terminal device such as a mobile terminal.

[0135] Optionally, in this embodiment, the computing device may be located in at least one network device among a plurality of network devices of a computer network.

[0136] In this embodiment, the above-mentioned computing device can execute the program code of the following steps in the material migration method: extracting a first material map of a first three-dimensional object from a first image; receiving a mapping instruction for mapping a second three-dimensional object, wherein the mapping instruction is used to indicate that the material in the first material map needs to be replaced on the surface of the second three-dimensional object, and the first three-dimensional object and the second three-dimensional object are of different styles; in response to the mapping instruction, using a material migration module to migrate the first material map to obtain a second material map of the second three-dimensional object, wherein the material migration module constructs a semantic migration relationship between three-dimensional objects of different styles; and rendering the second material map onto the second three-dimensional object.

[0137] Optionally, Figure 8 is a structural block diagram of a computing device according to an embodiment of the present invention. Figure 8 As shown, the computing device A may include: one or more (only one is shown in the figure) processors 802 , a memory 804 , and a peripheral interface 806 .

[0138] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the material migration method and device and the data processing method in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned material migration method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0139] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: extracting a first material map of a first three-dimensional object from a first image; receiving a mapping instruction for mapping a second three-dimensional object, wherein the mapping instruction is used to indicate that the material in the first material map needs to be replaced on the surface of the second three-dimensional object, and the first three-dimensional object and the second three-dimensional object are of different styles; in response to the mapping instruction, using a material migration module to migrate the first material map to obtain a second material map of the second three-dimensional object, wherein the material migration module constructs a semantic migration relationship between three-dimensional objects of different styles; and rendering the second material map onto the second three-dimensional object.

[0140] Optionally, the semantic transfer relation is used to characterize the rigid geometric transformation between image regions located on three-dimensional objects of different styles.

[0141] Optionally, the processor may also execute program code of the following steps: extracting a first material map of a first three-dimensional object from a first image, including: segmenting the first image, obtaining an object image and a segmentation mask of the first three-dimensional object in the first image; processing the segmentation mask using a material migration network model to generate a material pixel coordinate map on the first three-dimensional object; based on the material pixel coordinate map on the first three-dimensional object, collecting the material of the first three-dimensional object to generate a first material map, wherein the shape of the material pixel coordinate map is the same as the shape of the first material map.

[0142] Optionally, the processor may also execute the program code of the following steps: before processing the segmentation mask using the material migration network model, normalizing the coordinates of each pixel on the first image to obtain the image coordinates of the first image; and filling the image coordinates of the first image into the segmentation mask of the object image.

[0143] Optionally, the processor may also execute the program code of the following steps: before using the material migration network model to process the segmentation mask and generate a material pixel coordinate map on the first three-dimensional object, construct a migration network model, the step comprising: obtaining training data, wherein the training data is three-dimensional image samples and material map samples obtained based on a computer graphics rendering algorithm; and using a neural network model to train the training data to generate a material migration model.

[0144] Optionally, the processor may also execute program code of the following steps: after generating the first material map, performing material completion on the first material map.

[0145] Optionally, the processor may also execute program code for the following steps: before using the material migration module to migrate the first material map to obtain the second material map of the second three-dimensional object, semantically segment the first three-dimensional object and the second three-dimensional object to obtain a first component set of the first three-dimensional object and a second component set of the second three-dimensional object, wherein the component set is composed of block bounding boxes of the three-dimensional objects; obtaining a rigid body enclosing matrix between the first component set and the second component set, wherein the rigid body enclosing matrix records the semantic migration relationship between the block bounding boxes in the first component set and the block bounding boxes in the second component set; generating a material migration module based on the rigid body enclosing matrix between the first component set and the second component set, wherein the material migration module records the semantic migration relationship matrix between each component in the first component set and the second component set.

[0146] Optionally, the processor may also execute program code of the following steps: before rendering the second material map onto the second three-dimensional object, removing light and shadow wrinkle information on the second material map.

[0147] According to an embodiment of the present invention, a material migration method is provided. The method comprises extracting a first material map of a first three-dimensional object from a first image; receiving a mapping instruction for mapping a second three-dimensional object, wherein the mapping instruction is used to indicate that the material in the first material map needs to be replaced on the surface of the second three-dimensional object, and the first three-dimensional object and the second three-dimensional object are of different styles; responding to the mapping instruction, using a material migration module to migrate the first material map to obtain a second material map of the second three-dimensional object, wherein the material migration module constructs a semantic migration relationship between three-dimensional objects of different styles; and rendering the second material map onto the second three-dimensional object. The above scheme obtains the texture of the object from the image, restores the material map of the three-dimensional model under the same style, and then sends it to the material migration module to obtain the material map suitable for objects of other styles, and then renders based on the newly generated material map, so as to complete the migration of materials between objects of different styles, thereby realizing the migration of textures between objects of different styles, and solves the problem in the prior art that in the process of designing the three-dimensional image of an object, the material of the two-dimensional image can be used to generate the corresponding three-dimensional model map through a deep learning network, and then the three-dimensional model map is re-rendered back into the three-dimensional model of the object. Since the above-mentioned material migration is limited to the migration between objects of the same style, the technical problem of low efficiency of the material migration process is caused.

[0148] It can be understood by those skilled in the art that Figure 8 The structure shown is for illustration only, and the computing device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Figure 8 The structure of the electronic device is not limited. Figure 8 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 8 Different configurations are shown.

[0149] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0150] Example 6

[0151] The embodiment of the present invention further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the material migration method provided in the first embodiment.

[0152] Optionally, in this embodiment, the above storage medium may be located in any one of the computing devices in the computing device group in the computer network, or in any one of the mobile terminals in the mobile terminal group.

[0153] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: extracting a first material map of a first three-dimensional object from a first image; receiving a mapping instruction for mapping a second three-dimensional object, wherein the mapping instruction is used to indicate that a material in the first material map needs to be replaced onto a surface of a second three-dimensional object, and the first three-dimensional object and the second three-dimensional object are of different styles; in response to the mapping instruction, using a material migration module to migrate the first material map to obtain a second material map of the second three-dimensional object, wherein the material migration module constructs a semantic migration relationship between three-dimensional objects of different styles; and rendering the second material map onto the second three-dimensional object.

[0154] Example 7

[0155] According to an embodiment of the present invention, an embodiment of a data processing method is also provided. Fig. 9 is a flow chart of a data processing method according to Embodiment 7 of the present application, combined with Fig. 9 , the method comprising:

[0156] Step S91, acquiring a first image.

[0157] The first image may be an image containing the first three-dimensional object. It should be noted that, currently, in order to carry out related designs based on the same material texture, the designer needs to obtain the source file of the material texture, while based on the solution of the present application, only the first image containing the first three-dimensional object, such as a show picture or a street shot picture, is required, thereby greatly reducing the difficulty of obtaining materials and improving the efficiency of finding materials.

[0158] Step S93: extracting a first material map of a first three-dimensional object from the first image.

[0159] Specifically, the first material map may be a texture map on a first three-dimensional object. The first three-dimensional object may be an object from which material is to be extracted.

[0160] In an optional embodiment, the first image can be segmented to obtain an object image and a segmentation mask of a first three-dimensional object in the first image; the segmentation mask is processed using a material migration network model to generate a material pixel coordinate map on the first three-dimensional object; based on the material pixel coordinate map on the first three-dimensional object, the material of the first three-dimensional object is collected to generate a first material map, wherein the shape of the material pixel coordinate map is the same as the shape of the first material map.

[0161] Step S95: based on the geometric transformation relationship between the first three-dimensional object and the second three-dimensional object, render the first material map onto the second three-dimensional object.

[0162] Since the first three-dimensional object and the second three-dimensional object are objects of different styles, it is difficult to directly migrate the first material map to the second three-dimensional object. Based on the geometric transformation relationship between the first three-dimensional object and the second three-dimensional object, the first material map can be rendered onto the second three-dimensional object.

[0163] In an optional embodiment, the structure of the first three-dimensional object and the second three-dimensional object can be analyzed to obtain a rigid body transformation matrix between the first three-dimensional object and the second three-dimensional object, and then the first material map can be transformed based on the rigid body transformation matrix between the two, and then the transformed first material map can be rendered onto the second three-dimensional object.

[0164] The above scheme obtains the texture of the object from the image, restores the material map of the three-dimensional model of the same style, and then renders the first material map onto the second three-dimensional object based on the geometric transformation relationship between the two three-dimensional objects, thereby completing the migration of materials between objects of different styles, thereby realizing the migration of textures between objects of different styles. It solves the problem in the prior art that in the process of designing a three-dimensional image of an object, the material of the two-dimensional image can be used to generate a corresponding three-dimensional model map through a deep learning network, and then the three-dimensional model map is re-rendered back into the three-dimensional model of the object. Since the migration of the above-mentioned materials is limited to the migration between objects of the same style, the technical problem of low efficiency of the material migration process is caused.

[0165] Example 8

[0166] According to an embodiment of the present invention, another embodiment of a data processing method is also provided. Fig.10 is a flowchart of another data processing method according to Embodiment 8 of the present application, combined with Fig.10 , the method comprising:

[0167] Step S101: display a first image and a second image.

[0168] The first image may be an image containing a first three-dimensional object. The second image may be an image containing the following second three-dimensional object, wherein, in this embodiment, the texture material in the first image is migrated to the second image. It should be noted that, currently, in order to carry out related designs based on the same material texture, the designer needs to obtain the source file of the material texture, but based on the solution of the present application, only the first image containing the first three-dimensional object is obtained, such as a show picture or a street shot picture, thereby greatly reducing the difficulty of obtaining materials and improving the efficiency of finding materials.

[0169] Step S103: receiving a material extraction instruction, and extracting a first material map of a first three-dimensional object from the first image according to the material extraction instruction.

[0170] Specifically, the first material map may be a texture map on a first three-dimensional object. The first three-dimensional object may be an object from which material is to be extracted.

[0171] In an optional embodiment, the first image can be segmented to obtain an object image and a segmentation mask of a first three-dimensional object in the first image; the segmentation mask is processed using a material migration network model to generate a material pixel coordinate map on the first three-dimensional object; based on the material pixel coordinate map on the first three-dimensional object, the material of the first three-dimensional object is collected to generate a first material map, wherein the shape of the material pixel coordinate map is the same as the shape of the first material map.

[0172] Step S105 , receiving a material migration instruction, and rendering the first material map onto the second three-dimensional object according to the material migration instruction based on a geometric transformation relationship between the first three-dimensional object and the second three-dimensional object in the second image.

[0173] Since the first three-dimensional object and the second three-dimensional object are objects of different styles, it is difficult to directly migrate the first material map to the second three-dimensional object. Based on the geometric transformation relationship between the first three-dimensional object and the second three-dimensional object, the first material map can be rendered onto the second three-dimensional object.

[0174] In an optional embodiment, the structure of the first three-dimensional object and the second three-dimensional object can be analyzed to obtain a rigid body transformation matrix between the first three-dimensional object and the second three-dimensional object, and then the first material map can be transformed based on the rigid body transformation matrix between the two, and then the transformed first material map can be rendered onto the second three-dimensional object.

[0175] The above scheme obtains the texture of the object from the image, restores the material map of the three-dimensional model of the same style, and then renders the first material map onto the second three-dimensional object based on the geometric transformation relationship between the two three-dimensional objects, thereby completing the migration of materials between objects of different styles, thereby realizing the migration of textures between objects of different styles. It solves the problem in the prior art that in the process of designing a three-dimensional image of an object, the material of the two-dimensional image can be used to generate a corresponding three-dimensional model map through a deep learning network, and then the three-dimensional model map is re-rendered back into the three-dimensional model of the object. Since the migration of the above-mentioned materials is limited to the migration between objects of the same style, the technical problem of low efficiency of the material migration process is caused.

[0176] Example 9

[0177] According to an embodiment of the present application, a data processing device for implementing the data processing method in the above-mentioned embodiment 7 is also provided. Fig.11 is a schematic diagram of a data processing device according to Embodiment 9 of the present application, such as Fig.11 As shown, the device 1100 includes:

[0178] The acquisition module 1102 is used to acquire a first image.

[0179] The extraction module 1104 is configured to extract a first material map of a first three-dimensional object from the first image.

[0180] The rendering module 1106 is configured to render the first material map onto the second three-dimensional object based on a geometric transformation relationship between the first three-dimensional object and the second three-dimensional object.

[0181] It should be noted that the acquisition module 1102, the extraction module 1104 and the rendering module 1106 correspond to steps S91 to S95 in Example 7, and the three modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the contents disclosed in the above-mentioned embodiment 1. It should be noted that the above-mentioned modules, as part of the apparatus, can be run in the computing device 10 provided in the embodiment 1.

[0182] Example 10

[0183] According to an embodiment of the present application, a data processing device for implementing the data processing method in the above-mentioned embodiment 8 is also provided. Fig.12 is a schematic diagram of a data processing device according to Embodiment 10 of the present application, such as Fig.12 As shown, the device 1200 includes:

[0184] The display module 1202 is configured to display the first image and the second image.

[0185] The first receiving module 1204 is configured to receive a material extraction instruction, and extract a first material map of a first three-dimensional object from the first image according to the material extraction instruction.

[0186] The second receiving module 1206 is used to receive a material migration instruction, and render the first material map onto the second three-dimensional object according to the material migration instruction based on the geometric transformation relationship between the first three-dimensional object and the second three-dimensional object in the second image.

[0187] It should be noted that the display module 1202, the first receiving module 1204 and the second receiving module 1206 correspond to steps S91 to S95 in Example 7, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned embodiment 1. It should be noted that the above-mentioned modules, as part of the apparatus, can be run in the computing device 10 provided in the embodiment 1.

[0188] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0189] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0190] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0191] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0192] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0193] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0194] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A material migration method, characterized in that: include: Extracting a first material map of a first three-dimensional object from the first image; Receiving a mapping instruction for mapping a second three-dimensional object, wherein the mapping instruction is used to indicate that a material in the first material map needs to be replaced on a surface of the second three-dimensional object, and the first three-dimensional object and the second three-dimensional object are different in style; In response to the mapping instruction, a material migration module is used to migrate the first material map to obtain a second material map of the second three-dimensional object, wherein the material migration module constructs a semantic migration relationship between three-dimensional objects of different styles, and the semantic migration relationship is used to characterize the rigid body geometric transformation between the three-dimensional objects of different styles, and the rigid body geometric transformation is obtained by converting the rigid body transformation matrix between the three-dimensional objects of different styles, and the rigid body transformation matrix is ​​obtained by performing structural analysis on the three-dimensional objects of different styles; Rendering the second material map onto the second three-dimensional object.

2. The method according to claim 1, characterized in that: The semantic transfer relationship is used to characterize the rigid geometric transformation between image regions located on three-dimensional objects of different styles.

3. The method according to claim 1, characterized in that: Extracting a first material map of a first three-dimensional object from a first image includes: segmenting the first image to obtain an object image and a segmentation mask of the first three-dimensional object in the first image; Processing the segmentation mask using a material migration network model to generate a material pixel coordinate map on the first three-dimensional object; Based on the material pixel coordinate map on the first three-dimensional object, the material of the first three-dimensional object is collected to generate the first material map, wherein the shape of the material pixel coordinate map is the same as the shape of the first material map.

4. The method according to claim 3, characterized in that Before processing the segmentation mask using the material migration network model, the method further includes: Normalizing the coordinates of each pixel on the first image to obtain image coordinates of the first image; The image coordinates of the first image are filled into the segmentation mask of the object image.

5. The method according to claim 3, characterized in that: Before using the material migration network model to process the segmentation mask to generate a material pixel coordinate map on the first three-dimensional object, the method further includes: Constructing the migration network model, this step includes: Acquire training data, wherein the training data is a three-dimensional image sample and a material texture sample obtained by processing based on a computer graphics rendering algorithm; The training data is trained using a neural network model to generate a material migration model.

6. The method according to claim 3, characterized in that After generating the first material map, the method further includes: performing material completion on the first material map.

7. The method according to claim 1, characterized in that Before adopting the material migration module to migrate the first material map to obtain the second material map of the second three-dimensional object, the method further includes: Performing semantic segmentation on the first three-dimensional object and the second three-dimensional object respectively to obtain a first component set of the first three-dimensional object and a second component set of the second three-dimensional object, wherein the component sets are composed of block bounding boxes of the three-dimensional objects; Acquire a rigid body enclosing matrix between the first component set and the second component set, wherein the rigid body enclosing matrix records a semantic migration relationship between a block enclosing box in the first component set and a block enclosing box in the second component set; The material migration module is generated based on the rigid enclosing matrix between the first component set and the second component set, wherein the material migration module records the semantic migration relationship matrix between each component in the first component set and the second component set.

8. The method according to claim 1, characterized in that: Before rendering the second material map onto the second three-dimensional object, the method further includes: removing light and shadow wrinkle information on the second material map.

9. A material migration method, characterized in that: include: Displaying a first image and a second image in a display interface, wherein the object displayed in the first image is a first three-dimensional object, the object displayed in the second image is a second three-dimensional object, and the first three-dimensional object and the second three-dimensional object are of different styles; Selecting the first image, and extracting the material of the first three-dimensional object from the first image to obtain a first material map; triggering a mapping instruction for mapping the second three-dimensional object, wherein the mapping instruction is used to indicate that the material in the first material map needs to be replaced onto the surface of the second three-dimensional object; In response to the mapping instruction, a material migration module is used to migrate the first material map to obtain a second material map of the second three-dimensional object, wherein the material migration module constructs a semantic migration relationship between three-dimensional objects of different styles, and the semantic migration relationship is used to characterize the rigid body geometric transformation between the three-dimensional objects of different styles, and the rigid body geometric transformation is obtained by converting the rigid body transformation matrix between the three-dimensional objects of different styles, and the rigid body transformation matrix is ​​obtained by performing structural analysis on the three-dimensional objects of different styles; A second image is displayed which is a result of rendering the second material map onto the second three-dimensional object.

10. A material migration device, characterized in that: include: An extraction module, used for extracting a first material map of a first three-dimensional object from a first image; a trigger module, configured to receive a mapping instruction for mapping a second three-dimensional object, wherein the mapping instruction is used to indicate that a material in the first material map needs to be replaced on a surface of the second three-dimensional object, and the first three-dimensional object and the second three-dimensional object are of different styles; a response module, used to respond to the mapping instruction, and use the material migration module to migrate the first material map to obtain a second material map of the second three-dimensional object, wherein the material migration module constructs a semantic migration relationship between three-dimensional objects of different styles, and the semantic migration relationship is used to characterize the rigid body geometric transformation between the three-dimensional objects of different styles, and the rigid body geometric transformation is obtained by converting the rigid body transformation matrix between the three-dimensional objects of different styles, and the rigid body transformation matrix is ​​obtained by performing structural analysis on the three-dimensional objects of different styles; A rendering module is used to render the second material map onto the second three-dimensional object.

11. A material migration device, characterized in that: include: A first display module, configured to display a first image and a second image in a display interface, wherein the object displayed in the first image is a first three-dimensional object, the object displayed in the second image is a second three-dimensional object, and the first three-dimensional object and the second three-dimensional object are of different styles; An extraction module, configured to select the first image and extract the material of the first three-dimensional object from the first image to obtain a first material map; A trigger module, used to trigger a mapping instruction for mapping the second three-dimensional object, wherein the mapping instruction is used to indicate that the material in the first material map needs to be replaced on the surface of the second three-dimensional object; a response module, used to respond to the mapping instruction, and use the material migration module to migrate the first material map to obtain a second material map of the second three-dimensional object, wherein the material migration module constructs a semantic migration relationship between three-dimensional objects of different styles, and the semantic migration relationship is used to characterize the rigid body geometric transformation between the three-dimensional objects of different styles, and the rigid body geometric transformation is obtained by converting the rigid body transformation matrix between the three-dimensional objects of different styles, and the rigid body transformation matrix is ​​obtained by performing structural analysis on the three-dimensional objects of different styles; A second display module is used to display a second image rendered by the second material map onto the second three-dimensional object.

12. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to perform the migration of the material according to any one of claims 1 to 9.

13. A processor, characterized in that: The processor is used to run a program, wherein the program executes the migration of the material described in any one of claims 1 to 9 when running.

14. A material migration system, comprising: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: Extracting a first material map of a first three-dimensional object from the first image; Receiving a mapping instruction for mapping a second three-dimensional object, wherein the mapping instruction is used to indicate that a material in the first material map needs to be replaced on a surface of the second three-dimensional object, and the first three-dimensional object and the second three-dimensional object are different in style; In response to the mapping instruction, a material migration module is used to migrate the first material map to obtain a second material map of the second three-dimensional object, wherein the material migration module constructs a semantic migration relationship between three-dimensional objects of different styles, and the semantic migration relationship is used to characterize the rigid body geometric transformation between the three-dimensional objects of different styles, and the rigid body geometric transformation is obtained by converting the rigid body transformation matrix between the three-dimensional objects of different styles, and the rigid body transformation matrix is ​​obtained by performing structural analysis on the three-dimensional objects of different styles; Rendering the second material map onto the second three-dimensional object.

15. A data processing method, characterized in that: include: acquiring a first image; Extracting a first material map of a first three-dimensional object from the first image; Based on a geometric transformation relationship between the first three-dimensional object and the second three-dimensional object, rendering the first material map onto the second three-dimensional object; Among them, based on the geometric transformation relationship between the first three-dimensional object and the second three-dimensional object, the first material map is rendered onto the second three-dimensional object, including: performing structural analysis on the first three-dimensional object and the second three-dimensional object to obtain a rigid body transformation matrix between the first three-dimensional object and the second three-dimensional object; transforming the first material map based on the rigid body transformation matrix, and rendering the transformed first material map onto the second three-dimensional object.

16. A data processing method, characterized in that: include: displaying a first image and a second image; receiving a material extraction instruction, and extracting a first material map of a first three-dimensional object from the first image according to the material extraction instruction; receiving a material migration instruction, and rendering the first material map onto the second three-dimensional object according to the material migration instruction based on a geometric transformation relationship between the first three-dimensional object and the second three-dimensional object in the second image; Among them, according to the material migration instruction and based on the geometric transformation relationship between the first three-dimensional object and the second three-dimensional object in the second image, the first material map is rendered onto the second three-dimensional object, including: performing structural analysis on the first three-dimensional object and the second three-dimensional object according to the material migration instruction to obtain a rigid body transformation matrix between the first three-dimensional object and the second three-dimensional object; transforming the first material map based on the rigid body transformation matrix, and rendering the transformed first material map onto the second three-dimensional object.

Images