A 3D image construction method, device and terminal based on style transfer

Abstract

The present application is applicable to the field of image processing technology, and provides a method, device and terminal for constructing 3D images based on style transfer, wherein the method comprises: obtaining a 2D original image containing an object to be processed; collecting the spatial point coordinates of the object to be processed from the original image, and obtaining a 3D position map of the object to be processed based on the spatial point coordinates; generating a 3D texture map of the object to be processed based on the position map and in combination with the pixel information in the original image; inputting the texture map into a style transfer network to generate a 3D target texture map of a target style; constructing a 3D target image corresponding to the original image based on the target texture map and the spatial point coordinates. The method can realize the construction of a 3D special effect image based on a 2D plane image while ensuring the fineness and fidelity of the image processing effect.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to a 3D image construction method, device and terminal based on style transfer. Background Art

[0002] Image effects, such as aging and juvenation of face images, or animation of photographed objects, have attracted more and more attention from researchers as a comprehensive cross-domain problem. It is usually defined as a rendering process that aims to render the target object with special effects such as aging and animation while preserving identity information.

[0003] Currently, most existing image special effects processing methods are aimed at special effects processing of 2D images, but little attention is paid to special effects processing of 3D images, which are more widely used. At the same time, since the number of 3D image samples is limited and difficult to collect, if 3D special effects images are constructed directly from 2D images, the processing effect is often poor. This makes it difficult for traditional 2D image processing technology to play a role in 3D application fields, and the implementation of 3D image special effects processing tasks is more challenging. Summary of the invention

[0004] The embodiments of the present application provide a 3D image construction method, device and terminal based on style transfer to solve the problem in the prior art that the number of 3D image samples is limited and the effect is poor when 3D special effect images are directly constructed through 2D images.

[0005] A first aspect of an embodiment of the present application provides a 3D image construction method based on style transfer, comprising:

[0006] Obtain a 2D original image containing the object to be processed;

[0007] Collecting spatial point coordinates of the object to be processed from the original image, and obtaining a 3D position map of the object to be processed based on the spatial point coordinates;

[0008] Based on the position map and in combination with pixel information in the original image, a 3D texture map of the object to be processed is generated;

[0009] Inputting the texture map into a style transfer network to generate a 3D target texture map of the target style;

[0010] Based on the target texture map and the spatial point coordinates, a 3D target image corresponding to the original image is constructed.

[0011] A second aspect of an embodiment of the present application provides a 3D image construction device based on style transfer, comprising:

[0012] A first acquisition module is used to acquire a 2D original image containing an object to be processed;

[0013] A second acquisition module is used to collect the spatial point coordinates of the object to be processed from the original image, and obtain a 3D position map of the object to be processed based on the spatial point coordinates;

[0014] A first generating module, configured to generate a 3D texture map of the object to be processed based on the position map and in combination with pixel information in the original image;

[0015] A second generation module is used to input the texture map into a style transfer network to generate a 3D target texture map of a target style;

[0016] An image construction module is used to construct a 3D target image corresponding to the original image based on the target texture map and the spatial point coordinates.

[0017] A third aspect of an embodiment of the present application provides a terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the computer program.

[0018] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0019] A fifth aspect of the present application provides a computer program product, which, when executed on a terminal, enables the terminal to execute the steps of the method described in the first aspect.

[0020] As can be seen from the above, in the embodiment of the present application, a position map and a texture map of the object to be processed are obtained based on the 2D original image processing, and the image style transfer is performed on the basis of the texture map to obtain a 3D target texture map of the target style, and finally the construction of a 3D target image is realized. This process retains more original structural features of the object to be processed while realizing image special effects processing, maximizes the retention of the identity information of the object to be processed, and realizes the construction of a 3D special effects image based on a 2D plane image while ensuring the fineness and realism of the image processing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 The process of the 3D image construction method based on style transfer provided in the embodiment of the present application is as follows: Figure 1 ;

[0023] Figure 2 It is a schematic diagram of the processing flow when processing a face image provided by an embodiment of the present application;

[0024] Figure 3 The process of the 3D image construction method based on style transfer provided in the embodiment of the present application is as follows: Figure 2 ;

[0025] Figure 4 is a structural diagram of a 3D image construction device based on style transfer provided in an embodiment of the present application;

[0026] Figure 5 It is a structural diagram of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0028] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0029] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0030] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0031] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0032] In a specific implementation, the terminal described in the embodiments of the present application includes, but is not limited to, other portable devices such as mobile phones, laptop computers, or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and / or touch pads). It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or touch pads).

[0033] In the following discussion, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse and / or joystick.

[0034] The terminal supports various applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disk burning application, a spreadsheet application, a game application, a telephone application, a video conferencing application, an email application, an instant messaging application, a workout support application, a photo management application, a digital camera application, a digital camcorder application, a web browsing application, a digital music player application, and / or a digital video player application.

[0035] Various applications that can be executed on the terminal can use at least one common physical user interface device such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the terminal can be adjusted and / or changed between applications and / or within corresponding applications. In this way, the common physical architecture of the terminal (e.g., the touch-sensitive surface) can support various applications with user interfaces that are intuitive and transparent to the user.

[0036] It should be understood that the size of the serial numbers of the steps in this embodiment does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0037] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.

[0038] See also Figure 1 , Figure 1 This is a process of a 3D image construction method based on style transfer provided in an embodiment of the present application. Figure 1 .like Figure 1 As shown, a 3D image construction method based on style transfer includes the following steps:

[0039] Step 101: Acquire a 2D original image containing an object to be processed.

[0040] The original image is the image to be processed, and the image to be processed includes an object to be processed. The object to be processed is specifically part or all of the image content in the original image.

[0041] For example, when the original image is an image containing a human face, the object to be processed is specifically the human face in the original image; when the original image is an image containing a driving road, the object to be processed is specifically the driving road in the original image; or when the original image is another type of image, the object to be processed is the set target object contained therein.

[0042] The original image is a 2D image, that is, a planar image. In the embodiment of the present application, the generation of a 3D image after special effect processing is realized based on the planar 2D image.

[0043] Step 102 : collecting the spatial point coordinates of the object to be processed from the original image, and obtaining a 3D position map of the object to be processed based on the spatial point coordinates.

[0044] Specifically, the spatial point coordinates may be the key point coordinates of the object to be processed, such as the pupil coordinates, mouth corners, nose tip and other position coordinates in a face image, or the edge point coordinates of the object to be processed, the contour point coordinates of the object to be processed, and so on.

[0045] The spatial point coordinates are used to identify the spatial position of the object to be processed.

[0046] After obtaining the spatial point coordinates of the object to be processed, the spatial position of the object to be processed can be known, and then a spatial position map corresponding to the object to be processed, that is, a 3D position map, can be obtained.

[0047] When obtaining a spatial position map based on the spatial point coordinates of the object to be processed, the two-dimensional coordinates of the object to be processed in the 2D original image and the depth information of the object to be processed itself can be used to obtain the three-dimensional coordinates of the spatial point of the object to be processed, and then obtain the spatial position map.

[0048] Alternatively, as an optional implementation, the spatial point coordinates of the object to be processed are collected from the original image, and a 3D position map of the object to be processed is obtained based on the spatial point coordinates, including:

[0049] The three-channel pixel values ​​of each pixel point in the object to be processed are obtained from the original image; based on the three-channel pixel values, the three-dimensional coordinates of the space point corresponding to the object to be processed are mapped; based on the three-dimensional coordinates of the space point, a 3D position map of the object to be processed is obtained.

[0050] That is, based on the three-channel pixel value (R, G, B) of each pixel point in the object to be processed, the three-dimensional coordinates of the space point corresponding to the image content of each pixel point in the object to be processed can be converted.

[0051] The generation process of the position map removes the background area in the original image and only processes the image content in the object to be processed.

[0052] The content included in the position map can be specifically formed by spreading out the three-dimensional components of the object to be processed based on the three-dimensional coordinates of each spatial point corresponding to the object to be processed. For example, based on the three-dimensional coordinates of each spatial point corresponding to the face, specifically based on the three-dimensional coordinates of the spatial points of the three-dimensional component of the nose, the image content of the nose wings on both sides of the nose is spread out. The image content of the posterior side of the forehead, ears and other parts, and the shadow of the chin can also be spread out based on the three-dimensional coordinates of the spatial points of the three-dimensional components such as the ears and chin. The spreading is implemented by estimating the coordinates of each three-dimensional component based on the spatial point coordinates of the object to be processed collected from the original image, and filling the estimated coordinate information into the corresponding position in the position map to obtain the 3D position map of the object to be processed, so that the image size of the 3D position map of the object to be processed is the same as that of the original image, thereby reducing the influence of background noise.

[0053] Therefore, the processed position map contains the three-dimensional position information of the spatial points in the object to be processed.

[0054] Step 103: Generate a 3D texture map of the object to be processed based on the position map and in combination with pixel information in the original image.

[0055] When generating a 3D texture map, it is necessary to match the pixel information of the image content corresponding to each spatial point coordinate from the pixel information in the original image based on the spatial point coordinates corresponding to each position point in the position map of the object to be processed.

[0056] For example, the position of the pupil of the face in the position map corresponds to a pupil space point coordinate, so the pixel information of the pupil in the face image part is matched from the pixel information in the original image; the position of the corner of the mouth of the face in the position map corresponds to the mouth corner space point coordinate, so the pixel information of the corner of the mouth in the face image part is matched from the pixel information in the original image.

[0057] The matched pixel information is used to fill pixels at corresponding positions in the position map, so as to obtain a 3D texture map of the object to be processed.

[0058] That is, the spatial position information of each part of the object to be processed is added to the color information in the original image to obtain a 3D texture map.

[0059] Correspondingly, as an optional implementation, based on the position map and in combination with pixel information in the original image, a 3D texture map of the object to be processed is generated, including:

[0060] From the original image, match the target position points corresponding to the position points in the position map; extract the first target pixel points at the target position points from the original image in turn; arrange the first target pixel points according to the position points in the position map to obtain a 3D texture map of the object to be processed.

[0061] Each position point in the position map has a corresponding position point in the original image. When generating a 3D texture map, it is necessary to match the corresponding position point, i.e., the target position point, from the original image based on each position point in the position map. The target position point is a point in the object to be processed in the original image.

[0062] After matching the target position point, the pixel points at each target position point can be obtained based on the pixel information in the original image. These pixel points contain pixel composition information. By extracting these pixel points and arranging them according to the position points in the position map, the 3D texture map of the object to be processed can be obtained.

[0063] The implementation process of the entire style transfer-based 3D image construction method can be achieved by constructing a global network model.

[0064] As an implementation method, the global network model includes: a 3D vertex and texture estimator. The implementation of the above steps 101 to 103 is realized by the 3D vertex and texture estimator module, which is responsible for estimating the spatial vertex coordinates and texture map of the object to be processed in the original image given a 2D original image as input. Specifically, the 3D vertex and texture estimator module can specifically include a 3D vertex estimator and a 2D & 3D rendering reconstructor. First, the 3D vertex estimator is used to estimate the image vertices, obtain the position map, and establish a mapping relationship between the 2D object to be processed and the 3D vertices. Then, the 2D & 3D rendering reconstructor acts as a texture estimator to estimate the 3D texture map based on the established mapping relationship, and the estimated texture map is fed into the style transfer network for subsequent image processing operations.

[0065] Step 104: input the texture map into the style transfer network to generate a 3D target texture map of the target style.

[0066] In this step, the generated 3D texture map corresponding to the object to be processed in the original image is used as the input of the style transfer network, so that the style transfer network performs style rendering migration on the basis of the texture map, and finally generates a 3D target texture map of the target style. On the basis of achieving style migration, the structural characteristics of the object to be processed are retained to the greatest extent. Compared with directly performing style rendering migration on the basis of the original plane image, this method will retain more identity information of the object to be processed, thereby ensuring the authenticity of the image effect after the final special effects processing.

[0067] Among them, the style transfer network is a part of the aforementioned global network model. The style transfer network is specifically a GAN (Generative Adversarial Networks) network, through which the image style evolution process is inferred. The style transfer network can be trained to have different style transfer functions, such as the style transfer function of face aging, the style transfer function of face rejuvenation, the comic style transfer function of the target object, etc. Correspondingly, the target style can be an old age style, a childhood style, a comic style, etc., which can be set based on actual application needs.

[0068] Among them, taking the style transfer of face aging as an example, face aging is essentially regarded as a style transfer problem or a domain adaptation problem, which aims to use aging effects to render young face images while retaining identity information.

[0069] In the specific implementation process, the style transfer of face aging requires the style transfer network to have the style transfer function of face aging. Here, the style transfer network is specifically set as a 3D face aging generator module, and the 3D face aging generator specifically adopts a GAN-based cyclic style cGAN (conditional generative adversarial network) model to implement face aging processing. The cGAN consists of two texture map generators responsible for aging estimation and two discriminators used to make the aging results difficult to distinguish, ensuring the authenticity of the style transfer effect.

[0070] In order to ensure the quality of the image generated after style transfer using the structural features of the object to be processed carried in the position map and the detailed texture features of the object to be processed carried in the texture map, in this embodiment, the attribute information and random noise of the object to be processed can also be added to the texture map generator to ensure the consistency of the features of the object to be processed after style transfer, while ensuring the diversity of special effect textures, so as to accurately identify the original image face while ensuring the differences in aging textures of different face images after style transfer. The attribute information of the object to be processed, for example, is the specific age, skin color, race, etc. corresponding to the face, the road type (such as rural roads, highways, etc.), road construction style, etc. corresponding to the road.

[0071] Step 105: construct a 3D target image corresponding to the original image based on the target texture map and the spatial point coordinates.

[0072] The spatial point coordinates in the position map indicate the structural features of the object to be processed, and the target texture map indicates the texture features of the object to be processed after style transfer. Based on the target texture map and the spatial point coordinates, the construction of a 3D target image containing the original image of the object to be processed is realized.

[0073] Specifically, other areas in the 3D target image except the object to be processed may still retain the image style in the original image, or be transformed into other image styles different from the style of the object to be processed after style transfer processing through other processing processes, or only the 3D image part corresponding to the object to be processed may be retained, and no specific limitation is made here.

[0074] As an optional implementation, the 3D target image corresponding to the original image is constructed based on the target texture map and the spatial point coordinates, including:

[0075] The spatial point coordinates are arranged to form a target number of three-point pairs, wherein each three-point pair forms a triangle, and the target number of triangles forms a 3D surface of the target image; based on the correspondence between the spatial point coordinates and each pixel point in the target texture map, the pixel value of the corresponding pixel point is extracted from the target texture map, and the pixel value is assigned to the vertex position of the triangle; the relative distances between the remaining position points in each triangle and the three vertices of the triangle are calculated; based on the relative distances, the weight values ​​of the pixel values ​​are set; the target pixel values ​​of the remaining position points in each triangle are calculated respectively in combination with the pixel values ​​assigned to the vertex positions of the triangle and the weight values ​​of the pixel values; the target pixel values ​​are assigned to the corresponding remaining position points to obtain a 3D target image corresponding to the original image.

[0076] That is, the coordinates of the spatial points are divided into three pairs of points, and the triangles formed by each three-point pair are arranged to construct a 3D surface. The corresponding pixel points in the target texture map obtained by style transfer are assigned to the vertex positions of the triangles. The pixel values ​​in the triangle are assigned based on the pixel values ​​of the three vertices of the triangle. Specifically, the value weights are assigned according to the relative distances between each position point in the triangle and the three vertices. The pixel values ​​of each position point are calculated based on the value weights and the pixel values ​​of the three vertices of the triangle to which it belongs, so as to determine the values ​​of each pixel point on the 3D surface, assign pixels on the constructed 3D surface, and realize the texture construction of the 3D surface. At this point, the final 3D face image after style transfer can be obtained.

[0077] In a specific face image processing process, combined with Figure 2 As shown, the implementation process of the above steps is:

[0078] First, a young face image is input as an input image to the E network (i.e., a 3D vertex and texture estimator). After convolution processing, the position point information pi (x, y, z) of the face part in the face image is obtained to obtain a position map. Based on the position map and the pixel information in the original image, a 3D texture map is generated by a 2D&3D rendering reconstructor. The 3D texture map is used as the input of the G network (i.e., a 3D face aging generator). The 2D&3D rendering reconstructor here plays the role of bridging the vertex estimation model and the 3D face aging generator, so that the loss calculated in the generative adversarial network (including the G network and the D network) corresponding to the 3D face aging generator can be propagated to the E network, thereby achieving joint optimization of the two and realizing end-to-end learning of the entire model. After the texture map is input into the generative adversarial network, the style transfer of face aging can be realized, so that the texture of the face part changes from a young texture to an old texture, and an aging texture map is obtained. Based on the aging texture map, the spatial point coordinates in the position map and the triangles constructed by the three-point pairs are combined to render a 3D face image.

[0079] Specifically, when constructing the 3D target image, a 2D&3D rendering reconstructor may be used to achieve rendering and reconstruction of the final 3D face image.

[0080] In an embodiment of the present application, through the above-mentioned implementation steps, a 3D special effects image is constructed based on a 2D plane image. Specifically, a position map and a texture map of the object to be processed are obtained based on the 2D original image processing, and an image style transfer is performed based on the texture map to obtain a 3D target texture map of the target style, and finally the construction of a 3D target image is realized. This process retains more original structural features of the object to be processed while realizing image special effects processing, maximizes the retention of the identity information of the object to be processed, and realizes the construction of a 3D special effects image based on a 2D plane image while ensuring the fineness and realism of the image processing effect.

[0081] The embodiments of the present application also provide different implementations of a 3D image construction method based on style transfer.

[0082] See also Figure 3 , Figure 3 This is a process of a 3D image construction method based on style transfer provided in an embodiment of the present application. Figure 2 .like Figure 3 As shown, a 3D image construction method based on style transfer includes the following steps:

[0083] Step 301: Acquire a preset reference image.

[0084] The reference image is specifically a standard image that meets set requirements, and the reference image is used as a reference for preprocessing the sample image.

[0085] When acquiring the reference image, the set reference image may be directly read from a database, or a pre-set reference image uploaded by a user may be acquired.

[0086] Step 302: Detect image key points from a 2D sample image containing the object to be processed.

[0087] The key points of the image are, for example, key position points of the pupil, mouth corner, brow, etc. of the face image. Based on the detected key points of the image, subsequent comparison with the reference image is performed to perform image preprocessing.

[0088] Step 303: Based on the detected image key points and the reference image, the sample image is aligned to obtain an original image.

[0089] The reference image also has image key points. The image key points in the reference image and the image key points in the sample image describe the same image components. Based on the image components contained in both the reference image and the sample image, it is necessary to perform image alignment processing on the sample image with the reference image as the reference object, and deform the feature parts corresponding to the image key points in the sample image into the same spatial layout as the same feature parts in the reference image, so as to facilitate the model to achieve accurate style transfer processing of the sample image.

[0090] Specifically, the processing process may correspond to a model training process, which requires preprocessing of several collected sample images and performing image registration on the sample images based on reference images to meet the model training requirements.

[0091] Step 304 , collecting the spatial point coordinates of the object to be processed from the original image, and obtaining a 3D position map of the object to be processed based on the spatial point coordinates.

[0092] The implementation process of this step is the same as the implementation process of step 102 in the aforementioned embodiment, and will not be repeated here.

[0093] Step 305: Generate a 3D texture map of the object to be processed based on the position map and in combination with pixel information in the original image.

[0094] The implementation process of this step is the same as the implementation process of step 103 in the aforementioned embodiment, and will not be repeated here.

[0095] Step 306: input the texture map into the style transfer network to generate a 3D target texture map of the target style.

[0096] The implementation process of this step is the same as the implementation process of step 104 in the aforementioned embodiment, and will not be repeated here.

[0097] Step 307: construct a 3D target image corresponding to the original image based on the target texture map and the spatial point coordinates.

[0098] The implementation process of this step is the same as the implementation process of step 105 in the aforementioned embodiment, and will not be repeated here.

[0099] Step 308: construct a 2D target image corresponding to the original image based on the target texture map and the position map.

[0100] Combination Figure 2 As shown, while constructing a 3D target image, the construction of a 2D target image can also be achieved, meeting various image special effects processing requirements.

[0101] Through the position map corresponding to the object to be processed, the distribution of the position points of each component in the object to be processed can be known, and the texture information of each component in the object to be processed after style transfer processing can be obtained through the target texture map, so that based on the information of both, the 2D target image of the object to be processed including the special effects processing can be reconstructed.

[0102] Specifically, as an optional implementation, the 2D target image corresponding to the original image is constructed based on the target texture map and the position map, including:

[0103] The second target pixel points are extracted from the target texture map respectively; the three-dimensional coordinate points corresponding to each second target pixel point are extracted from the position map; the three-dimensional coordinate points are affine transformed into two-dimensional coordinate points, and the second target pixel points are correspondingly assigned to the two-dimensional coordinate points to obtain a 2D target image corresponding to the original image.

[0104] When generating a 2D target image, the corresponding three-dimensional coordinates in the position map are converted into two-dimensional coordinates, and pixel values ​​are assigned to the corresponding pixel points in the texture map after style transfer. 2D and 3D aged faces can be synthesized simultaneously according to needs, and the generation effect is excellent.

[0105] In the embodiment of the present application, through the above-mentioned implementation steps, a 3D special effects image is constructed based on the preprocessed plane sample image, and in the specific processing process, style transfer is performed based on the texture map of the object to be processed. While realizing the image special effects processing, more original structural features of the object to be processed are retained, the identity information of the object to be processed is retained to the maximum extent, and excessive loss of identity information in the special effects processing is avoided, thereby improving the realism of the final 2D and 3D image effects.

[0106] See also Figure 4 , Figure 4 This is a structural diagram of a 3D image construction device based on style transfer provided in an embodiment of the present application. For the sake of convenience of explanation, only the parts related to the embodiment of the present application are shown.

[0107] The 3D image construction device 400 based on style transfer includes:

[0108] A first acquisition module 401 is used to acquire a 2D original image containing an object to be processed;

[0109] A second acquisition module 402 is used to collect the spatial point coordinates of the object to be processed from the original image, and obtain a 3D position map of the object to be processed based on the spatial point coordinates;

[0110] A first generating module 403 is used to generate a 3D texture map of the object to be processed based on the position map and in combination with pixel information in the original image;

[0111] A second generating module 404 is used to input the texture map into a style transfer network to generate a 3D target texture map of a target style;

[0112] The image construction module 405 is used to construct a 3D target image corresponding to the original image based on the target texture map and the spatial point coordinates.

[0113] The second acquisition module is specifically used for:

[0114] Obtaining three-channel pixel values ​​of each pixel point in the object to be processed from the original image;

[0115] Based on the three-channel pixel values, mapping is performed to obtain the three-dimensional coordinates of the space point corresponding to the object to be processed;

[0116] Based on the three-dimensional coordinates of the spatial point, a 3D position map of the object to be processed is obtained;

[0117] The 3D position map of the object to be processed has the same size as the original image.

[0118] Wherein, the first generation module is specifically used for:

[0119] Matching target position points corresponding to each position point in the position map from the original image;

[0120] Extracting first target pixel points at the target position points from the original image in sequence;

[0121] The first target pixel points are arranged according to the position points in the position map to obtain a 3D texture map of the object to be processed.

[0122] Among them, the image construction module is specifically used for:

[0123] Arranging the spatial point coordinates to form a target number of three-point pairs, wherein each of the three-point pairs forms a triangle, and the target number of triangles form a 3D surface of the target image;

[0124] Based on the correspondence between the spatial point coordinates and each pixel point in the target texture map, extract the pixel value of the corresponding pixel point from the target texture map, and assign the pixel value to the vertex position of the triangle;

[0125] Calculating the relative distances between the remaining position points in each of the triangles and the three vertices of the triangle;

[0126] Based on the relative distance, setting a weight value of the pixel value;

[0127] Calculate the target pixel values ​​of the remaining positions in each of the triangles by combining the pixel values ​​assigned to the vertex positions of the triangles and the weight values ​​of the pixel values;

[0128] The target pixel value is assigned to the corresponding remaining position points to obtain a 3D target image corresponding to the original image.

[0129] The image building module is also specifically used for:

[0130] A 2D target image corresponding to the original image is constructed based on the target texture map and the position map.

[0131] The image construction module is more specifically used for:

[0132] Extracting second target pixel points respectively from the target texture map;

[0133] Extracting a three-dimensional coordinate point corresponding to each of the second target pixel points from the position map;

[0134] The three-dimensional coordinate points are affine transformed into two-dimensional coordinate points, and the second target pixel points are correspondingly assigned to the two-dimensional coordinate points to obtain a 2D target image corresponding to the original image.

[0135] The first acquisition module is specifically used for:

[0136] Get a preset reference image;

[0137] Detect image key points from a 2D sample image containing the object to be processed;

[0138] Based on the detected image key points and the reference image, image alignment processing is performed on the sample image to obtain the original image.

[0139] The 3D image construction device based on style transfer provided in the embodiment of the present application can implement each process of the embodiment of the above-mentioned 3D image construction method based on style transfer, and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0140] Figure 5 is a structural diagram of a terminal provided in an embodiment of the present application. As shown in the figure, the terminal 5 of this embodiment includes: at least one processor 50 ( Figure 5 Only one is shown in the figure), a memory 51, and a computer program 52 stored in the memory 51 and executable on the at least one processor 50, wherein the processor 50 implements the steps of any of the above-mentioned method embodiments when executing the computer program 52.

[0141] The terminal 5 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal 5 may include, but is not limited to, a processor 50 and a memory 51. Those skilled in the art will appreciate that Figure 5 It is only an example of terminal 5 and does not constitute a limitation on terminal 5. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal may also include input and output devices, network access devices, buses, etc.

[0142] The processor 50 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0143] The memory 51 may be an internal storage unit of the terminal 5, such as a hard disk or memory of the terminal 5. The memory 51 may also be an external storage device of the terminal 5, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal 5. Further, the memory 51 may also include both an internal storage unit of the terminal 5 and an external storage device. The memory 51 is used to store the computer program and other programs and data required by the terminal. The memory 51 may also be used to temporarily store data that has been output or is to be output.

[0144] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0145] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0146] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0147] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0148] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0150] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0151] The present application implements all or part of the processes in the above-mentioned embodiment method, and may also be implemented through a computer program product. When the computer program product runs on a terminal, the terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0152] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A 3D image construction method based on style transfer, characterized in that: include: Obtain a 2D original image containing the object to be processed; The spatial point coordinates of the object to be processed are collected from the original image, and a 3D position map of the object to be processed is obtained based on the spatial point coordinates; the content contained in the 3D position map is formed by spreading out the three-dimensional components of the object to be processed based on the three-dimensional coordinates of the spatial points corresponding to the object to be processed, wherein the spreading out means that based on the spatial point coordinates of the object to be processed collected from the original image, the coordinates of each three-dimensional component are estimated and the estimated coordinate information is filled into the corresponding position in the position map to obtain the 3D position map of the image to be processed; Based on the position map and in combination with pixel information in the original image, a 3D texture map of the object to be processed is generated; Inputting the texture map into a style transfer network to generate a 3D target texture map of the target style; Based on the target texture map and the spatial point coordinates, a 3D target image corresponding to the original image is constructed.

2. The method according to claim 1, characterized in that The step of collecting the spatial point coordinates of the object to be processed from the original image and obtaining a 3D position map of the object to be processed based on the spatial point coordinates includes: Obtaining three-channel pixel values ​​of each pixel point in the object to be processed from the original image; Based on the three-channel pixel values, mapping is performed to obtain the three-dimensional coordinates of the space point corresponding to the object to be processed; Based on the three-dimensional coordinates of the spatial point, a 3D position map of the object to be processed is obtained; The 3D position map of the object to be processed has the same size as the original image.

3. The method according to claim 1, characterized in that The step of generating a 3D texture map of the object to be processed based on the position map and in combination with pixel information in the original image includes: Matching target position points corresponding to each position point in the position map from the original image; Extracting first target pixel points at the target position points from the original image in sequence; The first target pixel points are arranged according to the position points in the position map to obtain a 3D texture map of the object to be processed.

4. The method according to claim 1, characterized in that: The step of constructing a 3D target image corresponding to the original image based on the target texture map and the spatial point coordinates includes: Arranging the spatial point coordinates to form a target number of three-point pairs, wherein each of the three-point pairs forms a triangle, and the target number of triangles form a 3D surface of the target image; Based on the correspondence between the spatial point coordinates and each pixel point in the target texture map, extract the pixel value of the corresponding pixel point from the target texture map, and assign the pixel value to the vertex position of the triangle; Calculating the relative distances between the remaining position points in each of the triangles and the three vertices of the triangle; Based on the relative distance, setting a weight value of the pixel value; Calculate the target pixel values ​​of the remaining positions in each of the triangles by combining the pixel values ​​assigned to the vertex positions of the triangles and the weight values ​​of the pixel values; The target pixel value is assigned to the corresponding remaining position points to obtain a 3D target image corresponding to the original image.

5. The method according to claim 1, characterized in that After the texture map is input into the style transfer network to generate a 3D target texture map of the target style, the method further includes: A 2D target image corresponding to the original image is constructed based on the target texture map and the position map.

6. The method according to claim 5, characterized in that The step of constructing a 2D target image corresponding to the original image based on the target texture map and the position map includes: Extracting second target pixel points respectively from the target texture map; Extracting a three-dimensional coordinate point corresponding to each of the second target pixel points from the position map; The three-dimensional coordinate points are affine transformed into two-dimensional coordinate points, and the second target pixel points are correspondingly assigned to the two-dimensional coordinate points to obtain a 2D target image corresponding to the original image.

7. The method according to claim 1, characterized in that The step of obtaining a 2D original image containing the object to be processed comprises: Get a preset reference image; Detect image key points from a 2D sample image containing the object to be processed; Based on the detected image key points and the reference image, image alignment processing is performed on the sample image to obtain the original image.

8. A 3D image construction device based on style transfer, characterized in that: include: A first acquisition module is used to acquire a 2D original image containing an object to be processed; A second acquisition module is used to collect the spatial point coordinates of the object to be processed from the original image, and obtain a 3D position map of the object to be processed based on the spatial point coordinates; the content contained in the 3D position map is formed by spreading out the various three-dimensional components of the object to be processed based on the three-dimensional coordinates of each spatial point corresponding to the object to be processed, wherein the spreading out means that based on the spatial point coordinates of the object to be processed collected from the original image, the coordinates of each three-dimensional component are estimated and the estimated coordinate information is filled into the corresponding position in the position map to obtain the 3D position map of the image to be processed; A first generating module, configured to generate a 3D texture map of the object to be processed based on the position map and in combination with pixel information in the original image; A second generation module is used to input the texture map into a style transfer network to generate a 3D target texture map of a target style; An image construction module is used to construct a 3D target image corresponding to the original image based on the target texture map and the spatial point coordinates.

9. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images